Question-Interlocutor Scope Realized Graph Modeling over Key Utterances for Dialogue Reading Comprehension

Given a dialogue consisting of N utterances: $\mathcal{D}$ = $[utter_{1}$ $,utter_{2},...,utter_{N}]$, the task aims to extract the answer span $a$ for a question $q=[qw_{1},qw_{2},...,qw_{L_{q}}]$ from $\mathcal{D}$, where $qw_{i}$ is the i-th word in $q$ and $L_{q}$ is the length of $q$. In $\mathcal{D}$, each utterance $utter_{i}=\{\mathrm{speaker}:s_{i},\mathrm{text}:t_{i}\}$ contains its corresponding speaker (e.g., $s_{i}=$“Chandler Bing”) and text content $t_{i}=[tw_{1},tw_{2},...,tw_{L_{i}}]$, where $tw_{j}$ the j-th word in $t_{i}$ and $L_{i}$ is the length of $t_{i}$. For some unanswerable questions, there is no answer span to be found in $\mathcal{D}$. Under such a circumstance, $a$ is assigned to be null.

Dialogue is a process whose utterances are not isolated but contextually interact with each other. Therefore, encoding words in an utterance requires considering contextual information from other utterances. To better encode words contextually using pretrained models (PTM), following previous work (Li and Zhao 2021), we chronologically concatenate utterances in the same conversation to form a text sequence: $\mathcal{C}$ = “$s_{1}$: $t_{1}$ $\mathrm{[SEP]}$ … $\mathrm{[SEP]}$ $s_{N}$: $t_{N}$”, where [SEP] is a special token of PTM for separation. Holding the conversational context $\mathcal{C}$, PTM can deeply encode $\mathcal{C}$ with the question $q$ to make it question-aware by concatenating them as $QC$ = “$\mathrm{[CLS]}\ q\ \mathrm{[SEP]}\ \mathcal{C}\ \mathrm{[SEP]}$” (it is okay that $\mathcal{C}$ goes first), where [CLS] is also a special token for classification.

Following Li and Zhao (2021), we utilize the ELECTRA discriminator to encode the sequence $QC$:

where $H_{QC}\in\mathbb{R}^{L_{QC}\times d_{h}}$, $L_{QC}$ is the length of $QC$, and $d_{h}$ is the hidden size of PLM. $H_{QC}$ can be split into $H_{Q}\in\mathbb{R}^{L_{q}\times d_{h}}$ and $H_{C}\in\mathbb{R}^{L_{C}}$ according the position of [SEP] between $q$ and $C$, where $L_{C}$ is the length of $\mathcal{C}$.

Treating every single utterance as a unit to pair with the question prefers utterances similar to the question. However, the utterance containing the answer is not always in the case, where it can appear near the similar utterance within several steps due to the topic relevance. Therefore, we apply a window along the dialogue. Utterances in the window are treated as a unit so that the similar utterance and the answer-contained utterance can co-occur and more answer-contained utterances can be realized.

To guide models to further realize the question, speakers in the question, and the scope of the corresponding speakers in $\mathcal{D}$, we construct a Question-Interlocutor Scope Realized Graph (QuISG) based on key utterances in $key$. QuISG can be formulated as $\mathcal{G}$ = ($\mathcal{V}$, $\mathcal{A}$), where $\mathcal{V}$ denotes the set of nodes involved in $\mathcal{G}$ and $\mathcal{A}$ denotes the adjacent matrix to indicate edges. After the construction of QuISG, we utilize a node type realized graph attention network to process it. We elaborate the graph modeling below.

After graph modeling, nodes in the QuISG are then mapped back into the original token sequence. As answers are extracted from the dialogue, only nodes in key utterances are considered. We locate the dialogue word (scene) node $v_{x}$ with its corresponding token representation $H_{C}[utter_{i}[x]]$ in $\mathcal{C}$, we then update the token representation by $H_{C}[utter_{i}[x]]$ += $h_{\mathrm{v}_{x}}^{T}$. For the speaker token representation $H_{C}[$Ross_i, Gellar${}_{i}]$ in key utterances, the mapped dialogue speaker node $\mathrm{v}_{s_{i}}$ updates it by $H_{C}[$Ross_i, Gellar${}_{i}]$ += [$h_{\mathrm{v}_{s_{i}}}^{T}$, $h_{\mathrm{v}_{s_{i}}}^{T}$]. As a speaker name $s_{i}$ may appear several times, we repeat adding duplicated $h_{\mathrm{v}_{s_{i}}}^{T}$ to the corresponding token representations. We denote the updated $H_{C}$ as $H_{C}^{{}^{\prime}}$.

Tab. 1 shows the results achieved by our method and other baselines on FriendsQA. Baselines listed in the first three rows are all based on BERT (Devlin et al. 2019). We can see that SelfSuper achieves better and competitive results compared with ULM+UOP and KnowledgeGraph, where ULM+UOP considers no speaker information and KnowledgeGraph introduces external knowledge. This indicates the effectiveness of the self-supervised tasks for speaker and key utterance modeling of SelfSuper. When it comes to ELECTRA, the performance reach a new elevated level, which shows that ELECTRA is more suitable for DRC. By comparing with SelfSuper based on ELECTRA, our method can achieve significantly better performance. This improvement shows the advantage of both the higher coverage of answer-contained utterances by our proposed key utterances extractor and better graph representations to consider the question and interlocutor scopes by QuISG.

Results on Molweni are listed in Tab. 2. Our approach still gives new state-of-the-art, especially a significantly improvement on EM scores. However, the absolute improvement is smaller compared to that on FriendsQA. This is mainly for two reasons. First, the baseline results are close to the human performance on Molweni, so the space for improvement is also smaller. Second, Molweni consists of unanswerable questions, which are not the main focus of our work. To see how the unanswerable questions affect the results, we further show the performance of our method and the ELECRTA-based baselines on Molweni-A in Tab. 3, i.e., the subset of Molweni with only answerable questions. We observe that our method still achieves better EM score against SelfSuper and gains slightly better F1 score, which indicates that our method can better deal with questions with answers. As for unanswerable questions, we believe that better performance can be achieved with related-techniques plugged in our method, which we leave to future work.

By comparing the performance gains of our method in FriendsQA and Molweni, we can observe that our method is more significant in FriendsQA. We think the reason may be that (1) our key utterance extractor can cover more answer-contained utterances in FriendsQA, as will be shown in Fig. 3; (2) questions with speaker mention show more frequently in FriendsQA than in Molweni, and therefore QuISG can help achieve better graph representations in FriendsQA. On all accounts, this further demonstrates that our method alleviates the problems that we focus on.

To demonstrate the importance of our proposed modules, we adapt ablation study on both datasets. The results are shown in Tab. 4. We study the effects of node type information (NodeType), key utterances extractor and its scaling factor on logits (KeyUttExt); question and question speaker nodes (Q); edges between dialogue word nodes and dialogue speaker nodes to model interlocutor scope (SpkScope). We further remove both KeyUttExt and QuISG, leading to fully connections between every two tokens in dialogues, and apply transformer layers to further process dialogues (w/o All).

By removing NodeType, the performance drops, which demonstrates minding different node behaviors can help better model graph representations. Our method w/o KeyUttExt decreases the performance, which demonstrates that key utterance extractor is a crucial module for our method to find more answer-contained utterances and guides our model to pay more attention to the key part in a dialogue for the question. As for the model w/o KeyUttExt shows more performance drop in FriendsQA, we think the reason may be that dialogues in FriendsQA are much longer than Molweni. Therefore, KeyUttExt can reduce more question unrelated parts of dialogues for further graph modeling in FriendsQA. Removing Q or SpkScope also shows performance decline, which indicates the importance of realizing question and interlocutor scope for DRC. The worse results of w/o Q indicates that questions and speakers mentioned in questions are more important to DRC due to the question-involved particularity of DRC. Replacing KeyUttExt and QuISG with transformer layers even performs worse than ELECTRA, which indicates that the further process of dialogues without speaker and question realized modeling can be redundant.

As we claim that our method covers more answer-contained utterances compared with SelfSuper, in this section, we show the recall of answer-contained utterances by different methods. Besides our method and SelfSuper, we further consider retrieval methods appearing other reading comprehension tasks. As the similarity-based seeker is usually used, we apply the SOTA model SimCSE (Gao, Yao, and Chen 2021) to compute similarity score between utterances and the question. However, directly using top similar utterances produces a extremely low recall. Therefore, we also add utterances around every picked top utterance as key utterances like ours. We consider top 3 similar utterances and its historical 2 and future 2 utterances as key utterances. The results are illustrated in Fig. 3. As shown in Fig. 3, we choosing top 3 units for key utterances selection does not affect the recall a lot and can keep the average size of key utterance set to 4.13 for FriendsQA and 3.61 for Molweni. Compared with our method, SelfSuper achieves undesirable recall for answer-contained utterance extraction, which indicates the efficacy of our method. As for SimCSE, equipped with our enhancement, it can achieve competitive recall to ours. Especially on the dev and test sets of Molweni. However, the average size of key utterance set of SimCSE is 7.73, where the average length of dialogue in Molweni is 8.82. Additionally, SimCSE cannot judge whether a question is answerable and therefore extracts key utterances for every question, leading to a low recall for the training set in Molweni.

To further demonstrate that our method is more suitable for DRC against SimCSE, we run a variant with key utterances sets extracted by SimCSE. The results are shown in Tab. 5. Our method achieves better performance with high coverage of answer-contained utterances and fewer key utterances involved.

As QuISG focuses on the question speaker information and dialogue interlocutor scope modeling, whether it can help answer questions mentioning speaker names is crucial to verify. We illustrate the F1 scores of questions containing different speakers in FriendsQA and questions with or without mentioning speakers in Fig. 4. We can see that SelfSuper outperforms our method only on “Rachel” and is slightly better on “Joey” and “Monica”. Our method can outperform SelfSuper by a great margin on “Ross”, “Phoebe”, “Chandler” and other casts. Specifically, our method can improve the F1 score of questions mentioning speaker by a wider margin compared to questions without speakers. This indicates that our speaker modeling is benefit from our proposed method.

As shown in the very beginning of the paper, Fig. 1 provides two cases that SelfSuper fails. On the contrary, attributing to our proposed key utterances extractor and QuISG, our method can answer the two questions correctly.