Towards Understanding Omission in Dialogue Summarization

In summarization tasks, omission²²2In some previous works, omission is also called missing information Chen and Yang (2020); Tang et al. (2022). is one of the most common factual errors in abstractive summaries. It usually refers to the missing content in the candidates, which is presented in the gold reference. The definition of omission content is flexible, which could refer to either the omitted keywords, text spans, or utterances. In dialogues, an utterance has a clearer boundary compared to text spans and can be viewed as a basic unit for identification and evaluation. Therefore, in this paper, we mainly focus on utterance-level omission and provide utterance-level labels. Table 1 shows an example of our Olds dataset, which contains the original dialogue, reference summary, candidate summary, and omission labels. In this example, the candidate summary omits three key messages: the person “Adam”, the attitude “worried” and the persona “psychologist”, and thus the corresponding utterance-level omission labels are the 3rd and 15th utterances in the original dialogue.

Olds is a dataset that collects multiple candidates for dialogue summarization and provides their corresponding omission labels at the utterance level. This dataset first collects multiple public benchmarks, including SAMSum Gliwa et al. (2019), DialogSum Chen et al. (2021), EmailSum Zhang et al. (2021), QMSum Zhong et al. (2021) and TweetSumm Feigenblat et al. (2021), to cover different dialogue domains.

Then, in order to collect samples with omission contents, we still need to generate candidate summaries for each dialogue. To gain deeper insights into the omission problem induced by models with different capacities, we select 6 different model settings ³³3We do not use extractive models because dialogue summaries are very abstractive. There is a huge gap in the format and style between summary sentences and dialogue utterances., including BART_large/base Lewis et al. (2020), T5_base/small Raffel et al. (2020), Transformers Vaswani et al. (2017), and Pegasus_large Zhang et al. (2020), to generate candidate summaries ⁴⁴4Pegasus_large is only used to generate summaries for dialogues in the validation/test sets. The purpose is to conduct the robustness evaluation on candidates from unseen sources. .

Finally, based on the collected candidate summaries, we need to identify which salient information is omitted in these candidates. Therefore, we elaborately design a strategy to label omission automatically and the details are described in the next subsection. As a result, our Olds is able to obtain multiple candidates and their corresponding omission label for each dialogue. More details about the dataset creation can refer to Appendix A.

It is generally a non-trivial task to identify the missing critical content in candidate summary. Fortunately, the existing datasets provide reference summaries as ground truths. We could locate the omitted information in dialogue by directly comparing candidates with references. Thus, we design a pipeline strategy for automatic omission labeling, which is composed of three steps: oracle extraction, omission identification, and redundancy removal. Appendix A.1 shows an example of the complete process of automatic omission labeling.

To assess the quality of the extracted omission labels for the Olds dataset, we also conducted human evaluation to validate the correctness of the labeled utterances. We recruited three annotators with NLP backgrounds and each annotator is required to answer the question whether the set of labeled omission utterances is Accept or Reject. The set should be marked as Reject as long as it misses any critical utterance (recall of labeled omissions), or includes any redundant or uninformative utterance (precision of labeled omissions). Otherwise, it should be marked as Accept. To this end, we randomly sampled 200 dialogue-candidate pairs from each domain for assessment. Table 3 reports the results of the human evaluation for quality assessment. The acceptance rate of human evaluation ranges between 91.2%-98.5%, which validates the effectiveness of our omission extraction strategy. Furthermore, in order to evaluate the reliability of this assessment, we measure the agreement between different annotators by reporting Fleiss’ Kappa values Fleiss et al. (1971) among the possible combinations of two annotators, as reported in Table 3. We find that the overall Kappa score is 0.653, which shows the substantial agreement between annotators. Overall, the results of human evaluation demonstrate that our omission extraction strategy is able to produce high-quality omission labels automatically. More details about human evaluation can refer to Appendix A.4.

An example of our Olds dataset is shown in Table 1, which contains the basic information, such as dialogue, reference, candidate, and omission labels. In the released version of Olds, we further provide some auxiliary information. The detailed dataset format and a complete example can be seen in Appendix A.5. Table 2 shows the statistics of the Olds dataset. We can see that the dialogues are from different domains, with different lengths and turns. Besides, the lengths of summaries also differ from each other, and the employed abstractive models are able to produce candidates with different qualities. We expect that our dataset could pave the way for analyzing the omission problem across different domains and diverse candidate summaries.

To explain the importance of the omission problem, we answer the following two questions.

Since omission is defined by the difference between references and candidates, we thus investigate the correlation between the amount of omission content and a variety of reference-based metrics, to verify whether the omission rate of a candidate summary could affect these metrics. Here, we calculate the omission rate as follows:

$${\rm OmissionRate}=\frac{\sum_{u\in O}|W^{u}|}{\sum_{u\in G}|W^{u}_{G}|},$$

(1)

where $W^{u}$ and $W^{u}_{G}$ denote the set of omitted words and the set of gold oracle words shared across $u$ and the reference, respectively. It directly measures the amount of key information omitted by a summary, and a lower rate indicates the candidate is of higher quality. Table 4 demonstrates the Pearson correlations between omission rate and other reference-based metrics, including $n$-gram based metrics ROUGE Lin (2004) and BLEU Papineni et al. (2002), embedding-based metric BERTScore Zhang et al. (2019), and learning-based metric BLEURT Sellam et al. (2020). The results indicate that most of the reference-based metrics moderately correlate with the omission rate, among which BERTScore_Large is the most stable metric that has a better correlation with the amount of omission content. By contrast, BLEU shows the least correlation because it is a precision-oriented metric. Empirical analyses indicate that the omission rate is strongly correlated with a wide range of evaluation metrics, and so how to mitigate the omission problem is one of the most important priorities to improve the quality of dialogue summaries.

The above analyses demonstrate the importance of omission information. So we raise another question: what happens if we utilize the omissions to refine the summary quality? Hence, we adopt a post-editing method to investigate the potential of using omissions. Specifically, we formulate summary refinement as a seq2seq task to predict the gold summary. Instead of inputting raw dialogue, we use the concatenation of candidate summary, omission utterances, and non-omission utterances as the input: “Candidate <sep> Omission <sep> Non-Omission”. By dividing dialogue utterances into the omission and non-omission groups, the model is able to distinguish omission information while perceiving the whole dialogue simultaneously. If the omission group is empty, it is identical to using candidate and raw dialogue for refinement, and we consider it as the baseline for comparison. We use BART_large and T5_small as the backbone model, and the results are shown in Figure 3. The results show that performances are significantly enhanced by the refinement using omissions compared to that using raw dialogues. Notably, on some datasets like SAMSum and DialogSum, T5_small with supplementary omission information even outperforms the raw BART_large, which indicates that omission-based refinement is a promising direction for quality improvement in dialogue summarization.

In addition, Figure 3 also shows an upper bound of performance boost by post-editing because we directly employ the gold omission utterances. However, in real situations, we may identify some incorrect omissions. To further explore the impact of wrong omissions on the post-editing results, we investigate three different perturbations by gradually injecting errors into the omission group: 1) we keep the precision as 1 and decrease the recall by moving utterances from the omission group to the non-omission group; 2) we keep the recall as 1 and decrease the precision by moving utterances from the non-omission group to the omission group; 3) we gradually exchange utterances in the two groups until they are swapped, and both the precision and recall decrease from 1 to 0. Figure 4 depicts the trend of performance degradation as the error rate increases. From the curves, we can find that the precision is relatively more important because the refinement model performs more robustly in the first type of perturbation and is sensitive to the addition of wrong omissions.

To build a foundation for the omission detection task and explore what model architecture the task could benefit from, we investigate three frameworks as baselines, which have different input formats and structures. Their implementation and training details can be found in Appendix B.1.

We use the standard Precision (P), Recall (R), and F1-score (F1) metrics on the utterance level to evaluate omission detection models. Furthermore, we calculate the percentage of gold omission words that are hit in the detected utterances to measure the word-level omission recall:

$$WR=\frac{\#\mathrm{hit\ omission\ words}}{\#\mathrm{gold\ omission\ words}}.$$

(2)

$\#$ means the counted number. The closer the word-level omission recall is to 1, the more the omission information is collected by the detection model.

Table 5 presents the experimental results on Olds. All detection models are separately trained on the five domains. For each omission detection framework, we employ BERT_base and RoBERTa_base as the backbone model to extract text features. Among these three frameworks, pair-wise classification performs the worst in most cases since it does not consider contextual information of dialogue. Meanwhile, sequence labeling is on par with the pointer network, which indicates that dialogue context is a crucial factor for models to detect the omitted content. However, although omission detection models only need to make a choice of whether the given utterance is an omission, the task is still very challenging. In Table 5, the best F1 score is around 50% in all five domains, while the recalled omission words in extracted utterances (WR) are around 60%. Besides, models in QMSum only achieve at most a F1-score of 41.35 and we guess it is due to the effect of longer dialogue in QMSum (over 1K tokens in Table 2). Intuitively, summarizers produce the candidates that have picked the low-hanging fruit, and the remaining omission information is a tough nut to crack. In other words, there exists some salient information omitted by the summarizer that is still difficult for detection models to capture.

To understand what factors may affect the performance of the detection model, we conduct the following explanatory experiments.

From the results in Table 5, we could observe that omission detection is a challenging task. Hence, we summarize some research directions as follows:

• •

  One direction is to develop a more advanced model for omission detection. Based on the analysis of Section 3.3, we could focus on improving the precision of omission detection results because a high precision of detected omissions would bring benefit to the refinement model. An ideal detection model could serve as a model-based metric for reference-free summary evaluation. Besides, we could use the detected omission to improve the results of summarization.

• •

  Another research direction is to develop a refinement model for summary improvement using the detected omissions. In this paper, we briefly touch on this by introducing a post-editing approach in Section 3.3. The approach is straightforward, and the whole summarization procedure becomes a summarize-then-refine pipeline. However, the results show that the model is sensitive to wrong omissions. Hence, how to design a robust refinement model is also noteworthy.

Dialogue summarization is a challenging and valuable task that has recently received much attention, where a variety of dialogue domains are investigated, such as mail threads Rambow et al. (2004); Zhang et al. (2021), meetings Chen and Yang (2020); Zhong et al. (2021), customer service Zou et al. (2021a, b); Feigenblat et al. (2021), medical conversations Joshi et al. (2020); Song et al. (2020), and daily chats Gliwa et al. (2019); Chen et al. (2021). Different from conventional documents, dialogues have several inherent characteristics that make the summarization task more challenging Zou et al. (2021c); Feng et al. (2022), e.g., multi-party information, coreferences, topic drifting, etc. Recent works have explored the types of errors in generated dialogue summaries to develop robust models Tang et al. (2022), and omission is assessed as the most dominant error type in candidate summaries, which is also supported by human evaluations in previous works Chen and Yang (2020); Liu and Chen (2021); Liu et al. (2021). In this work, we comprehensively analyze the omission problem in dialogue summarization based on the curated benchmark and investigate the feasibility of omission detection for generated candidates.

Omission is a common error in machine translation (MT) Russell (1999); Sharma (2015); Yang et al. (2019) and automatic speech recognition (ASR) tasks Weng et al. (2020), which usually denotes the missing source information in the generated sequences. Although both summarization and MT/ASR belong to generation tasks, the definitions of omission error are different among these tasks. In MT/ASR tasks, the tokens between source and target sequences are usually well aligned, which means each token in the target sequence can locate its corresponding content in the source sequence. Due to such characteristics, previous works Tu et al. (2016) in MT/ASR tasks usually adopted coverage mechanisms to eliminate the influence of omission error. Nevertheless, the source sequences in summarization tasks usually include abundant redundant and useless information, especially in dialogue scenarios, which makes omission a more serious problem in summarization-like tasks.