Extractive is not Faithful: An Investigation of Broad Unfaithfulness
Problems in Extractive Summarization

Text summarization is the process of distilling the most important information from a source to produce an abridged version for a particular user or task Maybury (1999). Although there are many types of text summarization tasks, in this work, we focus on the task of general purpose single document summarization. To produce summaries, usually either extractive summarization methods, i.e., extracting sentences from the source, or abstractive summarization methods, i.e., generating novel text, are applied Saggion and Poibeau (2013).

Abstractive summarization attracts more attention from recent works because it can produce more coherent summaries and behaves more like humans Cohn and Lapata (2008). Impressive progress has been made for abstractive summarization by large-scale pre-trained models Lewis et al. (2020); Zhang et al. (2020a). However, unfaithfulness problems, i.e., hallucinating new information or generating content that contradicts the source, are widely spread across models and tasks Cao et al. (2018); Maynez et al. (2020). Although these problems do not necessarily get captured by typically-used evaluation metrics, e.g., ROUGE Lin (2004), even minor unfaithfulness can be catastrophic and drive users away from real-world applications. Therefore, an increasing volume of research has focused on analyzing Falke et al. (2019); Maynez et al. (2020); Goyal and Durrett (2021), evaluating Kryscinski et al. (2020); Goyal and Durrett (2021); Wang et al. (2020a); Durmus et al. (2020); Scialom et al. (2021); Xie et al. (2021), or addressing Cao et al. (2018); Li et al. (2018); Fan et al. (2018); Chen et al. (2021); Cao and Wang (2021); Xu et al. (2022); Wan and Bansal (2022) unfaithfulness problems in abstractive summarization.

Extractive summarization is known to be faster, more interpretable, and more reliable Chen and Bansal (2018); Li et al. (2021); Dreyer et al. (2021). And the selection of important information is the first skill that humans learn for summarization Kintsch and van Dijk (1978); Brown and Day (1983). Recently, some works discuss the trade-off between abstractiveness and faithfulness Ladhak et al. (2022); Dreyer et al. (2021). They find that the more extractive the summary is, the more faithful it is.²²2Note that some previous works seemed to interchange the usage of factuality and faithfulness. But we think they are slightly different. Thus, we stick to faithfulness that represents the property of staying true to the source. This may give the community the impression that if the content is extracted from the source, it is guaranteed to be faithful. However, is this always true? In this work, we will show that, unfortunately, it is not.

The problems of extractive summarization are usually referred as coherence, out-of-context, or readability issues Nanba and Okumura (2000); Nenkova and McKeown (2012); Saggion and Poibeau (2013); Dreyer et al. (2021). Though they may sound irrelevant to faithfulness, some early works give hints of their unfaithful ingredients. Gupta and Lehal (2010) describe the ‘dangling’ anaphora problem – sentences often contain pronouns that lose their referents when extracted out of context, and stitching together extracts may lead to a misleading interpretation of anaphors. Barzilay et al. (1999) comment on extractive methods for multi-document summarization, that extracting some similar sentences could produce a summary biases towards some sources. Cheung (2008) says that sentence extraction produces extremely incoherent text that did not seem to convey the gist of the overall controversiality of the source. These all suggest that even though all information is extracted directly from the source, the summary is not necessarily faithful to the source. However, none of these works has proposed an error typology nor quantitatively answered how unfaithful the model extracted summaries are, which motivates us to fill in this missing piece.

In this work, we conduct a thorough investigation of the broad unfaithfulness problems in extractive summarization. Although the literature of abstractive summarization usually limits unfaithful summaries to those that are not entailed by the source Maynez et al. (2020); Kryscinski et al. (2020), we discuss broader unfaithfulness issues including and beyond not-entailment. We first design a typology consisting five types of unfaithfulness problems that could happen in extractive summaries: incorrect coreference, incomplete coreference, incorrect discourse, incomplete discourse, and other misleading information (see definitions in Figure 2). Among them, incorrect coreference and incorrect discourse are not-entailment based errors. An example of incorrect coreference is shown in Summary 1 of Figure 1, where that in the second sentence should refer to the second document sentence –But they do leave their trash, but it incorrectly refers to the first sentence in the summary. Summaries with incomplete coreferences or discourses are usually entailed by the source, but they can still lead to unfaithful interpretations. Lastly, inspired by misinformation O’Connor and Weatherall (2019), our misleading information error type refers to other cases where, despite being entailed by the source, the summary still misleads the audience by selecting biased information, giving the readers wrong impressions, etc (see Section 2).

We ask humans to label these problems out of 1600 model extracted summaries that are produced by 16 extractive summarization systems for 100 CNN/DM English articles Hermann et al. (2015). These 16 systems cover both supervised and unsupervised methods, include both recent neural-based and early graph-based models, and extract sentences or elementary discourse units (see Section 3). By analyzing human annotations, we find that 30.3% of the 1600 summaries have at least one of the five types of errors. Out of which, 3.9% and 15.4% summaries contain incorrect and incomplete coreferences respectively, 1.1% and 10.7% summaries have incorrect and incomplete discourses respectively, and other 4.9% summaries still mislead the audience without having coreference or discourse issues. The non-negligible error rate demonstrates that extractive is not necessarily faithful. Among the 16 systems, we find that the two oracle extractive systems (that maximize ROUGE Lin (2004) against the gold summary by using extracted discourse units or sentences) surprisingly have the most number of problems, while the Lead3 model (the first three sentences of the source document) causes the least number of issues.

We examine whether these problems can be automatically detected by 5 widely-used metrics, including ROUGE Lin (2004) and 4 faithfulness evaluation metrics for abstractive summarization (FactCC Kryscinski et al. (2020), DAE Goyal and Durrett (2020), QuestEval Scialom et al. (2021), BERTScore Zhang et al. (2020b)). We find that, except BERTScore, they have either no or small correlations with human labels. We design a new metric, ExtEval, for extractive summarization. It contains four sub-metrics that are used to detect incorrect coreference, incomplete coreference, incorrect or incomplete discourse, and sentiment bias, respectively. We show that ExtEval performs best at detecting unfaithful extractive summaries (see Section 4 for more details). Finally, we discuss the generalizability and future directions of our work in Section 5.

In summary, our contributions are (1) a taxonomy of broad unfaithfulness problems in extractive summarization, (2) a human-labeled evaluation set with 1600 examples from 16 diverse extractive systems, (3) meta-evaluations of 5 existing metrics, (4) a new faithfulness metric (ExtEval) for extractive summarization. Overall, we want to remind the community that even when the content is extracted from the source, there is still a chance to be unfaithful. Hence, we should be aware of these problems, be able to detect them, and eventually resolve them to achieve a more reliable summarization.

In this section, we will describe the five types of broad unfaithfulness problems (Figure 2) we identified for extractive summarization under our typology. In previous works about abstractive summarization, unfaithfulness usually only refers to the summary being not entailed by the source Maynez et al. (2020); Kryscinski et al. (2020). The formal definition of entailment is $t$ entails $h$ if, typically, a human reading $t$ would infer that $h$ is most likely true Dagan et al. (2005). While we also consider being not entailed as one of the unfaithfulness problems, we will show that there is still a chance to be unfaithful despite being entailed by the source. Hence, we call the five error types we define here the ‘broad’ unfaithfulness problems, and we provide a rationale for each error type in Figure 2.

The most frequent unfaithfulness problem of abstractive summarization is the presence of incorrect entities or predicates Gabriel et al. (2021); Pagnoni et al. (2021), which can never happen within extracted sentences (or elementary discourse units³³3Elementary Discourse Unit (or EDU) is a concept from the Rhetorical Structure Theory Mann and Thompson (1988). Each unit usually appears as a sub-sentence.). For extractive summarization, the problems can only happen ‘across’ sentences (or units).⁴⁴4Even though some may argue that extracted sentences should be read independently, in this work, we take them as a whole and follow their original order in the document. We think this is a reasonable assumption and shares the same spirit of previous works that talk about the coherence issue of extractive summaries Gupta and Lehal (2010). Hence, we first define four error types about coreference and discourse. Following SemEval-2010 Màrquez et al. (2013), we define coreference as the mention of the same textual references to an object in the discourse model, and we focus primarily on anaphors that require finding the correct antecedent. We ground our discourse analysis for systems that extract sentences in the Penn Discourse Treebank Prasad et al. (2008), which considers the discourse relation between sentences as “lexically grounded”. E.g., the relations can be triggered by subordinating conjunctions (because, when, etc.), coordinating conjunctions (and, but, etc.), and discourse adverbials (however, as a result, etc). We refer to such words as discourse linking terms. For systems that extract discourse units, we follow the Rhetorical Structure Theory Mann and Thompson (1988) and assume every unit potentially requires another unit to complete the discourse.

Finally, inspired by the concept of misinformation (incorrect or misleading information presented as fact), we define the fifth error type – misleading information that captures other misleading problems besides the other four errors. The detailed definitions of the five error types are as follows:

Incorrect coreference happens when the same anaphor is referred to different entities given the summary and the document. The anaphor can be a pronoun (they, she, he, it, etc.) or a ‘determiner (the, this, that, etc.) + noun’ phrase. This error makes the summary not entailed by the source. An example is Summary 1 of Figure 1, where the mention that refers to the sentence –But they do leave their trash. Thousands of pounds of it – in the document but incorrectly refers to Most climbers who try don’t succeed in summiting the 29,035-foot-high Mount Everest. Users who only read the summary may think there is some connection between cleaning up trash and the fact that most climbers do not succeed in summiting the Mount Everest.

Incomplete coreference happens when an anaphor in the summary has ambiguous or no antecedent.⁵⁵5Note that sometimes a ‘determiner + noun’ phrase has no antecedent, but it does not affect the understanding of the summary or there is no antecedent of the mention in the document either. In which case, it is not an anaphor, and thus we do not consider it as an incomplete coreference. Following the formal definition of entailment, these examples are considered to be entailed by the document. Nonetheless, it sometimes can still cause unfaithfulness, as it leads to ‘ambiguous interpretation’. For example, given the source “Jack eats an orange. John eats an apple” and the faithfulness of “He eats an apple” depends entirely on whom “he” is. Figure 1 illustrates an example of incomplete coreference, where Summary 2 starts with that’s why, but readers of that summary do not know the actual reason. Please refer to Figure 4 in the Appendix for another example with a dangling pronoun and ambiguous antecedents.

Incorrect discourse happens when a sentence with a discourse linking term (e.g., but, and, also, etc.)⁶⁶6We do not consider implicit (without a linking term) discourse relations between sentences because it hardly appears in our data and will cause a lot of annotation ambiguity. or a discourse unit falsely connects to the following or preceding context in the summary, which leads the audience to infer a non-exiting fact, relation, etc. An example is shown by Summary 2 in Figure 1, where More than 200 climbers have died falsely connects to clean up the trash, which makes readers believe 200 climbers have died because of cleaning up the trash. But in fact, they died attempting to climb the peak. This summary is also clearly not entailed by the source.

Incomplete discourse happens when a sentence with a discourse linking term or a discourse unit has no necessary following or preceding context to complete the discourse. Similar to incomplete coreference, summaries with this error are considered entailed, but the broken discourse makes the summary confusing and thus may lead to problematic interpretations. An example is shown in Figure 1. Summary 3 starts with but, and readers expect to know what leads to this turning, but it is never mentioned. See Figure 5 for another example that may leave readers with a wrong impression because of incomplete discourse.

Other misleading information refers to other misleading problems besides the other four error types. It includes but does not limit to leading the audience to expect a different consequence and conveying a dramatically different sentiment. This error is also difficult to capture using the entailment-based definition. Summaries always select partial content from the source, however, sometimes, the selection can mislead or bias the audience. Gentzkow et al. (2015) show that filtering and selection can result in ‘media bias’. We define this error type so that annotators can freely express whether they think the summary has some bias or leaves them with a wrong impression. The summary in Figure 6 is labeled as misleading by two annotators because it can mislead the audience to believe that the football players and pro wrestlers won the contest and ate 13 pounds of steak.

Note that we think it is also valid to separate misleading information and incomplete coreference/discourse, as they are less severe in unfaithfulness compared to not-entailment-based incorrect coreference/discourse, but we choose to cover all of them under the ‘broad unfaithfulness’ umbrella for completeness.

In this section, we describe how we ask humans to find and annotate the unfaithfulness problems.

Here, we analyze whether existing automatic faithfulness evaluation metrics can detect unfaithful extractive summaries. We additionally propose a new evaluation approach, ExtEval.

One future direction for resolving these unfaithfulness problems is to use the errors automatically detected by ExtEval as hints for humans or programs to fix the summary by doing necessary yet minimal edits. Here we illustrate the possibility for incorrect coreference. We manually examined the automatically detected incorrect coreferences by ExtEval. 28 out of 32 detected incorrect coreferences are true incorrect coreferences¹⁶¹⁶16It shows that ExtEval has high precision of $87.5\%$. However, we have 60 human-labeled incorrect coreferences, so the recall is only $46.7\%$ (28 out of 60)., which we attempt to fix by developing a simple post-edit program, similar to the revision system proposed by Nanba and Okumura (2000). The program replaces the problematic mention in the summary with the first mention in the correct coreference cluster of the document. We manually checked the corrected examples and found that 16 out of 28 were fixed correctly (see an example in Figure 7). We leave the improvement and the extension of post-edit systems for future work.

It is worth noting that all of the five error types we define in Section 2 can also happen in abstractive summarization, though they are less studied and measured in the literature. To our best knowledge, FRANK Pagnoni et al. (2021) and SNaC Goyal et al. (2022) have discussed the coreference and discourse errors in the abstractive summaries. Gabriel et al. (2021) define a sentiment error as an adjective or adverb appearing in the summary that contradicts the source, while our misleading information has a more general definition. We hope that our taxonomy can shed some light for future works to explore the broad unfaithfulness of all summarization methods.

We conducted a systematic analysis of broad unfaithfulness problems in extractive summarization. We proposed 5 error types and produced a human-labeled evaluation set of 1600 examples. We found that (i) 30.3% of the summaries have at least one of the 5 issues, (ii) existing metrics correlate poorly with human judgment, and (iii) our new faithfulness evaluation metric ExtEval performs the best at identifying these problems. Through this work, we want to raise the awareness of unfaithfulness issues in extractive summarization and stress that extractive is not equal to faithful.

We thank anonymous reviewers for their valuable comments. We thank Yinuo Hu and Abhay Zala for helping with the data, and Jiacheng Xu for helping us get the system outputs of the Oracle (discourse) model. We also thank Ido Dagan for the helpful discussions. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, and a Bloomberg Data Science Ph.D. Fellowship.

Since we focus on extractive summarization in this work, the conclusions will be more useful for summarization tasks where extractive methods perform decently well (e.g., CNN/DM Hermann et al. (2015)) compared to extremely abstractive summarization tasks (e.g., XSum Narayan et al. (2018a)). Experts, two of the authors (PhD students trained in NLP/CL) and two other CS undergraduate students (researchers in NLP/CL), conducted our annotations. Hence, to scale up data annotation by working with crowdsourcing workers may require additional training for the workers. Our ExtEval is designed for extractive summarization, which is currently not directly applicable for abstractive summaries except for SentiBias.

As our data is collected on CNN/DM, the percentages of each error type may change when evaluating a different summarization dataset, though we believe that the conclusion, extractive is not faithful, will not change. To initially verify our conjecture, we manually examine 23 oracle summaries from the test set of PubMed Sen et al. (2008) and find 2 incorrect coreferences, 5 incomplete coreferences, 1 incorrect discourse, and 1 incomplete discourse.

Many works have shown that model-generated summaries are often “unfaithful”, where the summarization model changes the meaning of the source document or hallucinates new content Cao et al. (2018); Maynez et al. (2020). This potentially causes misinformation in practice. Our work follows the same idea, but, as opposed to focusing on abstractive summarization, we show that even extracting content from the source document can still alter the meaning of the source document and cause misinformation. Hence, we want to remind NLP practitioners that even extractive is not faithful and these issues need to be addressed before we can trust model-produced extractive summaries for real-world applications.