Entity-level Factual Consistency of Abstractive Text Summarization

Many recent advances in deep neural networks have led to significant improvement in the quality of abstractive summarization Radford et al. (2019); Gehrmann et al. (2019); Lewis et al. (2019). Despite this progress, there are still many limitations facing neural text summarization Kryscinski et al. (2019), the most serious of which is their tendency to generate summaries that are not factually consistent with the input document; a factually consistent summary only contains statements that can be derived from the source document. Recent studies show that about 30% of the summaries generated by neural network sequence-to-sequence models suffer from fact fabrication Cao et al. (2018). Unfortunately, the widely used ROUGE score is inadequate to quantify factual consistency Kryscinski et al. (2019).

Factual inconsistency can occur at either the entity or the relation level. At the entity level, a model generated summary may contain named-entities that never appeared in the source document. We call this the entity hallucination problem. For example, consider the following model generated summary:

People in Italy and the Netherlands are more likely to consume fewer cups of coffee than those in the UK, a study suggests.

“UK” never appeared in the input source document (taken from the test set of the XSUM dataset Narayan et al. (2018)). In fact, the source document mentioned a study involving people in Italy and Netherlands; “UK” was a result of model hallucination. Another type of inconsistency occurs when the entities indeed exist in the source document but the relations between them are not in the source document. This type of inconsistency is much harder to identify. Open Information Extraction (OpenIE) and dependency parsing tools have been used Cao et al. (2018) to identify the underlying relations in a summary, but are not yet accurate enough for practical use. Ultimately, these researchers relied on manually classifying generated summaries into faithful, fake, or unclear.

In this paper, we propose a set of simple metrics to quantify factual consistency at the entity-level. We analyze the factual quality of summaries produced by the state-of-the-art BART model Lewis et al. (2019) on three news datasets. We then propose several techniques including data filtering, multi-task learning and joint sequence generation to improve performance on these metrics. We leave the relation level consistency to future work.

Large transformer-based neural architectures combined with pre-training have set new records across many natural language processing tasks Vaswani et al. (2017); Devlin et al. (2019); Radford et al. (2019). In particular, the BART model Lewis et al. (2019) has shown superior performance in many text generation tasks including abstractive summarization. In contrast to encoder-only pre-training such as in BERT Devlin et al. (2019) or decoder-only pre-training such as in GPT-2 Radford et al. (2019), BART is an encoder-decoder transformer-based neural translation model jointly pre-trained to reconstruct corrupted input sequences of text.

Several authors have pointed out the problem of factual inconsistency in abstractive summarization models Kryscinski et al. (2019); Kryściński et al. (2019); Cao et al. (2018); Welleck et al. (2019). The authors in Kryściński et al. (2019) proposed to train a neural network model to classify if a summary is factually consistent with a given source document, similar to a natural language inference task. In the dialogue generation setting, authors in Li et al. (2019) proposed using unlikelihood to surpress logically inconsistent responses. Our work is complementary to such existing approaches as we focus on simple entity-level metrics to quantify and improve factual consistency. Our goal of improving entity-level metrics of summaries is also related to controllable abstractive summarization Fan et al. (2018), where a list of named-entities that a user wants to see in the summary can be passed as input to influence the generated summary. In contrast, our goal is to predict which entities are summary-worthy while generating the summary that contains them. In this view we are trying to solve a more challenging problem.

We propose three new metrics that rely on off-the-shelf tools to perform Named-Entity Recognition (NER). ¹¹1We use Spacy Honnibal and Montani (2017). We use $\mathcal{N}(t)$ and $\mathcal{N}(h)$ to denote the number of named-entities in the target (gold summary) and hypothesis (generated summary), respectively. We use $\mathcal{N}(h\cap s)$ to denote the number of entities found in the generated summary that can find a match in the source document. If a named-entity in the summary consists of multiple words, we consider it a match as long as any n-gram of the named-entity can be found in the source document. This is meant to capture the situation where the named-entity can be shortened; for example, “Obama ” is a match for “Barack Obama” and “Harvard” is a match for “Harvard University”. When the match is at the unigram level, we make sure that it is not a stop word such as “the”. We also make the match case-insensitive to accommodate casing variances.

In addition to entity-based data filtering, we also explore another method to further improve the summarization quality. In particular, we incorporate an additional task of classifying summary-worthy named-entities in the source document. A summary-worthy named-entity in the source document is one that appears in the ground truth summary and thus, is a salient entity, worthy of inclusion in the generated summary. Intuitively, if we can identify these summary-worthy named-entities using the encoder representation, we may potentially increase the entity-level precision and recall metrics as well as the overall quality of the summary. We achieve this by adding a classification head to the encoder of BART. To prepare for the classification label, we first identify the named-entities in the ground truth summary and find the matching tokens in the source document. We then assign the (B)eginning-(I)nside-(O)utside labels to each token of the source document to denote if the token is beginning, inside or outside of a summary-worthy named-entity, respectively. During training, we simply add the classification loss for each token at the encoder to the original sequence-to-sequence loss.

More precisely, let $\{\left(x^{i},y^{i}\right)\}_{i=1}^{N}$ be a dataset of $N$ examples where $x^{i}=x^{i}_{1},\dots,x^{i}_{ts(i)}$ are the tokens of the $i$th source document and $y^{i}=y^{i}_{1},\dots,y^{i}_{tt(i)}$ are the tokens of the target (ground truth summary). The standard sequence-to-sequence training minimizes the maximum log likelihood estimation (MLE) loss:

$$\mathcal{L}^{i}_{\text{MLE}}(\theta,x^{i},y^{i})=-\sum_{t=1}^{tt(i)}\log p_{\theta}(y^{i}_{t}|x^{i},y^{i}_{<t}).$$

With summary-worthy entity classification, each example has an additional sequence of BIO labels $z^{i}=z^{i}_{1},\dots,z^{i}_{ts(i)},z^{i}_{t}\in\{0,1,2\}$. By adding an additional fully connected layer on top of the BART encoder, we obtain the classification loss

$$\mathcal{L}^{i}_{\text{BIO}}(\theta(\text{enc}),x^{i},z^{i})=-\sum_{t=1}^{ts(i)}\log p_{\theta(\text{enc})}(z^{i}_{t}|x^{i}).$$

Finally, we can minimize the joint loss $\mathcal{L}^{i}_{\text{Multitask}}=\mathcal{L}^{i}_{\text{MLE}}+\alpha\mathcal{L}^{i}_{\text{BIO}},$ where $\alpha$ is a hyper parameter. We choose $\alpha$ between 0.1 to 0.5 via the validation sets.

We also explore another generative approach to promote entity-level precision and recall metrics. In particular, instead of just generating the summary, we train the BART model to generate the sequence of summary-worthy named-entities, followed by a special token, and then the summary. We call this approach JAENS (Join sAlient ENtity and Summary generation). Similar to the multi-task learning approach discussed earlier, JAENS encourages the model to jointly learn to identify the summary-worthy named-entities while learning to generate summaries. Since the decoder generates the salient named-entities first, the summaries that JAENS generate can further attend to these salient named-entities through decoder self-attention.

We use the pre-trained BART-large model in the Fairseq library Ott et al. (2019) to fine-tune on the 3 summarization datasets.³³3Our code is available at https://github.com/amazon-research/fact-check-summarization The appendix contains additional details of experimental setup.

In Table 3, we show the effect of the entity-based data filtering. For each dataset, we train two separate models: using the training data before and after entity-based data filtering as shown in Table 2. We evaluate both models on the “clean” test set after entity-based data filtering. We choose this filtered version of the original test set because we only want to measure entity-level consistency against the correct set of entities; using the unfiltered dataset means we could count a hallucinated entity as correct. We observe improvements of $\mathbf{prec}_{s}$ across all three datasets trained using the filtered subset of data. For example in XSUM, the $\mathbf{prec}_{s}$ is increased from 93.6% to 98.2%, indicating a significant reduction in entity hallucination. In addition, the entity-based data filtering generally improves other entity-level metrics as well. Even with less training data, the entity-based data filtering is able to maintain the ROUGE scores quite well. For XSUM, about 34% of the training data is filtered out (c.f. Table 2), which explains the more noticable impact on the ROUGE scores. The results in Table 3 suggest that entity-level data filtering is a simple yet effective approach to achieve higher entity-level factual consistency as well as general summarization quality. In Table 4 we provide qualitative examples where the model trained on the original data produces hallucination and the entity-level data filtering removes such hallucination.

Table 3 shows that adding the classification task (multi-task) futher increases the $\mathbf{prec}_{t}$ and $\mathbf{recall}_{t}$ metric and therefore the overall entity-level $F1_{t}$ on top of the improvements from data filtering. Similar gains can be observed with JAENS, which out-performs the multi-task approach on CNNDM and Newsroom datasets. The result confirms our intuition that the summaries in JAENS can benefit from attending to the generated salient entities in terms of the entity level metrics. However, the additional complexity during decoding may have hurt the ROUGE scores.

For the interested readers, we also evaluated the PEGASUS Zhang et al. (2020) models for the ROUGE and entity level metrics on these three datasets in the appendix.

In this paper we study the entity-level factual consistency of the state-of-the-art summarization model. We propose precision-source score $\mathbf{prec}_{s}$ to quantify the degree of entity hallucination. We also propose additional metrics $\mathbf{prec}_{t}$ and $\mathbf{recall}_{t}$ to measure entity level accuracy of the generated summary with respect to the ground truth summary. We found that the ground truth summaries of the XSUM dataset contain a high level of entity hallucination. We propose a simple entity-level data filtering technique to remove such hallucination in the training data. Experiments show that such data filtering leads to significant improvement in $\mathbf{prec}_{s}$. ($\mathbf{prec}_{s}$ increases from below 94% to above 98% in XSUM for example.) We futher proposed a multi-task learning and a joint sequence generation approach to further improve the entity-level metrics. Overall, combining our proposed approaches significantly reduces entity hallucination and leads to higher entity level metrics with minimal degradation of the ROUGE scores.