Fill in the BLANC:
Human-free quality estimation
of document summaries

Two most widely used methods for measuring the quality of a summary are ROUGE Lin (2004) and human evaluation Kryściński et al. (2019a).

The ROUGE family of methods are well-defined and reproducible. However, these methods typically require a human-written reference summaries for comparison, completely disregarding the original document text. Even if one assumes that a reference summary is available and of optimal quality, the ROUGE method is limited to measuring a mechanical overlap of text tokens with little regard to semantics. This deficiency may be partially addressable through measurement of the similarity not of text tokens but named entities or other preprocessed features Mao et al. (2019); Cohan and Goharian (2016); Elghannam and El-Shishtawy (2015); Ng and Abrecht (2015); Ganesan (2018) or embeddings Zhao et al. (2019); Zhang et al. (2020); Gao et al. (2020). In the latter work Gao et al. (2020) the references are not human-written but unsupervisedly constructed from selected salient sentences. An overlap can be measured as well between summary and document text Shao et al. (2017).

Human evaluation of summary quality is far more meaningful and powerful than ROUGE, but it is far less reproducible. Summary quality estimation is a cognitively demanding and highly subjective task. Humans are also vulnerable to biases, such as the preference for phrases and sentences copied directly from the document text into summaries Ziegler et al. (2020). Improving human evaluation may require prompting labelers to pay higher attention Hardy et al. (2019), as well as splitting quality scores into multiple dimensions such as fluency, informativeness, and factual correctness Kryściński et al. (2019a, b); Fan et al. (2018). Even if humans can be trained to be more reliable, reproducible estimators of summary quality, they will forever remain a slow, expensive, limiting resource.

One possible route to a better automatic method for summary quality estimation is to train a model on document summaries annotated with human quality scores Louis and Nenkova (2009, 2013); Xenouleas et al. (2019). Such a model could be used to evaluate summaries without further human involvement. But even if such a model could achieve high agreement with human labelers, its performance would only be as objective and reproducible as the summary quality scores generated by one particular group of humans on a particular group of documents. Such a model may not generalize beyond the domain and style of the training samples unless they are a massive, representative sample of all documents of interest.

A more fundamental approach to the problem is to estimate how ”helpful” a summary is for the task of understanding a text. For example this might be achieved through a series of question-answers Eyal et al. (2019); Chen et al. (2018); Scialom et al. (2019). However, with this approach one must choose from a vast set of questions one might ask of a text, presupposing knowledge of the document itself and seriously limiting its reproducibility.

In the following section we suggest a new approach that is fundamentally justifiable as an estimator of summary quality, as well as being conceptually simple and reproducible.

As part of the validation of these new measures we performed experiments to determine how a substitution of an obviously bad summary affects the measure. One example is a summary generated by selecting random words from the text. The random words summary is generated with the same length as the original summary. Our original summaries are generated for randomly selected daily news by three different methods: by Microsoft’s abstractive UniML model Dong et al. (2019), by semi-abstractive summarization model (based on Vasilyev et al. (2019)), and by extractive LexRank model (based on Erkan and Radev (2004)). The summaries generated by these models are not flawless and vary widely in overall quality when evaluated by human labelers.

In another validation experiment, we generate a ”random sentences summary”, which is constructed from the sentences of a document. For this example, we apply BLANC-help with the ”no-copy-pair” guard introduced above. But we use the second version of the guard rule, because it is less exclusive of text sentences overall, and we also compensate for the length of the summary by replacing the copy-sentence of the summary with another sentence, rather than simply removing the copy-sentence.

BLANC-help results for both examples (in comparison to the measure of the original summaries) are shown in Figure 5.

We can see that the BLANC value for the real generated summary is almost always higher than the value for the random-sentences summary. This confirms that the measure takes into account the context as well as the informativeness of the summary to assess the quality.

Selecting only summaries with exactly three sentences, we can observe how BLANC-help deteriorates if we spoil some of the sentences of the summary. We replace one, two or all three sentences with random words, keeping the same length of the resulting randomized summary as the original summary. We also take care to run on each possible choice of replacement sentences twice, and average the resulting BLANC-help. The result is shown up in Figure 6.

The BLANC measures do not require any ”gold-labeled” data: No human-written summaries nor human-annotated quality scores are needed. Theoretically, the measures should reflect how fluent, informative, and factually correct a summary is, simply because only fluent, informative, correct summaries are helpful to the underlying language model. We now turn to the question of whether the BLANC measures correlate with summary quality scores assigned by human readers.

Human scoring is fallible; a correlation with human scores should not be considered as a full validation of our measures, but rather as an independent confirmation that the measures are sensible.

For purposes unrelated to this study, we have undertaken a series of human evaluations of many generated summaries of approximately similar length. As mentioned in the previous section, the summaries were generated by Microsoft’s abstractive UniML model Dong et al. (2019), by semi-abstractive model Vasilyev et al. (2019), and by extractive LexRank model Erkan and Radev (2004). The summaries from the latter two sources were ”equalized” in length to the UniML, so that at least on average the summaries from all three generation sources would be equal, and also so that most summaries would not differ significantly in length. Altogether, we assembled 555 summary-text pairs for human scoring, with the texts taken from the CNN / Daily Mail dataset Hermann et al. (2015).

We hired 10 annotators through Odetta.ai and trained them to assess the overall quality of each summary on a 5-point scale: 0 = VERY BAD, 1 = BAD, 2 = OK, 3 = GOOD or 4 = VERY GOOD. The annotators worked independently from each other and had access to only one summary-text pair at a time. The task was performed through the online text annotation tool LightTag (lighttag.io).

The values of the correlations are illustrated in Figure 7.

The green step-function shows the value of correlation of an annotator score (with Id ranging from 1 to 10) with the averaged score of the 9 other annotators. The number of samples used for the correlation is 555 - the summaries generated by the three models. The red and blue lines show correlations of BLANC-help and rouge correspondingly with the averaged score of all 10 annotators. The rouge here is calculated using the google-research package (github.com/google-research/google-research/tree/master/rouge) as F1 value of ”rougeL” (lower blue line on the plot) and F1 value of ”rougeLsum” (upper blue line). The latter is the ’summary-level LCS’, with summaries split to sentences and using a union longest common subsequence  Lin (2004)

The yellow line in the figure shows how a simplest combination of BLANC-help and ROUGE correlates with the annotators. The ”BLANC-help + ROUGE-Lsum” is literally a simple sum of BLANC-help and the ROUGE-Lsum. As usual a blending of two different models produces better results, though it is not our purpose here to fit human scores, and we do not fit the weights in the sum. (For example, using a $score=3*blanc_{help}+rouge_{Lsum}$ with the weight $3$ for BLANC-help would increase the correlation with human scores by 1%).

All shown correlations have p-values of order $10^{-6}$ and lower. We observe that both BLANC-help and ROUGE correlate with annotators as good as or better than about 30% of annotators.

In Figure 8 we present correlations with human scores on summaries generated for 100 typical daily news documents. The summaries were generated by the same three models; there were 300 summary-text pairs for scoring, again by 10 annotators.

Since there are no ”gold-labeled” summaries for these news documents, there is no ROUGE score in the figure.

As we see in all these examples, the human-human agreement is not impressive. We have observed from yet another evaluation dataset that if the texts and the generated summaries are challenging with very low inter-annotator agreement, the correlation of our measure with human scores is similarly diminished (with borderline p-values).

The values assigned to humans scores (0,1,2,3,4) are not a perfect translation of the human perception of the corresponding labels (”very bad”, ”bad”, ”OK”, ”good”, ”very good”). From multiple evaluations unrelated to this study we know that when an evaluation is repeated, human annotators are far more likely to substitute ”OK” and ”good” with each other than other tags. When we obtain an averaged human score, a weighting of the values $(0,1,2,3,4)$ with $weights=(3.0,3.0,1.0,1.0,2.0)$ may be more inline with human perception, but the changes to results we presented here are not essential, of order 1%.

BERTScore Zhang et al. (2020), similarly to ROUGE, requires reference summaries, but is using overlaps of BERT embeddings rather than strings. In Figure 7 the BERTScore F1 would be at 0.35 - close to BLANC, BERTScore Precision at 0.16, and BERTScore Recall at impressive 0.47 (calculated using python package bert-score).

A few simple observations may serve as an evidence that our measure deals with the length of a summary more reasonably than either humans or ROUGE. In Table 1 we show the correlations with the summary length and with the compression factor, which is defined as the ratio of summary length to document text length. The length here is the number of characters.

The table is based on the same data as Figure 7. The table shows that similarly to humans, our measure is helped by longer summaries in general. But unlike humans, it is much more sensitive to a summary’s compression factor. A disregard for the compression factor by humans may be caused by the anchoring effect.

Table 2 gives similar insight for very different kind of documents - random daily news, same as were used for Figure 8.

Whenever we used BLANC or human annotators for comparison of quality of summaries generated by different models of by different versions of a model, we generated summaries on average of the same length. It is clear that both humans and BLANC will estimate longer summary better, at least when it is a single score of overall summary quality. If the length of individual summary has to be excluded as a factor, the BLANC score should be normalized by the compression $C$. A longer summary adds proportionally more help, while a longer text adds proportionally more tokens for masking.

In Table 3 we show comparison of BLANC with negated Jensen-Shannon divergence (JS) which is a no-references measure showed up as the strongest in Louis and Nenkova (2009). The JS is a mean of text-summary and summary-text Kullback-Leibler divergences. For a purely statistical measure which we would assume misses a lot of semantics, JS works surprisingly well on CNN / Daily Mail news examples. The modest performance at first row by both measures can be explained by high variety of in styles of the summaries, which affects both the human scoring and the measures. On human-only summaries JS is still better than BLANC. In order to confirm that BLANC grasps more semantics, we considered three subsets of summaries that might have less signal from pure statistics. The summaries of similar length, close to peak of the distribution, is one example; summaries with low human scores is another one. More important example is the highly compressed summaries, with the ratio of the summary length to the text length $<0.05$. In this case JS correlation value would be $0.12$, but p-value=$0.15$ is too high. Following Louis and Nenkova (2009), the JS was calculated with filtering stop words and with stemming.

Simple correlation with a consensus score of annotators is not an easy criterion for judging the usefulness of the measure. When annotators are tasked with scoring several different qualities of a summary, their final score for the overall quality should be more grounded, because more attention has been spent on the summary and the text. In Figure 9 we show values of correlations obtained from such evaluation.

The data used here are the same as the data for Figure 8: summaries generated on randomly selected daily news documents. For this illustration, however, we split our 10 annotators into a small group of 3 and an ”others” group of the remaining 7. There are 120 ways to chose the split (on the X-axis). The circle markers show human-human correlation, i.e. the correlation between the average score of the small group and the average score of the ”others” group. The plus markers show BLANC-human correlation, i.e. a correlation of the BLANC with the ”others” group of annotators. Hence we see how well the BLANC measure performs against the team of 3 annotators in correlating with the ”others”. For simplicity of the presentation, each type of correlation was sorted independently. If a correlation is unreliable (p-value $>0.05$) then the marker is not shown.

We see that BLANC can be competitive to a team of three human annotators on all summary qualities, especially on the ’overall’ and fluency.

In this paper we present BLANC, a new family of objective and reproducible measures of summary quality. BLANC does not require human-written reference summaries; it is based on how helpful the summary for understanding the document.

By comparison, it is difficult to suspend disbelief when considering a method like ROUGE that does not inspect the document itself when estimating the quality of a summary. It is notable that ROUGE scores are often cited even for headline generation Ayana et al. (2016); Kiyono et al. (2017); Xu and Fung (2019); Gu et al. (2020) where it is hard to imagine that any single headline could be regarded as the best possible headline for a document.

One may argue that ROUGE requires less processing power, unless we recall that applying it requires the processing power of a human who must write the reference summary for ROUGE. In future research we will consider variations of BLANC and, for convenience, provide a public package blanc.

We thank Charlene Chambliss (Primer) for help in preparing the design of human evaluations, and Rosanne Liu (UberAI), Nina Lopatina (In-Q-Tel) and anonymous reviewers for review of the paper and valuable feedback.