Corpora Evaluation and System Bias Detection
in Multi-document Summarization

Multi-document summarization (MDS) deals with compressing more than one document into a textual summary. It has a wide range of applications – gaining insights from tweets related to similar hashtags, understanding product features amongst e-commerce reviews, summarizing live blogs related to an ongoing match, etc. Most studies on MDS were performed during the DUC¹¹1https://duc.nist.gov/ and TAC²²2http://tac.nist.gov challenges starting in the early 2000s. Each version of the challenges released a new dataset. Most of the MDS systems submitted to these challenges were unsupervised and extractive in nature. Gradually, the data released in these challenges became the de facto for MDS. These datasets were manually curated and had less than a hundred instances each. The recent development of deep neural architecture has led to a significant increase in the number of supervised document summarization systems. Large labeled corpora which are mostly crowd-sourced have been introduced to meet the training requirements of the supervised systems. However, the crowd-sourced corpora widely differ in quality based on factors like genre, size of the community, presence of moderation in the community, etc. This is further aggravated by the complexity of the task, the hardness of accumulating labeled data, or more so in the definition of what constitutes a multi-document summary.

Recently, a few large datasets for MDS have been introduced (Fabbri et al., 2019; Chowdhury and Chakraborty, 2019a). However, there has been no study to measure the relative complexity of these datasets. We observe that existing MDS systems behave differently on different corpora. For example, a system achieving state-of-the-art performance on one corpus fails to achieve reasonable performance on another. Although the ROUGE points of MDS systems are increasing day-by-day, manual inspection reveals an increased presence of bias in generated summaries. New systems are being introduced and evaluated on a few selected corpora, leading to difficulty in understanding whether the bias is introduced by the system or it is present in the corpus used for training.

Our research questions are as follows:
Q1. How should one model the quality of a MDS corpus as a function of its intrinsic properties?
Q2. Why do the ROUGE-based ranks of different MDS systems differ across different corpora? How should an MDS system which intends to achieve high ROUGE scores across all corpora, look like?
Q3. Why do systems show bias on different metrics, and which other system and corpus attributes are the reason behind it?
Q4. Is the task of MDS almost solved, or is there still scope for improvement?
We study five MDS corpora – DUC (DUC, 2002), TAC (TAC, 2008), Opinosis (Ganesan et al., 2010), Multinews (Fabbri et al., 2019), and CQASumm (Chowdhury and Chakraborty, 2019b). We consider eight popular summarization systems – LexRank (Erkan and Radev, 2004), TextRank (Mihalcea and Tarau, 2004), MMR (Carbinell and Goldstein, 2017), ICSISumm (Gillick et al., 2008), PG (See et al., 2017), PG-MMR (Lebanoff et al., 2018), Hi-Map (Fabbri et al., 2019), and CopyTransformer (Gehrmann et al., 2018).

Our major contributions are four-fold:
$\bullet$ We propose a suite of metrics to model the quality of an MDS corpus in terms of – Abstractness, Inter Document Similarity (IDS), Redundancy, Pyramid Score, Layout Bias and Inverse-Pyramid Score.
$\bullet$ We develop an interactive web portal for imminent corpora to be uploaded and evaluated based on our proposed metrics.
$\bullet$ We explore different biases that the MDS systems exhibit over different corpora and provide insight into properties that a universal MDS system should display to achieve reasonable performance on all types of corpora.
$\bullet$ We look into metrics to capture bias shown by MDS systems and explore the extent to which corpus properties influence them.

To the best of our knowledge, the current study is the first of its kind.

Throughout the paper, we use the term candidate documents for the documents participating in summarization, and the term reference to indicate the ground-truth summary.

Oracle summary is the extractive set of sentences selected from the candidate documents, exhibiting maximum ROUGE-N score w.r.t. the reference summary. It is an NP-hard problem Hirao et al. (2017), and approximate solutions can be found greedily or using ILP solvers.

Here, we briefly introduce a suite of corpus and system metrics proposed by us to better understand the MDS task. These metrics are further explained in detail in Supplementary.

$\bullet$ News derived corpora show a strong layout bias where significant reference information is contained in the introductory sentences of the candidate documents (Fig. 2).
$\bullet$ Different MDS corpora vary in compression factors with DUC at $56.55$, TAC at $54.68$, Multinews at $8.18$ and CQASumm at $5.65$. A high compression score indicates an attempt to pack candidate documents to a shorter reference summary.
$\bullet$ There has been a shift in the size and abstractness of reference summaries in MDS corpora over time – while DUC and TAC were small in size and mostly extractive ($11\%$ novel unigrams); crowd-sourced corpora like CQASumm are large enough to train neural models and highly abstractive ($41.4\%$ novel unigrams).
$\bullet$ Candidate documents in Opinosis, TAC and DUC feature a high degree of redundant information as compared to Opinosis and CQASumm, with instances of the former revolving around a single key entity while that of the latter tending to show more topical versatility.
$\bullet$ MDS corpora present a variation in inter-document content overlap as well: while Multinews shows the highest degree of overlap, CQASumm shows the least and the rest of the corpora show moderate overlap (see Fig. 1).
$\bullet$ Pyramid Score, the metric which evaluates if the important and redundant SCUs⁴ from the candidate documents have been elected to be part of the reference summary, shows considerably positive values for DUC, TAC and Multinews as compared to crowdsourced corpora like CQASumm (Fig. 3.b).
$\bullet$ Inverse-Pyramid Score, the metric which evaluates how well SCUs⁴ of the reference summary are distributed amongst candidate documents, also shows better performance on human-annotated corpora compared to crowd-sourced ones (Fig. 3(b)).
$\bullet$ A comparison amongst corpus metrics presents a strong positive correlation between IDS and Pyramid Score (Pearson’s $\rho$ = 0.8296) and a strong negative correlation between the metrics of Redundancy and IDS (Pearson’s $\rho$ = -0.8454).

$\bullet$ MDS systems under consideration are ranked differently in terms of ROUGE on different corpora; leading to a dilemma whether to declare a system superior to others without testing on all types of datasets (Fig. 4(c)) and Table 2).
$\bullet$ MDS systems under consideration outperform abstractive summarization systems by up to $10\%$ on ROUGE-1 and up to $30\%$ on F1 Scores, showing contradictory behavior in comparison to single-document summarization systems where state-of-the-art abstractive systems are known to outperform the former (Figs. 4(b)-(c)).
$\bullet$ The best summarization system on each corpus obtains a score $39.6\%,47.8\%,75.02\%,54.5\%,49.9\%$ of the oracle upper bound on DUC, TAC, Opinosis, Multinews and CQASumm respectively, indicating that summarization on Opinosis and Multinews is a partially solved problem, while DUC, TAC and CQASumm exhibit considerable scope for improvement (Fig. 4(c)).
$\bullet$ Hi-Map and CopyTransformer generate more abstract summaries ($17.5\%$ and $16\%$ novel unigrams respectively) in comparison to PG and PG-MMR (Fig. 4(a)).
$\bullet$ Averaging over systems and comparing corpora, we notice that Multinews and CQASumm achieve the highest abstractness ($27\%$ and $7\%$ respectively), which might be a result of these two corpora having the most abstract reference summaries (Fig. 4(a) and (Table 2)).
$\bullet$ Abstractive systems exhibit a $55\%$ shift in importance between the first and the second segments of generated summaries, whereas extractive systems show an average shift of only $40\%$, implying that abstractive systems have a stronger tendency to display layout bias (Fig. 2(b) and Fig. 2(c)).
$\bullet$ While DUC, TAC and Opinosis summaries generated from PG trained models exhibit lower novel unigrams formation, the same for CopyTransformer and Hi-Map on DUC, TAC and Opinosis shows a higher unigram formation on average (Fig. 4(a)).
$\bullet$ In terms of Inter Document Distribution, LexRank summary for TAC and CQASumm shows more variance across documents compared to DUC, Opinosis and Multinews. TextRank summary on DUC, TAC and CQASumm, MMR summary on DUC, and Hi-Map summary on CQASumm show higher variances as well. Systems such as PG, PG-MMR and CopyTransformer show minimal deviation in the document participation across corpora (Table 2).
$\bullet$ In terms of Topic Coverage, extractive systems show better coverage than abstractive systems (Table 2), which might be a result of extractive systems being based on sentence similarity algorithms which find important sentences, reduce redundancy and increase the spread of information from different segments of the candidate document. (Fig. 5).

Previous attempts to evaluate the quality of the benchmark summarization corpora are few in number and mostly from the time when corpora were manually accumulated. Hirao et al. (2004) primarily used the intrinsic metrics of precision and recall to evaluate corpus quality. In addition, the authors proposed an extrinsic metric, called ‘Pseudo Question Answering’. This metric evaluates whether a summary has an answer to a question that is otherwise answerable by reading the documents or not. Although effective, the cost of such an evaluation is enormous and is not scalable to modern day corpora sizes. For such corpora where multiple references are available, Benikova et al. (2016) used an inter-annotator agreement to model the quality of the corpora. They also used non-redundancy, focus, structure, referential clarity, readability, coherence, length, grammaticality, spelling, layout, and overall quality as quantitative features for an MDS corpus. Recently, Chowdhury et al. (2020) proposed an MDS system that used the baseline PG model along with Hierarchical structural attention to take into account long-term dependencies for superior results compared to baseline models.

There have been a series of very recent studies that look into how to strengthen the definition and discover system biases in single-document summarization. Very recently, Jung et al. (2019) studied how position, diversity and importance are significant metrics in analyzing the toughness of single-document summarization corpora. Another recent work (Kryscinski et al., 2019) extensively studied the Layout Bias in news datasets that most single-document summarization systems seem to exploit. Two seminal works, namely Peyrard (2019a) and Peyrard (2019b), exploited the theoretical complexity of summarization on the ground of importance, analyzing in-depth what makes for a good summary. Peyrard (2019a) mathematically modeled the previously intuitive concepts of Redundancy, Relevance and Informativeness to define importance in single-document summarization. Grusky et al. (2018) proposed a new single-document summarization corpus and quantified how it compares to other datasets in terms of diversity and difficulty of the data. They introduced metrics such as extractive fragment density and extractive fragment coverage to plot the quality of SDS corpus. To the best of our knowledge, no comparative work exists for either corpora or systems in MDS, and the current paper is the first in this direction.

In this paper, we aimed to study the heterogeneous task of multi-document summarization. We analyzed interactions between widely used corpora and several state-of-the-art systems to arrive at a line of conclusions. We defined MDS as a mapping from a set of non-independent candidate documents to a synopsis that covers important and redundant content present in the source. We proposed intrinsic metrics to model the quality of an MDS corpus and introduced a framework for future researches to consider while proposing a new corpus.

We analyzed how ROUGE ranks of different systems vary differently on different corpora and described what a system that achieves reasonable performance on all corpora would look like. We evaluated how different systems exhibit bias and how their behavior is influenced by corpus properties. We also commented on the future scope for the task of MDS.

Future directions to take forward this work would include a causal analysis of how corpus bias is responsible for bias in model prediction across different corpora and systems. This might bring forward measures to de-bias NLP algorithms with/without de-biasing the corpora.

The work was partially supported by the Ramanujan Fellowship and DST (ECR/2017/00l691). T. Chakraborty would like to acknowledge the support of the Infosys Center for AI, IIIT-Delhi.