ACM - Attribute Conditioning for Abstractive Multi Document Summarization

Conditional language modeling has been approached both through applying global constraints on text generation, by applying control only at inference time and by directly optimizing through policy gradient methods. Ranzato et al. (2016), Holtzman et al. (2018), Dathathri et al. (2020). Alternative approaches include relying on predefined sets of control tokens or control codes as a form of a copy mechanism Luong et al. (2015). These approaches have been successful at steering language models towards specific features or as in Dathathri et al. (2020) with conditioning for the outputs to include words conditioned for through a simple bag of words attribute model. We build on these approaches to design the attribute conditioning module.

Abstractive summarization (AS) is a task in which a language model is trained to generate text that matches a pre-generated summary for a given article. AS has gone through several phases of which the pioneering work was carried out by Sutskever et al. (2014) where an Seq2Seq RNN Model was implemented to generate text. The Seq2Seq RNN Model inherently had multiple challenges such as altering factual details and redundancy. See et al. (2017) circumvented the issues by creating a pointer-generator model which keeps track of words and sequence in the original text and using them in the result hence ensuring the meaning of summary is in-line with the original text. This paper also included a coverage mechanism to keep of track of which parts of the original text have been summarized thus penalizing repetition. BART, the bi-directional auto-regressive transformer Lewis et al. (2019), built on this work by improving on the task of abstractive summarization by introducing arbitrary noise in the input text and training the model to reconstruct the original text.

Multi document summarization has evolved through four primary approaches since the task was first introduced. The first set of approaches focused on graph ranking based extractive methods through TextRank Mihalcea and Tarau (2004), LexRank Erkan and Radev (2004) and others. These approaches came before syntax and structure based compression methods which aimed to tackle issues of information redundancy and paraphrasing between multiple documents. Compression-based methods as shown in Li et al. (2014) and paraphrasing based were improved upon with the advent of neural seq2seq based abstractive methods in 2017. This allowed multi document summarization to further improve upon the work done with single document abstractive summarization through approaches such as pointer generator- maximal marignal relevance Lebanoff et al. (2018), T-DMCA Liu et al. (2018) the paper that also introduced the foundational WikiSum dataset and HierMMR Fabbri et al. (2019) that introduced MultiNews. These approaches aimed to tackle information compression through maximal marginal relevance scores across documents and through attention based mechanisms.

Improvements upon those baseline models include further leveraging graph based approaches to pre-synthesize dependencies between the articles prior to multi document summarization as tackled in Li et al. (2020). Further work needs to be done to further exploit these graphical representations as Li et al. (2020) essentially works to establish baselines with tf-idf, cosine similarity and a graphical representation first described in Christensen et al. (2013). These papers primarily aim to address de-duplicating information and learning relationships between the different topics shared across documents however none of these architectures are built to deal with conflicting information.

We preprocess the inputs to the model through a graph conditional weighting layer. As shown in Figure 1, for each of the input paragraph vectors both the positional encoding layer and the attribute conditioning layer outputs are computed. The graph representation is then constructed by creating a matrix of values where each row and column represents a paragraph in one of the input documents. The baseline model for GraphSum Li et al. (2020) computed a similarity score between these paragraphs by using tf-idf. Our graph conditional weighting layer aims to improve this approach by adding the weighted sum of linearly transformed paragraph vectors with the product of the attribute conditioning module for each paragraph as represented by $\beta_{i}$ and $\beta_{j}$ incorporating similarity between the attribute scores for each paragraph.

Let $x_{i}^{l-1}$ represent the output of the graph encoding layer for paragraph $x_{i}$. Let $R_{ij}$ represent the pairwise relation bias of the weights of graph representation matrix $G$. Then graph informed self attention with conditional weighting can be defined as taking the product of the attribute conditioning scores between each pair of paragraphs with the weighted sum of the linearly transformed paragraph vectors. Thus the graph will learn stronger weights between paragraphs that are of the same polarity or sentiment when generating the output summary.

Attribute future discriminators approaches the problem with Bayesian factorization - effectively decoupling the pre-trained summarization model from the fine tuning classification model. This approach is described as using a future discriminator as it takes the sequence of text generated so far, appends to it the most likely next token predicted by the summarization model and then computes the likelihood that this sequence satisfies the desired classification attribute. In doing so it increases the likelihood of both generating text that satisfies the summarization model with a high likelihood as well as staying consistent with the desired text modeling attributes.

We can model the standard summarization task as

Let $X$ refer to a set of input documents $d_{1},..d_{n}$ pertaining to a specific MDS summary $s_{i}$. Let each document $d_{i}={x_{i1},..x_{in}}$ where $x_{ij}$ refers to token $j$ in document $i$. Prior to the MDS task, all input documents are appended, thus absolving the need to index each token with its corresponding document. Lastly, let $a$ refer to the conditioning attribute. When conditioning on a certain attribute, we model the task as

We rely on the following factorization to decouple the summarization module from the attribute conditioning model:

This approach allows us to train the conditional model and the summarization model separately and in a composable manner to achieve the desired output conditioning as shown in figure 2.

As both the summarization model and the classification models are pretrained, during evaluation each of the top k words selected by the decoder logits in the summarization model is added to the best sequence so far and then passed into the attribute classification model. The output probabilities of both models are then combined in selecting the next best word generated by the combined model. This allows for a high degree of composability as each of the attribute models can be layered on top of each other and added to the final logits with different weights.

Conditional training combines the probability distribution over the next words generated by the decoder with the polarity scores as determined by the attribute classification model. In contrast to other models such as Dathathri et al. (2020), this allows the base model to be conditioned to output based on the desired attribute similar to the approaches taken to train class conditional language models.

For each model architecture, the decoder logits are multiplied with the attribute conditioning model’s logits during training. This trains the abstractive multi document summarization model to output text conditioned on that particular attribute. Thus for each of the input documents, information that aligns with the selected attribute is proportionally more likely to be generated. The same classifier models and baseline MDS models were used to evaluate this approach as in the previous method. We show that this approach can be applied to any baseline MDS architecture in the follow up ablations.

In order to determine conflicting information in text, we first train attribute classification models on two different datasets AllTheNews Thompson (2020) and the MPQA Opinion Corpus dataset Deng and Wiebe (2015) in order to determine the sentiment and polarity scores of input text. These attributes are strongly correlated through human analysis with differing viewpoints on text. In order to create a dataset most similar to the input sequences that would be passed through ACM, the input text in the dataset was augmented to include all prefix length subsequences with the same label. This corresponds to the prefix length subsequences that will be evaluated for the above approaches.

AllTheNews dataset consists of 2.7 million articles from 26 different publications ranging from January 2016 to April 2020 in EnglishThompson (2020). This dataset was augmented with polarity labels according to the news source label. The MPQA Opinion Corpus was chosen over other sentiment analysis datasets as it consists of articles pulled from news articles on a broad range of news sources and is consistent with the approach used in Ramsauer and Kruschwitz (2021).

The MPQA Opinion Corpus is copyrighted by MITRE Corporation and consists of 700 English articles from a range of English language sources. The dataset provides labeled entity spans with the corresponding positive or negative annotations. Deng and Wiebe (2015)

MultiNews Fabbri et al. (2019) consists of a varied set of news articles spanning over 1500 sites and their corresponding human written summaries. The average length of the source documents is 2100 tokens which allows the Bart+Longformer model trained with input length 4096 tokens to consume multiple concatenated articles at a time. For all models, each of the input documents are concatenated together and fed into into the model in batches of the maximum token size. XLNet was used as the primary architecture for each of the classification modules. Each of the attribute classification modules were trained with a 80-10-10 train, val, test split using the entire corpus of data. Classification results were averaged over three runs.

ACM was trained using 8 transformer encoder heads and 6 graph decoding layers. Beam size was set to 5 with length penalty factor 0.6 trained with gradient accumulation every 4 steps. Additionally we ran a hyper parameter search to determine the ideal weighting terms for the attribute conditioning module at each phase of summarization. For the ablation studies, BART and BART+Longformer self attention models were trained using the same set of hyperparameters used to train the baseline models. All models were trained using 2 NVIDIA K-90 GPUs over the span of  1000 hours. BART, BART+Longformer, XLNet, and GraphSum each have 406M, 555M, 110M, and 121M parameters. Due to the composable attribute classifier modules in ACM, the number of parameters in ACM ranges from 130M to 240M including one attribute classifier. BART was trained with 8 transformer encoder heads and 6 decoder layers with gradient accumulation every 4 steps. The BART model was trained with max token size of 512 and the Bart+Longformer self attention model was trained with max token size of 4096. GraphSum Li et al. (2020) was trained using the same configurations present in the original paper in order to reproduce results with 8 transformer encoder heads and 6 graph decoding layers. To maintain consistency with ACM, beam size was set to 5 with length penalty factor 0.6 trained with gradient accumulation every 4 steps. We performed hyper parameter search to determine the optimal weights for each of the modules. Let $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ represent the weighting terms for each module, attribute conditioning with future discriminators, graph conditional weighting and conditional training, respectively. We found that $\alpha_{1}=0.22$, $\alpha_{2}=0.4$ and $\alpha_{3}=0.01$ showed the strongest ROUGE results. We evaluate our model on both ROUGE scores as well as perform human annotations to evaluate output summaries for fluency, informativeness, and consistency Lin and Och (2004). ROUGE score is a standard metric for measuring summarization quality - ROUGE-1 measures overlap of unigrams, ROUGE-2, the overlap of bigrams and ROUGE-L, the longest common sub-sequence between the generated summaries and the gold standard references. ROUGE-L is often used as a measure of assessing fluency. Li et al. (2020). In addition we perform a series of ablation studies for each technique presented in order to assess the contribution of each to the final result.

We present an example summary highlighting the difference between the summaries generated with ACM versus the highest performing GraphSum baseline model in figure 2. These examples were provided for human annotation as is including the input articles as well as the GraphSum baseline and the ACM model summary. We primarily compare ACM against other strong baseline model architectures including BART Lewis et al. (2019), BART+Longformer attention Beltagy et al. (2020) and GraphSum Li et al. (2020).

We evaluate each of these methods on the MultiNews dataset. MultiNews and WikiSum are the most commonly used datasets for multi document summarization. Since this paper is primarily concerned with addressing conflicting opinions, an entirely objective dataset such as WikiSum would not be ideal. Currently, there are no other large scale multi document summarization datasets from other domains. Table 3 shows the ROUGUE scores for each of the models. The first block of results represents the baseline models’ ROUGE scores followed by the results from ACM. Overall performance shows that the sentiment attribute module and the polarity attribute module perform on par with polarity performing slightly better. This trend holds across the other approaches as well. ACM outperformed each of the baselines as well confirming our hypothesis that learning the relationships between the input data improves attribute consistency in the final output summary.

In addition to the automatic evaluation reported above, we also perform a human evaluation study with 5 evaluators on Amazon Mechanical Turk. We randomly select 182 input test summaries from the MultiNews dataset and the corresponding output summaries generated by ACM. In order to assess inter-annotator agreement, a portion of the articles were held out for annotation from each of the annotators. This analysis showed a 78% agreement in scores amongst the annotators. Additionally, annotators were required to be fluent in the language. In order to assess the quality of the model irrespective of the classifier model chosen, we randomly selected output summaries between the two classifiers. Since it is the previous state of the art, we use GraphSum as the baseline model.

Annotators assess the overall quality of the summaries based on three different criteria: (1) informativeness, (2) fluency and (3) repetitiveness. Informativeness is defined as the number of unique facts / pieces of information present in the summary. Fluency is defined as the readability of the text accounting for good grammar, noun phrases and logical flow of information. Repetitive content comes from repeated words, phrases, or ideas throughout the output summary. Each of these attributes were assessed on a scale of 1 to 5. According to this scale, a high score is preferable for fluency and informativeness and a lower score is preferable for repetitiveness. A summary of the results from the study can be found in table 2.

In order to determine the contribution of each method used within the ACM model, we performed additional ablations and model analysis. The key ablation studies included evaluating each approach on one of the baseline models, BART, BART + Longformer and GraphSum. We analyzed the results here both in terms of achieving higher ROUGE scores as well as maintaining information consistency.

Table 5 presents the ROUGE scores for each of the approaches. We note that there are improvements from each approach individually with respect to the ROUGE score with attribute future discriminators performing marginally better than the other approaches. BART+Longformer achieves overall better performance on ROUGE scores as compared to BART primarily due to the longer input sequence lengths passed in. In addition to evaluating on ROUGE, an analysis is done on how well each approach was able to preserve the overall attribute conditioning.

Additionally we conducted a qualitative analysis of how well each approach was able to condition for polarity and sentiment in the output summaries by evaluating the summaries through the trained attribute conditioning module as shown in Table 6. This shows strong out of domain analysis results as the attribute conditioning module for polarity was trained on a different dataset, AllTheNews, and evaluated on the MultiNews dataset for MDS. This ablation study shows that MDS with future discriminators is the strongest attribute conditioning model. Since the sentiment and polarity of articles as determined by the XlNet classifiers are used to compute the opinion of an article, we used these models to analyze the MultiNews dataset. One possible risk that arises through these approaches is potentially increasing hallucinations given over conditioning using the attribute conditioning module. In this work, this risk was mitigated by fine tuning the weight of the attribute conditioning module to provide a balance between maximizing ROUGE while conditioning for a particular attribute.