Controllable Topic-Focused Abstractive Summarization

One of the early deep learning architectures that was shown to be effective in abstractive summarization was the Attention-based Encoder-Decoder (Nallapati et al., 2016) proposed by Bahdanau et al. (2014) also known as vanilla seq2seq. Attention mechanisms have been shown to enhance the basic encoder-decoder model (Bahdanau et al., 2014). The main bottleneck of the basic encoder-decoder architecture was its fixed-sized representation (“thought vector”), which could not capture all the relevant information of the input sequence as the model or input scales up. However, the attention mechanism that relies on the notion that, at each generation step, only parts of the input are relevant has shown to be an effective solution to this problem. Following the vanilla seq2seq scheme, models such as the Pointer Generator Network (PGN) (Vinyals et al., 2015) with copy mechanism (See et al., 2017) emerged in an effort to solve the challenge of out-of-vocabulary words.

Later, Attention was shown to be effective in Transformer architectures (Vaswani et al., 2017) as well. Several large pre-trained conditional language models for text generation were introduced. One such model is T5 Raffel et al. (2019) that can be fine-tuned for different seq2seq tasks. Another model in this category is BART (Lewis et al., 2020), a denoising autoencoder for pre-training seq2seq models. BART is trained by “corrupting text with an arbitrary noising function and learning a model to reconstruct the original text” (Lewis et al., 2020). Our proposed model, Conformer, is initialized by a pre-trained BART model. In order to demonstrate the ease of integration of our proposed topical cross attention mechanism into arbitrary seq2seq Transformer models without adding new model parameters, we also plug this component into a T5 model and show the efficacy of this approach.

Previous work has studied various control mechanisms to enforce a specific style (Blinova et al., 2023), form, or use of content in a generated summary. (Gehrmann et al., 2018) proposed to use a content selector to select phrases from a source document that should be part of a generated summary. Likewise, (Li et al., 2018) introduce an information selection layer to explicitly model the information selection process in abstractive summarization. Other work has studied guiding the generation process based on topics (Bahrainian et al., 2021b; Dathathri et al., 2020), aspects (Gu et al., 2022), entities (Narayan et al., 2021; Maddela et al., 2022), word relations such as triplets (Dou et al., 2021) to shift the content of a generated summary toward a desired target form such as formal or informal text (Briakou et al., 2021), biased versus neutral (Pryzant et al., 2020), simplified (Cao et al., 2020), or even sentimental stance (Shen et al., 2017).

Here we study controlled text summarization using an information selection signal in the form of latent topics.

Previous work has utilized topic information in seq2seq problems such as neural response generation (Bahrainian et al., 2021b; Xing et al., 2017). The work of Xing et al. (2017) uses a topic model named Twitter LDA to respond to messages, a different objective than our work. (Sim et al., 2022) studied topic preservation in multi-document summarization. They conclude, that many summaries appear less informative due to a lack of topic coherence between them and their source articles. (Cui and Hu, 2021) incorporates a topic-based approach in a multi-document summary setting to jointly discover latent topics that can act as a semantic bridge across documents and provide global information to guide the summary generation. (Liu and Yang, 2021) use a classifier to distill the training of a summarization model with respect to topical consistency between an input document and a generated summary. They observe that this brings improvements to standard abstractive summarization.

Another recent piece of related work integrates topic information into Transformer models for abstractive summarization (Wang et al., 2020). The authors utilize the Poisson factor analysis (Zhou et al., 2012) topic model. They add three new modules to a Transformer architecture at the cost of additional parameters. These three modules are semantic-informed attention, topic embedding with masked attention, and document-related modulation, which together integrate topic information into a Transformer architecture. Our work differs from theirs in three crucial ways: (1) our design does not introduce any additional model parameters to the original Transformer model, making it more efficient. (2) No training from scratch is required for the models in our approach, and standard fine-tuning for several epochs brings a significant improvement in summarization performance. (3) Our approach is more generic in that it can be integrated into various families of seq2seq models with attention and is not limited to Transformers.

In summary, topic information has been used in previous neural models as an input, and Wang et al. (2018) argue that it results in the diversification of words appearing in summaries. However, the novelty of this paper lies in using this source of information to systematically influence the output summaries to conform to a specific topic distribution. This results in enhanced ROUGE scores for standard abstractive summarization while introducing a mechanism to control the topical distribution of generated summaries. Moreover, our approach is highly generic such that it can be easily integrated into existing seq2seq models at no additional computational cost (i.e., no new parameters are added to the Transformer model). At the same time, Conformer’s novel topic-guided generation produces highly faithful summaries.

Although we use LDA as our topic model of choice, any other topic model that can factorize a corpus of text into two matrices, namely, the topic-word matrix and the document-topic matrix, may be used as a replacement. This requirement matches the properties of most existing topic models.

In order to compute the topical attention weights, after training an LDA model on the training data, we map the target summary corresponding to each document to its LDA space. In other words, we compute the prevalence of all topics for each target summary, which results in a per-target summary probability distribution over all LDA topics. Furthermore, for each topic, LDA computes a probability distribution over words of the entire vocabulary $\mathcal{V}$, i.e., the simplex. Therefore, for a given input document $d$ we can calculate a topical word vector $\tau^{d}$ of dimension $\left|\mathcal{V}\right|$ considering all the words in that document, such that:

After having obtained the topic distribution of a target summary, we can now compute the probability scores of all tokens/words of the corresponding source document. In this way during training, all tokens/words from the source article will be scored according to the target topics that an abstractive summarization decoder needs to focus on when generating output summaries. That also means that the topic-words distribution is sorted and trimmed to match the source documents dimensions. In other words, each word from the source article is associated with a corresponding topic-words probability score computed using the LDA model. For stopwords or rare words that do not appear in the topic model vocabulary, a very small positive probability of ‘$10^{-9}$’ is assigned.

The advantage of using topics of target summaries to guide the topical attention is that, if a word in the source document does not appear in any of the topics present in the target summary, this word will receive almost a ‘$0$’ probability in the topic-words probability vector described in Equation (1).

By relying on topic information derived from the target summaries, we achieve two goals: (1) This focuses the summary generation attention precisely on the target topics as indicated by the target summaries, making our model a suitable solution for both standard and controlled abstractive summarization. This will direct the model to assign very low probabilities to tokens/words of the source document outside the scope of the corresponding target summary’s topics. (2) According to previous research (Wang et al., 2018) the incorporation of topical information enables a language generation model to further diversify the vocabulary used in the generation process, thus generating a richer summary.

Due to the absence of target summary information at test time we train a feedforward neural network to map an input vector representing the frequency of words in an input article to its corresponding topic-words distribution computed by LDA defined in Equation 1. The feedforward network has one layer and with equal input and output size in the size of $\left|\mathcal{V}\right|$ optimizing the cross-entropy loss. The mapping of source article word frequencies to topic-words values is learned in the training phase when target summaries are available. Subsequently, the topic-words distribution computed by LDA is replaced by that of learned using the feedforward network both during training and testing to circumvent the absence of target summaries at test time. To elaborate, the feedforward network is trained based on LDA as described above, prior to fine-tuning the Summarizer model. From this point on $\tau^{d}$ refers to the topic-words distribution computed by the feedforward network.

This section discusses the integration of topical attention into the seq2seq Transformer architecture. Although we use a pre-trained BART model to initialize the Conformer, any summarization Transformer model may be used as well. In the experimental section, we demonstrate this on the example of a T5 model.

Once we have calculated the topic probabilities of an input source document according to the topics of its corresponding target summary, we end up with a vector of topic probability values from the feedforward network in the size of $\left|\mathcal{V}\right|$. Subsequently, we trim and reorder this vector to match the size and the order of the input document. Figure 2 shows the integration of topical attention into the Transformer architecture which is shared by both BART and T5 models.

To integrate this vector into a Transformer architecture, we insert the topic-words probability scores derived from the output of the feedforward network into the cross-attention modules that connect the encoder with the decoder. That is, for each word of the source article, we compute a topic probability score, indicating the word’s likelihood of appearing in the summary. Going back to the original Transformer paper (Vaswani et al., 2017) each encoder layer has keys, queries, and values within the same layer. However, in the cross-attention module, the keys and values are computed by the encoder, while the decoder computes the queries.

This is because cross-attention essentially bridges the encoder blocks and the decoder blocks. Figure 2 shows how the key and value matrices flow from the encoder side to the decoder.

While, for reasons of computational efficiency, the computations in a Transformer are performed at the matrix level, in the following, we present the steps of topical attention of the Conformer at the document level for illustrative clarity:

We remember from the original Transformer paper (Vaswani et al., 2017) that cross-attention between encoder and decoder is computed as:

The same equation holds in the case of cross-attention with the keys $K$ and values $V$ being outputs of the encoder while the queries $Q$ are computed on the decoder side. We observe from the equation that a dot product is computed between the keys and the queries. This dot product for a single source document and summary pair results in a single value indicating the importance of each key at its position given the query. Then this value is normalized, dividing it by $\sqrt{d_{K}}$ and then passing it to a softmax function. At this step, we compute an average between the result of the latter softmax function and the softmax of topic-words probabilities derived from Equation (1). We, therefore, define topical attention as:

Topical attention effectively reduces the importance of a word from the source document that is not covered in the topics discussed in the respective target summary by adding a small value or even ‘$0$’ to it and then dividing it by $2$. Thus, the resulting value will become too small to be the focus of the generation process. On the other hand, the probability of a word related to a topic discussed in the target summary can be potentially increased or remain unchanged. We conclude that as explained above Conformer obtains topic information from the feed-forward network component, and is trained in an end-to-end fashion.

We evaluate the effectiveness of all methods on three publicly available benchmark datasets.

NEWTS (Bahrainian et al., 2022) is a topical abstractive summarization dataset that provides two alternative human-written summaries of each source article, with different topical focus for a subset of the CNN/dailymail collection. This dataset contains 4,800 training examples and 1,200 test samples. We use this dataset for evaluating controlled topical summarization.

The CNN/DailyMail dataset (Hermann et al., 2015; Nallapati et al., 2016), contains news articles from the CNN and Daily Mail websites. The experiments reported in this paper are based on the non-anonymized version of the dataset, containing 287,226 pairs of training articles and summaries, 13,368 validation pairs, and 11,490 test pairs.

The XSum (Narayan et al., 2018) dataset is included to show the efficacy of our approach on shorter news articles. This dataset is a collection of online articles from the British Broadcasting Corporation (BBC) and their corresponding summaries. It contains 204,045 training samples, 11,332 validation samples, and 11,334 test samples.

For each dataset an LDA model is trained followed by a feedforward network reconstructing the topic-words vector. LDA returns $N$ probability distributions over the vocabulary representing the latent topics discussed in the respective training set. Since the actual number of underlying topics ($N$) is an unknown parameter in the LDA model, it needs to be estimated first. For this purpose, similar to the method proposed in (Griffiths and Steyvers, 2004; Bahrainian and Crestani, 2018), we perform a model selection process. It involves keeping the LDA parameters (commonly referred to as $\alpha$ and $\eta$) fixed while assigning several values of $N$ and running the LDA model for each value. We pick the model that minimizes the negative $\log P(V|N)$, where $V$ contains all the words in the vocabulary of all the documents in the training data. This process is repeated until we find the topic model with the optimal number of topics. The training of each LDA model on CNN/Dailymail takes nearly a day, so we could only repeat it for a limited number of $N$ values. concretely, we trained the LDA models with values $N$ ranging from $50$ to $300$ in increments of $50$, and the optimal value on the CNN/Dailymail and XSum datasets were found to be $250$ and $150$, respectively. Subsequently, the feed-forward network learns LDA output for each input document.

Tables 2 and 3 present the results of this experiment. For all models and metrics, we can note superior performance of the Conformer on both the CNN/Dailymail and the XSum dataset compared against original models.

Additionally, we observe that the relative performance improvement in all models against their corresponding original models is relatively higher in the case of the ROUGE 1 metric. One may interpret this observation such that the semantic similarity information obtained from the topic model forces a higher usage of individual topic-related words in the generated summaries. On the other hand, when it comes to ROUGE 2 and ROUGE L, since there are particular grammar (or word sequence) rules and patterns that a language model needs to enforce, the improvements are slightly smaller than those in ROUGE 1. (Dou et al., 2021) also states that generation guided by certain signals, e.g., in this case topicality, does improve the summarization performance.

It is worth noting that the demonstrated improvements in summarization performance come at no additional cost in terms of parameters added to the original models. This property of our scheme stands in stark contrast to other existing approaches such as (Wang et al., 2020) that require the addition of a significant number of model parameters. As an example, for the different Transformer models they study, they introduce an additional $5.69\%$ to $9.58\%$ of new parameters, which in absolute numbers amounts to 10.25 million and 38.91 million parameters according to their paper. Therefore, our approach is more generically integrated with Transformer-based models, and requires identical run time to the unmodified variants. Furthermore, our controlled generation approach is not language-specific and can be trained for any language.

Table 4 presents the results on the CNN/DailyMail and the XSum dataset. This table directly cites the results reported in the original papers, where available. We observe that our model performs superior to the latest Transformer models such as ProphetNet, the standard BART model and (Wang et al., 2020) on both datasets. At the same time, the top-performing approach of (Wang et al., 2020) which is also based on BART-large adds 38.91 million new parameters to the architecture while our approach adds no further parameters to the BART-large model. Comparing our results with theirs, we conclude that Conformer is highly effective to the point that the massive addition of hidden states in (Wang et al., 2020) does not lead to improvements over our approach. Moreover, in addition to the better computational efficiency of our model, our approach is more generic in that it can be integrated with various architectures as demonstrated in the case of T5 models. Finally, we observe that Conformer achieves superior performance over GSum (Dou et al., 2021) on XSUM dataset while matching GSum’s performance on the CNN/Dailymail dataset on abstractive summarization. This is while, the new BRIO model with its contrastive loss training outperforms our model. Studying the initialization of Conformer based on the BRIO model remains as future work.