Extractive Text Summarization Using
Generalized Additive Models with Interactions for Sentence Selection

Generalized Additive Models (GAMs) [10] are statistical approaches that have gained interest for their ability to be intelligible, appealing to their additive formulation and human intuition to provide interpretability in supervised problems.

These models have the form of Equation 1, working as middle ground between linear models (e.g., linear regression) and full-complexity models (e.g., ensembles of trees) [15], where the motivation could be, for example, obtaining predictors more accurate than the former while staying more intelligible than the latter, fitting individual features with non-linear functions that are combined additively [15]:

where $x_{i}$ is the $i$-th feature given a set $N$ of features, $f$ is called shape function, and $g$ is called link function.

On additive models, the contribution of each feature map towards final decisions can be seen more clearly than in dense ones. This property allows GAM’s learned shape functions to be visualized and their outputs to be investigated for individual or group of predictions, providing interpretability over the model and its decisions. Moreover, as GAMs can assume non-linear behaviour through their shape functions, they may fit a wider variety of problems when compared to linear models.

Recently, some works were focused on effectively building more powerful GAMs with the addition of modern machine learning strategies and supplementing the original univariate shape functions with a usual restricted set $K$ of pairwise interactions (bivariate shape functions), increasing the accuracy of the final model while maintaining some intelligibility, resulting in the following models:

where $f^{\prime}$ is a pairwise interaction.

In this work, two representatives of those algorithms are applied to the extractive summarization problem, i.e., EBM and GAMI-Net.

The preprocessing step is responsible for turning raw documents into sequences of sentences, capturing useful information for future feature extraction. The process starts by segmenting raw documents into text sentences, which are split in tokens. Then, the process follows by removing punctuation and stopwords, word tagging and stemming. After preprocessing, sentences correspond to lists of their respective words that are forwarded to the feature extraction step.

We perform most of the NLP preprocessing with the Python library spaCy [19], with the exception of stemming step which is done using the NLTK SnowballStemmer [3].

After sentences are preprocessed, six different features are extracted from sentences to become inputs vectors $x=\{x_{1},x_{2},...,x_{6}\}$ that are used to train and predict. The feature computations are formulated as described below:

In order to select sentences to produce summaries, as various works did in the past, we interpret the Extractive Summarization problem as a binary classification one, where the model is trained to decide between exclusion and inclusion of individual sentences in the summary.

The procedure consists in minimizing the binary loss in a supervised learning approach where the sentences’ feature vectors and the labels that classify whether they are present or not in their respective document summaries are used to train the model.

After training, we use the model’s ability to distinguish between “important” and “unimportant” sentences to, given an input document, score and select appropriate sentences among the others. The scoring process consists in obtaining the probability of a sentence being part of the summary given its feature vector $x$, for each of the sentences in the document. Then, similarly to [22, 12, 33], the document summary is obtained by ranking and selecting top sentences using that probability, respecting the length compression limit.

In this work, we compare EBM and GAMI-Net models with other approaches on two public text summarization datasets, namely CNN/Dailymail [11, 30] and Pubmed [4]. The CNN/Dailymail summarization dataset consists of pairs of news articles and their main highlights constituting 312K documents. This dataset have been widely adopted on recent ATS works, especially by Recurrent Neural Network and Transformer-based approaches. We use the non-anonymized version of this dataset [30].

The PubMed dataset is a collection of scientific papers totaling 133K documents in which the abstract section is used as the summary references. This dataset have been used for evaluating long document summarization approaches as both document and summaries are usually longer than in news datasets [33].

As mentioned in Section 3, our approach uses individual sentences as the input instances for training. Considering that initially both datasets only possess abstractive summaries, we needed extractive oracle labels to train our models. Recently, different authors obtained those labels using automatic heuristics while working with these datasets [22, 12, 14, 33], which we adopt here. For CNN/Dailymail, we generated labels using the scripts provided by [14]¹¹1https://github.com/nlpyang/BertSum, and, for Pubmed, we utilized labels extracted and made public by [33]²²2https://github.com/Wendy-Xiao/Extsumm_local_global_context. Moreover, we adopted random undersampling to handle label inbalance during the training step.

We compare EBM and GAMI-Net with some recent baselines, most of which produced by deep neural architectures, in the sense of outlining the summarization ability in contrast to these models, despite the notable differences in terms of interpretability. Tables 1 2 present the results and reporting authors in CNN/Dailymail and Pubmed datasets, respectively.

Additionally, we compare EBM and GAMI-Net models with other supervised machine learning classifiers, i.e., Logistic Regression (LR), Random Forest (RF) and XGBoost, using the exact same fashion and features described in Section 3. Each technique was trained and tested ten times and the average scores are considered for comparison purposes.

Overall approaches are evaluated using the ROUGE score metric [13] considering its broad adoption for extractive summarization systems. Also, we evaluate the sentence selection ability of the supervised classifiers computing summary F1 scores based on the oracle labels. Moreover, Lead baseline correspond to selecting the first sentences present in the documents as the summaries (respecting each dataset summary-length limit) and Oracle denote the pre-obtained oracle labels’ scores. Our ROUGE scores were obtained using pyrouge³³3https://pypi.org/project/pyrouge/, a python wrapper of the original ROUGE-1.5.5 scripts and, concerning summary sizes, CNN/Dailymail summaries were limited to three sentences [36] while Pubmed summaries were limited to 200 words [33].

As denoted in Tables 1 and 2, EBM and GAMI-Net achieved similar results on both datasets. Considering ROUGE scores, GAMI-Net is ahead on CNN/Dailymail while EBM is superior on Pubmed, for less than 0.1 for each variant.

As shown in Table 1, comparing to other approaches, EBM and GAMI-Net models were able to compete with SummaRunner [22] and Pointer-Generator [30] networks – two of the earliest deep summarization architectures proposed in the past, surpassing the former concerning R-L and both of them considering R-2 on the CNN/Dailymail dataset. On the other hand, they could not overcome BART+RD [32] and MatchSum [36] (Transformer-based models) or ExDoS [9] in terms of scores. On the Pubmed dataset, as Table 2 shows, EBM and GAMI-Net achieved higher R-L scores than SummaRuNNer, but failed to compete with ExtSum-LG [33] and ExtSum-LG+MMR-S+ [34].

Although EBM and GAMI-Net fail to approximate the ROUGE scores of the most advanced summarization approaches, they are able to provide higher levels of transparency on how predictions are being built. For Extractive Summarization, this could be useful for clarifying what is being considered by the models while deciding the importance of the sentences. Figure 1 presents the plot of shape functions built upon the Position feature (Equation 5) given the respective model and dataset, where the horizontal axes represent feature values and the vertical axes denotes the corresponding shape function outputs. In practice, this kind of view allows further investigation of the summarization models, for example, inspecting how training models on different document types may be affecting the learning.

As Figure 1 shows, the shape functions fit on CNN/Dailymail (Figures 1a and 1c) tend to give higher outputs to sentences in the beginning of the documents, while on Pubmed (Figures 1b and 1d) sentences both at the beginning and the end of the documents are likely to be prioritized by the models. In this case, a possible assumption is that models trained on CNN/Dailymail (news type) are likely to avoid sentences at the end of the documents when looking for relevant sentences, which is not the case for all types of documents, as seems to happen for models trained on Pubmed (scientific article type/long document summarization). EBM and GAMI-Net ability to capture such properties intrinsically and grant the possibility of further exploration can be seen as a great benefit when comparing to full-complexity models.

Additionally, EBM and GAMI-Net feature outputs can be easily investigated for single predictions or a group of samples, helping better understanding of feature contributions while producing the summaries. Figures 2 and 3 show the most contributive feature effects quantified by their variation [35] on test set considering CNN/Dailymail and Pubmed datasets, respectively, where Sentence-Sentence Similarity, TF-ISF and Position shape functions are, in general, prevailing in importance over the others. Comparing to other binary classifiers, both EBM and GAMI-Net placed in bewteen XGBoost and Logistic Regression models, reinforcing their position as a balance between predictive power and transparency.

A key limitation of using GAMs with interactions for extractive summarization is the absence of a mechanism to select good “sets” of sentences, rather than just individually relevant sentences. Furthermore, the semantic information gap, arising from the challenge of incorporating it through non-dense features, could be a means of obtaining even more powerful models in the future.