Tstr: Too Short to Represent, Summarize with Details!
Intro-Guided Extended Summary Generation

Over the past few years, summarization task has witnessed a huge deal of progress in extractive Nallapati et al. (2017); Liu and Lapata (2019); Yuan et al. (2020); Cui et al. (2020); Jia et al. (2020); Feng et al. (2018) and abstractive See et al. (2017); Cohan et al. (2018); Gehrmann et al. (2018); Zhang et al. (2019); Tian et al. (2019); Zou et al. (2020) settings. Many scientific papers such as those in arXiv and PubMed Cohan et al. (2018) posses abstracts of varying length, ranging from 50 to 1000 words and average length of approximately 200 words. While scientific paper summarization has been an active research area, most works Cohan et al. (2018); Xiao and Carenini (2019); Cui and Hu (2021); Rohde et al. (2021) in this domain have focused on generating typical short and abstract-like summaries Chandrasekaran et al. (2020). Short summaries might be adequate when the source text is of short-form such as those in news domain; however, to summarize longer documents such as scientific papers, an extended summary including 400–600 terms on average, such as those found in extended summarization datasets of arXiv-Long and PubMed-Long, is more appealing as it conveys more detailed information.

Extended summary generation has been of research interest very recently. Chandrasekaran et al. (2020) motivated the necessity of generating extended summaries through LongSumm shared task ¹¹1https://ornlcda.github.io/SDProc/sharedtasks.html. Long documents such as scientific papers are usually framed in a specific structure. They start by presenting general introductory information ²²2We will exchangeably use (non-)introductory information and (non-)introductory sentences in the rest of this paper.. This introductory information is then followed by supplemental information (i.e., non-introductory) that explain the initial introductory information in more detail. Similarly, as shown in Figure 1, this pattern holds in a human-written extended summary of a long document, where the preceding sentences (top box inside Figure 1) are introductory sentences and succeeding sentences (bottom box inside Figure 1) are explanations of the introductory sentences. In this study, we aim to guide the summarization model to utilize the aforementioned rationale in human-written summaries. We consider introductory sentences as those that appear in the first section of paper with headings such as Introduction, Overview, Motivations, and so forth. As such, all other parts of paper and their sentences are considered as non-introductory (i.e., supplementary). We use these definitions in the reminder of this paper.

Herein, we approach the problem of extended summary generation by incorporating the most important introductory information into the summarization model. We hypothesize that incorporating such information into the summarization model guides the model to pick salient detailed non-introductory information to augment the final extended summary. The importance of the role of introduction in the scientific papers was earlier presented in  Teufel and Moens (2002); Armağan (2013); Jirge (2017) where they showed such information provides clues (i.e. pointers) to the objectives and experiments of studies. Similarly, Boni et al. (2020) conducted a study to show the importance of introduction part of scientific papers as its relevance to the paper’s abstract. To validate our hypothesis, we test the proposed approach on two publicly available large-scale extended summarization datasets, namely arXiv-Long and PubMed-Long. Our experimental results improve over the strong baselines and state-of-the-art models. In short, the contributions of this work are as follows:

• •

  A novel multi-tasking approach that incorporates the salient introductory information into the extractive summarizer to guide the model in generating a 600-term (roughly) extended summary of a long document, containing the key detailed information of a scientific paper.

• •

  Intrinsic evaluation that demonstrates statistically significant improvements over strong extractive and abstractive summarization baselines and state-of-the-art models.

• •

  An extensive human evaluation which reveals the advantage of the proposed model in terms of cohesion and completeness.

Summarizing scientific documents has gained a huge deal of attention from researchers, although it has been studied for decades. Neural efforts in scientific text have used specific characteristics of papers such as discourse structure Cohan et al. (2018); Xiao and Carenini (2019) and citation information Qazvinian and Radev (2008); Cohan and Goharian (2015, 2018) to aid summarization model. While prior work has mostly covered the generation of shorter-form summaries (approx. 200 terms), generating extended summaries of roughly 600 terms for long-form source documents such as scientific papers has been motivated very recently Chandrasekaran et al. (2020).

The proposed models for the extended summary generation task include jointly learning to predict sentence importance and sentence section to extract top sentences Sotudeh et al. (2020); utilizing section-contribution computations to pick sentences from important section for forming the final summary Ghosh Roy et al. (2020); identifying salient sections for generating abstractive summaries Gidiotis et al. (2020); ensembling of extraction and abstraction models to form final summary Ying et al. (2021); an extractive model with TextRank algorithm equipped with BM25 as similarity function Kaushik et al. (2021); and incorporating sentences embeddings into graph-based extractive summarizer in an unsupervised manner Ramirez-Orta and Milios (2021). Unlike these works, we do not exploit any sectional nor citation information in this work. To the best of our knowledge, we are the first at proposing the novel method of utilizing introductory information of the scientific paper to guide the model to learn to generate summary from the salient and related information.

Contextualized language models such as Bert Devlin et al. (2019), and RoBerta Liu et al. (2019) have achieved state-of-the-art performance on a variety of downstream NLP tasks including text summarization. Liu and Lapata (2019) were the first to fine-tune a contextualized language model (i.e., Bert) for the summarization task. They proposed BertSum —a fine-tuning scheme for text summarization— that outputs the sentence representations of the source document (we use the term source and source document interchangeably, referring to the entire document). The BertSumExt model, which is built based on BertSum, was proposed for the extractive summarization task. It utilizes the representations produced by BertSum, passes them through Transformers encoder Vaswani et al. (2017), and finally uses a linear layer with Sigmoid function to compute copying probabilities for each input sentence. Formally, let $l_{1},l_{2},...,l_{n}$ be the binary tags over the source sentences $\mathbf{x}=\mathbf{\{}sent_{1},sent_{2},...,sent_{n}\mathbf{\}}$ of a long document, in which $n$ is the number of sentences in the paper. The BertSumExt network runs over the source documents as follows (Eq. 1),

where $\mathbf{h_{b}}$ and $\mathbf{h}$ are the representations of source sentences encoded by BertSum and Trasformers encoder, respectively. $W_{o}$ and $b_{o}$ are trainable parameters, and ${p}$ is the probability distribution over the source sentences, signifying extraction copy likelihood. The goal of this network is to train a network that can identify the positive sets of sentences as the summary. To prevent the network from selecting redundant sentences, BertSum uses Trigram Blocking Liu and Lapata (2019) for sentence selection in inference time. We refer the reader to the main paper for more details.

In this section, we describe our methodology to tackle the extended summary generation task. Our approach exploits the introductory information ³³3Introductory information is defined in Section 1. of the paper as pointers to salient sentences within it, as shown in Figure 2. It is ultimately expected that the extractive summarizer is guided to pick salient sentences across the entire paper.

The detailed illustration of our model is shown in Figure 3. To aid the extractive summarization model (i.e., right-hand box in Figure 3) which takes in source sentences of a scientific paper, we utilize an additional BertSum encoder called Introductory encoder (left-hand box in Fig. 3) that receives $\mathbf{x_{intro}}=\mathbf{\{}sent_{1},sent_{2},...,sent_{m}\mathbf{\}}$, with $m$ being the number of sentences in introductory section. The aim of adding second encoder in this framework is to identify the clues in the introductory section which point to the salient supplementary sentences ⁴⁴4Supplementary sentences are defined in Section 1.. The BertSum network computes the extraction probabilities for introductory sentences as follow (same way as in Eq. 1),

in which $\tilde{h}_{b}$, and $\tilde{h}$ are the introductory sentence representations by BertSum, Transformers encoder, respectively. ${\tilde{p}}$ is the introductory sentence extraction probabilities. $W_{j}$ and $b_{j}$ are trainable matrices.

After identifying salient introductory sentences, the representations associated with them are retrieved using a pooling function and further used to guide the first task (i.e., right-hand side in Figure 3) as follows,

where $\texttt{{Select}}(\cdot)$ is a function that takes in all introductory sentence representations (i.e., $\tilde{h}$), and introductory sentence probabilities $\tilde{p}$. It then outputs the representations associated with top ${k}$ introductory sentences, sorted by $\tilde{p}$. To extract top introductory sentences, we first sort $\tilde{h}$ vectors based on their computed probabilities $\tilde{p}$ and then we pick up top $k$ hidden vectors (i.e., $\tilde{h}_{top}$) that has the highest probability. $\mathbf{MLP}_{\mathbf{1}}$ is a multi-layer perceptron that takes in concatenated vector of top introductory sentences and projects it into a new vector called $\hat{h}$.

At the final stage, we concatenate the transformed introductory top sentence representations (i.e., $\hat{h}$) with each source sentence representations from Eq. 1 (i.e., $h_{i}$ where $i$ shows the $i$th paper sentence) and process them to produce a resulting vector ${r}$ which is intro-aware source sentence hidden representations. After processing the resulting vector through a linear output layer (with $W_{z}$ and $b_{z}$ as trainable parameters), we obtain final intro-aware sentence extraction probabilities (i.e., ${{p}}$) as follows,

in which $\mathbf{MLP_{2}}$ is a multi-layer perceptron, influencing the knowledge from introductory sentence extraction task (i.e., $\mathbf{t_{2}})$ into the source sentence extraction task (i.e., $\mathbf{t_{1}}$). We train both tasks through our end-to-end system jointly as follows,

where $\ell_{\mathbf{t_{1}}}$, and $\ell_{\mathbf{t_{2}}}$ are the losses computed for introductory sentence extraction and source sentence extraction tasks, $\alpha$ is the regularizing parameter that balances the learning process between two tasks, and $\ell_{\mathbf{total}}$ is the total computed loss that is optimized during the training.

In this section, we explain the datasets, baselines, and preprocessing and training parameters.

To determine the limitations of our model, we further analyze our system’s generated summaries and report three common defects, along with the percentage of these errors among underperformed cases. We found that (1) our end-to-end system’s performance is highly dependent on the introductory sentence extraction task’s performance (i.e., task $\mathbf{t_{2}}$ in Figure 3) as identification of salient introductory sentences (i.e., oracle introductory sentences) sets up a firm ground to explore detailed sentences from the non-introductory parts of the paper. In other words, identification of non-salient introductory sentences leads to a drift in finding supplemental sentences from the non-introductory parts. Our model often underperforms when it cannot find important sentences from the introductory part (65%); (2) in underperformed cases, our model fails in selecting motivation, objective sentences from the introductory part, and only identifies the contribution sentences (i.e., describing paper’s contributions), such that the final generated summary is composed of contribution sentences, rather than objective sentences. This observation hurts the system in cohesion and completeness (40%); and (3) as discussed, our model matches introductory sentences with sentences from non-introductory parts of the paper. Given that two sentences within a scientific paper might conceptually convey the exact same information, but are just paraphrased of each other, our model samples both to form the final summary as a high semantic correlation exists between them. This phenomenon leads to sampling two sentences that convey the same information without providing more details; hence, information redundancy (35%).

In this work, we propose a novel approach to tackle the extended summary generation for scientific documents. Our model is built upon the fine-tuned contextualized language models for text summarization. Our method improves over strong and state-of-the-art summarization baselines by adding an auxiliary learning component for identifying salient introductory information of long documents, which are then used as pointers to guide the summarizer to pick summary-worthy sentences. The extensive intrinsic and human evaluations show the efficacy of our model in comparison with the state-of-the-art baselines, using two large scale extended summarization datasets . Our error analysis further paves the path for future reseacrh.