SciBERTSUM: Extractive Summarization for Scientific Documents

We believe summarizing scientific articles is more challenging than summarizing generic text since such articles have a hierarchical structure [9]. They contain technical terms and formulas [24], and much valuable content can be embedded in figures, tables, and algorithms [3].

Scientific article summarization has been less investigated compared to news articles summarization [5, 19]. This seems to be mainly due to the lack of training data for full scientific articles. Types of reference summaries for scientific articles are:

• •

  Abstract: Most of the traditional summarization methods use the abstract as the reference summary of the paper. However, abstracts are extremely compressed versions of a papers and usually do not have enough space to include all of the contributions [7].

• •

  Citation-based: These types of summaries integrate the authors’ highlights in the abstract of the paper with citation context of the citing papers which in some ways reflects the impact in the paper of the research community [24].

• •

  Speaker Transcript: Many conference proceedings/ workshops require the authors to verbally present their work. TalkSum [10] uses the transcript of these presentations as a summary of the scientific article. However, the transcripts in the TalkSum data set are often noisy and can not be readily used as reference summaries.

Presentation slides for a paper are a different class of summaries that intend to cover in some way the important content of the entire paper, sometimes section by section. They contain the main highlights and also valuable images/tables. They are not as noisy as speaker transcripts and are becoming more available as more conferences are providing slides that go with their papers. We used the PS5k dataset [21, 20] to build our summarizer.

Pre-trained language models such as BART [11] produce state-of-the-art results on the summarization tasks. However, they are often used on short news articles such as XSum [17] or CNN-DailyMail [16] datasets. These models are not designed for scientific articles and their space/computational complexity grows quadratically with the size of the input.

HIBERT [25] is an extractive summarizer that learns context aware sentence representations using multiple layers of transformers. Here, 15% of the full sentences are masked (replaced with a single [mask] token) with the goal to predict the sentence embedding of the masked sentences. BERTSUM [12] is another BERT style extractive summarizer that extends BERT to multiple sentences by expanding the positional embedding and using interval segmentation embeddings to distinguish multiple sentences within a document. Sotudeh et al. [22] added section information to the objective function of BERTSUM so it could optimize both the sentence prediction and section prediction tasks in a multi-task setting. However, most of these transformer-based extractive summarizers do not scale for long documents with thousands of tokens nor can they be applied to many full scientific documents.

The embedding layer of BERT [6] applies the byte-pair encoding to tokenize the text. It adds [CLS] tokens to the beginning of the sequences and [SEP] tokens to separate the first and second sentences in the sequence. The embedding representation of the [CLS] token is the representation of the full sequence and is used for sentence classification tasks.

BERT combines 1) Semantic embedding (the meaning of the token) 2) Positional embedding (the position of the token in the sequence), 3) Segmentation embedding ( the embedding layer to distinguish the first and the second sentence in the sequence) to form the embedding of a token in a sequence.

BERTSUM [12] extends BERT to multiple sentences by adding [CLS] tokens to the beginning of all sentences. It changes the segmentation embedding to distinguish odd and even sentences. The embedding model is depicted in Figure 1. The green boxes are the segmentation embeddings. Light greens are the embedding for odd sentences and dark green boxes are for even sentences. BERTSUM extended the positional embedding of BERT beyond 512 tokens to cover all tokens of the input document.

The sentence embeddings are the embedding of the [CLS] tokens which are the combination of semantic, segment, and position embeddings. The Positional encoding is the sinusoid embedding from Vaswani et. al [23].

Long documents especially long scholarly articles contain multiple sections. The section of the document is important in the selection of salient sentences. For instance, the sentences in the ‘acknowledgment’ section are less important compared to other sections like ‘abstract’ or ‘results’. We enhance BERTSUM by adding section embedding as shown in Figure 2. The sentence embeddings ($E_{sents})$ are the combination of the section, semantic, position, and segmentation embeddings. The section embeddings are the blue boxes in Figure 2. All of the tokens of the sentences in the first section are embedded by dark blue and the tokens of sentences in the second section are embedded in light blue. Each section has the same segmentation embedding as in BERTSUM.

To overcome the memory issue where we cannot load the full document with the full position embedding in the memory. We get the sentence vectors section by section. We can load the maximum 3072 tokens to the memory based on experiments on an Nvidia GPU with 11,019 MiB memory capacity.

BERTSUM applies multiple layers of full dot-product attention. However, applying full attention on the sentence vectors is expensive where the number of sentences in documents is large. Scientific documents can have more than 500 sentences. Therefore, extracting document-level features is expensive for long scientific articles with a full attention layer. Therefore, we introduce a lightweight attention mechanism inspired by LongFormer [2]. The LongFormer language model applies sparse attention between tokens to learn the embedding of the masked tokens. We apply their attention mechanism at the sentence level where each sentence will fully attend locally to the nearby sentences and some sentences will attend globally to all sentences in the document.

This attention mechanism will help model select salient sentences locally from the window and at some random and selected positions sentences will attend to all other sentences to identify the salient sentences that are globally important regardless of the section they belong to. The attention window in Figure 3 is 2 which means each sentence will attend to 2 sentences before and after it and in Figure 4 sentences 2 and 7 (marked with *) are attending to all other sentences.

Applying window-based local attention requires a few preprocessing steps. We list the main steps in the following section.

Our sparse attention mechanism is applied in each layer of transformation [23] and the inputs to the first layer of transformer are the sentence embeddings that include the section information

where $h^{0}=E_{sents}$. We apply a sparse attention mechanism here instead of full attention.

The features used to predict the final scores are:

1. 1.

   Length: number of characters in the sentence $i$

   $$E_{length}=ReLU(Linear(Embedding(length[i]))).$$

   (12)

2. 2.

   Position: position of the sentence ($i$) in the document

   $$E_{position}=ReLU(Linear(Embedding(i))).$$

   (13)

3. 3.

   Section: section of the sentence $i$ in the document

   $$E_{section}=ReLU(Linear(Embedding(section[i]))).$$

   (14)

   Each of the embedding layers is a simple lookup table that stores embeddings of a fixed dictionary and size. The size of the Embedding layers is $d$. The linear layer applies a linear transformation to the input data $x$ ($y=Wx^{T}+b$).

4. 4.

   Correlations: the sentence correlations embed the correlation between sentences. The correlation embeddings help the model to identify sentences with a high degree of correlation to other sentences and then exclude them.

   $$Correlation=tanh(E_{sents}\times W_{c}\times E_{sents}^{T}),$$

   (15)

   $$E_{correlation}=ReLU(Linear(Correlation\times E_{sents}))$$

   (16)

   where $W_{c}\in R^{d\times d}$ is the learned correlation matrix and $Correlation\in R^{n\times n}$ and $E_{Correlation}\in R^{n\times d}$.

5. 5.

   Saliency: the saliency embedding will embed the importance of sentence vectors with respect to the document embedding. The saliency weight matrix $W_{s}\in R^{d\times d}$ is learned in the training phase.

   $$Saliency=tanh(E_{sents}\times W_{s}\times E_{D}^{T}),$$

   (17)

   $$E_{Saliency}=ReLU(Linear(Saliency*E_{sents}))$$

   (18)

   where $W_{s}\in R^{d\times d}$ is the learned saliency matrix and $Saliency\in R^{n\times 1}$. The document embedding ($E_{D}=$) is the weighted average of the sentence embeddings as explained in 5.2

The document encoder is a simply the weighted average of the sentence vectors:

where $E_{sents}\in R^{n\times d}$ and $W_{sents}\in R^{d\times 1}$. The wights are initialized randomly and will be learned during the training process. Therefore, $Weight\in R^{n\times 1}$ are the weights of the sentences.

Therefore, the embedding of document $D$ is:

where the terms are defined above.

The score prediction module concatenates all of the features and feeds them to a linear layer to generate the final scores. Our cross-entropy loss evaluates the difference between the prediction and the ground truth scores. We also evaluated the loss factored by the rewards to see if the model makes better predictions using reinforcement learning (in section 6)

where $p(y_{i})$ is the probability of adding sentence $i$ to the summary and the linear layer format is $Wx^{T}+b$ and $x\in R^{1\times d}$ and $w\in R^{1\times d}$.

Three NVIDIA GPUs (GeForce RTX 2080 Ti) with 11019MiB on memory were used. The batch size is set to 1 because of the size of the input document. Since we could not have a large batch size, we accumulate the gradients for 10 steps and then update the parameters. Learning rate is one of the most important hyper-parameters to tune. We used the NOAM scheduler to adjust the learning rate while training and apply gradient-clipping to prevent exploding gradients.

Table 1 shows different values for the size of the local attention window and the ratio of the sentences that will attend globally to all other sentences in the document. The results show that increasing the window size for local attention and global attention ratio will improve the ROUGE recall scores. We can set the size of global and local attentions based on the hardware available at hand. Our model converges faster with a larger attention window and more global attentions.

The effect of reinforcement learning on our model is shown in Table 2. The reinforcement learning does not improve the results on our dataset mainly due to reducing the bias toward the position and length of the sentences.

Our results outperform many of the tested extractive and abstractive models as seen in Table 3. The sentence scores of BERTSUM in Table 3 are generated chunk by chunk since this model is not designed for extractive summarization of long documents. The BART and T5 summaries are generated section by section since they were developed for short sequences and have problems with long documents.

Table 4 shows the effect of trigram blocking in our dataset. If we block adding a sentence if it has a shared trigram, the results are not improved (first row). We also tried allowing some shared trigram in the sentence. For example, the third row in table 4 only blocks sentences if there are more than 5 shared trigrams with the current summary. We see that the results get worse with trigram blocking. It shows that the scores predicted by the model are good enough to understand if adding a sentence with shared tokens can improve the ROUGE scores.