Neural Content Extraction for Poster Generation of Scientific Papers

As stated above, not all sections from a paper should be selected to make panels when making a poster. Some sections are not important enough, and it is unnecessary to include these sections in the poster due to the limited space. Thus we need to first filter some certain sections out.

Specifically, given a paper, we first encode the text of each section separately. We utilize a pre-trained model RoBERTaLiu et al. (2019) for encoding, and it returns a vector representation of each section. We consider the vectors of sections as a sequence, and further use a transformer encoder to obtain their paper-level representations. After that, a classifier is used to evaluate each section’s importance degree and decide whether to choose it as one of the source sections of poster.

In the experiments, we compare our model with the following two ablation models:

• •

  Ablation Model 1: We want to verify whether the content text of sections is necessary to evaluate their importance. The model is the same as the one described above while it takes only the titles of sections as input.

• •

  Ablation Model 2: This model does not use the transformer encoder. After using RoBERTa to get the vector representation of a section, the model directly uses a classifier to predict its label without learning and using its paper-level representation.

The section classification accuracy of our model, ablation model 1 and ablation model 2 is 88.01%, 74.52%, and 84.08% respectively. From the result, we notice that the titles of sections are not sufficient to determine their importance. It is the content texts of sections that matter. We also learn that the global information of the whole paper is beneficial for learning representations of sections.

Given a section from the paper which includes a few sentences and may additionally contain tables and figures, we want to select the most important sentences, together with the tables and figures which are highly relevant to selected sentences. Here we decide not to directly use the images of figures and tables but to utilize their captions instead. Usually the pictorial elements in figures are abstract and hard for models to understand, and models can be poor at learning the information revealed in the tables with a bunch of numbers. Fortunately, most tables and figures in papers are accompanied with captions which describe and summarize them in natural language texts. Thus it is easier for models to learn the meanings of tables and figures from their captions.

Formally we define the content extraction problem as follows: Given sentences from a section: $\{s_{1},s_{2},...s_{n}\}$, together with the captions of the graphs (i.e., figures and tables) included in the section: $\{c_{1},c_{2},...c_{m}\}$, the model needs to extract the most important sentences and graphs.

The tackle the problem, we propose a model which learns to predict the importance scores of both sentences and graphs simultaneously. The idea underlying our model is that the extraction of sentences and the extraction of graphs can help each other. We further make use of additional attention weight to help the model capture the relationship between sentences and graphs. This turns out to help stabilize and improve the performance of our model.

Specifically, we first use a pretrained RoBERTa model to encode both sentences and graph captions in a section. Because every caption starts with word “Figure” or “Table” which is followed by its number, the model can easily distinguish captions from sentences after training. Thus we do not employ additional mechanism to differentiate these two kinds of source inputs. All the input sentences and captions are encoded separately at this stage. We get vector representations $W=\{w_{1},w_{2},...w_{n}\}$ and $U=\{u_{1},u_{2},...u_{m}\}$ for section sentences and graph captions respectively: $w_{i}=RoBERTa(s_{i}),u_{j}=RoBERTa(c_{j})$.

We assume the context information of a section matters a lot to evaluate the importance of each sentence or graph in the section. Moreover, the captions can also be used to help learn the importance of sentences. If a sentence is semantically related to an important graph, it is highly possible that the sentence is also significant, and vice versa. Therefore, we add transformer encoder Vaswani et al. (2017) layers stacked on top of RoBERTa outputs, i.e., the RoBERTa outputs for section sentences and graph captions are concatenated together as input for the transformer encoder. These layers encode both sentences and graph captions to learn their section-aware representations.

We innovatively add additional attention weight during the calculation of self-attention, in order to help the encoder catch the relationship between sentences and graphs better. Recall that the original transformer encoder calculates attention matrix following the equation below:

$$Attention(Q,K)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})$$

(1)

$$Attention(Q,K)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}}+A)$$

(2)

The remaining parts of our encoder, such as feed-forward network, LayerNorm and residual connection, are the same as the original transformer encoder. After that, we use fully connected layers to predict the importance scores of the sentences and graphs $P=\{p_{1},p_{2},...p_{n+m}\}$.

Note that the number of negative sentences are overwhelmingly larger than that of positive ones. In addition, there are more training samples for sentence extraction than that for graph extraction. In order to deal with the issue of imbalanced data distribution, we make use of balanced cross entropy loss:

$$loss_{sentence/graph}=\left\{\begin{aligned} -\alpha log(p),\text{ if }y=1\\ -\beta log(1-p),\text{ if }y=0\end{aligned}\right.$$

(3)

$y$ is the gold label of one sentence or graph, $p$ is the importance score predicted by the model, and $\alpha$ and $\beta$ are hyper-parameters that control the importance of different parts. Finally, our model optimizes the weighted sum of sentence loss and graph loss simultaneously: $loss=loss_{sentence}+\gamma*loss_{graph}$. $\gamma$ is set to be larger because the numbers of graph captions are less than section sentences.

After getting contents of all panels, including text and corresponding graphs, the final step is to integrate the panels to get the poster. Previous work generates posters in portrait format only. However, some conferences may demand the posters to be certain format. We decide that the users should choose whether the poster is landscape or portrait format.

Following the work of Paramita and Khodra (2016), we predefine several templates of both portrait and landscape formats. The model will search for proper templates based on the format user chooses, the numbers of panels, text length, graph number, graph size and other attributes. The output of the model is a LaTeX document which makes use of tikzposter package. It generates scientific poster after compilation. People can make further modification to get customized poster.

Specifically, the encoder is composed of 3 identical layers. The dimensionality of input and output of encoder layers is $d_{k}=768$, the dimensionality of feed-forward inner-layer is $d_{ff}=1536$. $h_{1}$ and $h_{2}$ are set $1e-2$ and $1e-3$ separately for all layers. During training, the weights of $loss_{sentence}$ are set $\alpha=1.0,\beta=0.5$, while for $loss_{graph}$, $\alpha=\beta=1.0$, and $\gamma$ is set 3.0. We use Adam optimizer with an initial learning rate $lr=8e-5$, momentum $\beta_{1}=0.9$, $\beta_{2}=0.999$.

The following four extractive models and one abstractive model are used for comparison:

Lead-3: Lead-3 is an extractive baseline which directly chooses and concatenates the first three sentences of source document to get a summary.

TextRank: TextRank is an unsupervised method to calculate the importance of the sentences based on their similarity  Mihalcea and Tarau (2004). After applying TextRank algorithm, the model greedily selects the most important sentences to get summary until the total length reaches the limit. This is also the method used in Qiang et al. (2016, 2019) for sentence extraction.

BERTSum: BERTSum is a simple variant of BERT for extractive summarization Liu (2019). After concatenating all sentences of input document together, it uses pre-trained BERT model  Devlin et al. (2018) for encoding. The model further builds several layers on top of BERT outputs to capture document-level features, and then calculates importance scores of the sentences. Apart from BERTSum, we also try RoBERTaSum, which replaces BERT model with RoBERTa, because RoBERTa generally performs better than BERT.

GPT-2: Other than the extractive summarization models, we finetune GPT-2  Radford et al. (2019) on this task. The source input is the text of a section, and its target output is the text of the corresponding panel. This model is a strong abstractive model.

We also implement three ablation models for comparison:

Without weight: In this ablation model, we discard the additional attention weight and follow Equation 1, i.e., using the original transformer encoder to calculate attention weight.

Without caption: In this model, the graph captions in the data are all abandoned. The model only uses the sentences of the section and predicts their importance scores after learning their section-aware representations.

Image: We borrow the idea from multimodal summarization and come up with this baseline. The baseline is the same as our model except that it extracts graph features by utilizing ResNet He et al. (2016) while our model manages that by using a RoBERTa to encode graph captions.

Oracle: Finally we implement the extractive oracle model. It extracts sentences from source document to optimize ROUGE-2 F1 score. The performance of oracle is considered as the upper bound of extractive models.

Two baseline models and three ablation models are used for comparing graph extraction results:

Similarity: This model is simply based on textual similarity and the score of a graph is the maximum cosine similarity between the caption and each sentence in the section. We decide a threshold value based on training data. The graphs with scores larger than the threshold are labeled positive.

RoBERTa: For comparison, we finetune a RoBERTa model to classify graphs. The model encodes the caption and then use a classifier to decide whether to choose that table or figure without using the section’s text.

Without section: The ablation model abandons all section sentences and only uses captions.

In addition, we also use Without weight and Image models, as described in the prior subsection.

Note that we use 10-fold cross-validation in the content extraction experiments, we report the mean and standard derivation of the 10 experimental results. For text extraction part, we follow the previous work of document summarization and report ROUGE scores Lin (2004) in Table 2.

As illustrated in the table, our proposed model outperforms all baseline models. From the performance of Lead-3, we notice that the first several sentences can still cover some of the important points of a section. TextRank performs a little better than Lead-3. RoBERTaSum gets better scores than BERTSum on all three ROUGE metrics, and they generally perform better than Lead-3 and TextRank. It can be observed that the performance of GPT-2 is relatively low, as compared with the two neural extractive summarization models. The size of the training data may be still not large enough to finetune a generation model with huge parameters.

The difference between RoBERTaSum and the ablation model without caption lies in the way they encode the section text: the former encodes the whole text after concatenating all sentences, while the latter encodes all sentences separately. As shown in Table 1, the text length of section can easily reach the length limitation of BERT, which is 512. By encoding the section as a whole, truncation is required and can lead to information loss. As a result, RoBERTaSum performs a little lower than without caption. While our full model further outperforms without caption, indicating that the graph part can also benefit text extraction. The performance of Image is close to without caption, which shows that the graph features extracted from images cannot help much.

The ablation model without weight performs slightly lower than our full model. It seems that the model can learn a little bit better when the attention weight matrix between sentences and graphs is given explicitly. We also observe that the standard derivation of our model is lower than that of the ablation models, which indicates that with the help of the attention weight matrix, our model can perform more stably.

The accuracy of the graph extraction models is reported in Table 3. We can see that the accuracy of similarity based model reaches a relatively high level, which means the textual similarity does help decide whether the table or figure is important. The accuracy of RoBERTa is a little higher than similarity based model, although it makes no use of section text. We can draw a conclusion that section sentences can help the model better capture the features of graphs when comparing the performance of our full model with ablation model without section. The performance of Image is relatively lower, which indicates that the model performs poorly at learning from their images.

From the experimental results, we notice that the part of text extraction and graph extraction can benefit from each other during training because they are closely connected. By learning and predicting the importance of sentences and graphs at the same time, our model performs better on both parts than baseline models.

Human evaluation is also conducted for text extraction. We ask human judges to assess the output text of our model against other baseline models, namely TextRank, RoBERTaSum and GPT-2. The evaluation is conducted in three aspects:

1. 1.

   Informativeness: whether the output text covers important points of the section.

2. 2.

   Fluency: whether the output text is well-formed and grammatically correct.

3. 3.

   Non-Redundancy: whether the output text contains redundant or repeated information.

For each input sample, people are asked to judge whether the output of our model outperforms the specific baseline in the three aspects. Their answers are chosen from win, lose and tie and they are not told which one of the pair is the output of our model. We randomly choose 40 samples and get 120 pairs totally. Each output pair is represented to two judges. Judges are university students who are good at English and familiar with the research field of that paper. The exact match ratio of human annotation is 66.94%.

The results are presented in Figure 2. Our model performs better than or as well as other baseline models on all three aspects. The results indicate that our model can produce panel texts which are more fluent and less redundant.

In Table 4, we present examples of panel texts produced by our model and RoBERTaSum. The ground truth focuses on different factors that affect the performance of transformer model. The output of our model also states the influence of batch size, multiple GPUs and gradient accumulation, while still missing some points due to length limitation. However, RoBERTaSum only extracts a transition sentence, which is not quite important.

In Figure 3, we present a poster generated by our three-step method. As shown in the poster, our model manages to extract important points together with related graphs. The panels are well-placed and the layout is reasonable.