CiMaTe: Citation Count Prediction
Effectively Leveraging the Main Text

Existing methods for predicting citation counts include those that use textual information and those that do not. Among methods that do not use textual information, those of Castillo et al. (2007), Davletov et al. (2014), and Pobiedina and Ichise (2015) represent the citation relationships of papers as a graph. Abrishami and Aliakbary (2018) proposed a method that predicts the long-term citation count from short-term citation counts by using an RNN.

As for methods that use textual information, Ibáñez et al. (2009) proposed a method that uses a bag-of-words with abstracts, Fu and Aliferis (2008) proposed a method that uses frequency-based weighting, and Yan et al. (2011) and Chakraborty et al. (2014) proposed methods that use topics estimated by LDA. In recent years, there has been research on methods to capture the semantic features of papers’ textual information through deep learning. Ma et al. (2021) proposed a method that encodes the title and abstract with Doc2Vec Le and Mikolov (2014), and van Dongen et al. (2020) proposed a method that uses a paper’s main text via a pre-trained model. Hirako et al. (2023) proposed a BERT-based method that captures the latest research trends for newly-published papers.

There are two major approaches to processing long documents in a Transformer-based model. The first approach modifies the Transformer structure to efficiently process long sequences  Kitaev et al. (2020); Beltagy et al. (2020); Zaheer et al. (2020). These methods mainly modify the self-attention module and improve the computational complexity. The second approach processes long documents by dividing them up. This approach includes methods that encode the divided text with BERT and aggregate it with simple techniques such as attention or an RNN Afkanpour et al. (2022); van Dongen et al. (2020). The second approach also includes hierarchical methods in which a BERT-encoded representation is further input to a Transformer Pappagari et al. (2019); Xue et al. (2021).

SChuBERT van Dongen et al. (2020) is an existing method for predicting citation counts from the whole main text of papers. As shown in Figure 1 (a), it predicts the citation count by dividing the main text into overlapping chunks, encoding each of them with BERT, and aggregating them with a GRU Cho et al. (2014). As SChuBERT does not fine-tune BERT, it can leverage the whole main text with a relatively small computational cost. However, it divides the main text with a fixed number of characters and does not explicitly capture a paper’s structure.

Longformer Beltagy et al. (2020), a pre-trained language model for processing long sequences, can be used to predict citation counts leveraging the main text. As shown in Figure 1 (b), this method inputs the beginning of the main text by truncating it to a maximum input length of 4096 tokens. While this truncation is commonly used Xie et al. (2020), much of a paper’s main text may be truncated, and Longformer thus cannot consider the whole paper equally. For example, it truncated about 46% of the main text on average for the dataset used in our experiments (see Section 4.2).

Our proposed method, CiMaTe predicts citation counts by encoding each section with BERT and then pooling the encoded section representations. We introduce two versions of CiMaTe: CiMaTe${}_{\mbox{b}}$, which leverages only the beginning of each section, and CiMaTe${}_{\mbox{w}}$, which divides each section to leverage the whole main text. Figure 1 (c) shows an overview of CiMaTe${}_{\mbox{b}}$. This method inputs the section title as the first sentence and the content as the second sentence into BERT to generate section representations. The input content is truncated to not exceed 512 tokens, BERT’s maximum input length. While this method mitigates the increase in computational cost, only limited content is taken into account for each section.

Figure 1 (d) shows an overview of CiMaTe${}_{\mbox{w}}$. First, this method divides the content of each section into chunks; then it encodes each chunk with BERT. The chunk representations are then pooled with their means to generate section representations. Following previous studies Pappagari et al. (2019); van Dongen et al. (2020), each content is divided into chunks that overlap by 50 tokens each. In encoding each chunk, the section title is input as the first sentence, and the chunk, as the second sentence, even if it is the second or later chunk. While this method can take the whole content into account, it increases the computational cost.

These two methods predict citation counts by pooling section representations. For a more expressive pooling method, we use Transformers besides the means. In Transformer pooling, each section representation is input to a one-layer Transformer as a sequence, and then the output is averaged.

To validate the effectiveness of the CiMaTe, we compared several methods on the realistic citation count prediction task for newly published papers proposed by Hirako et al. (2023). In this task, the information used for training is strictly restricted to that available at the time of the target paper’s publication, which allows for a more realistic evaluation of the prediction method. The target of the prediction is the citation counts one year after publication. While the citation count one year after publication is available for papers published more than one year before the target paper, it is not available for papers published less than one year before the target paper. Thus, we used the method proposed by Hirako et al. (2023) to complement the citation counts for these papers. The specific target of the prediction is the logarithm $\mbox{log}(c+1)$ rather than the citation count $c$, reflecting the intuition that the difference between 0 and 10 cases is more significant than between 1,000 and 1,010 cases.

As with Hirako et al. (2023), we constructed two datasets from the preprint server papers: a CL dataset from papers in the computational linguistics field, and a Bio dataset from the biology field. The reason for using papers from the preprint server was that while a large number of papers are published, some papers are not peer-reviewed and their quality is not guaranteed, so it was considered highly important to predict citation counts.

The CL dataset was constructed from papers submitted to the cs.CL category of arXiv¹¹1https://arxiv.org/ between June 2014 and June 2020. Citation information for calculating the citation counts was collected from Semantic Scholar.²²2https://www.semanticscholar.org/ The main texts of papers were collected by parsing their HTML versions.³³3https://ar5iv.labs.arxiv.org/ Papers for which HTML versions could not be collected and parsed were excluded from the dataset. We created 13 subsets of the CL dataset. Specifically, starting with each month between June 2019 and June 2020 inclusive, we created 13 subsets comprising papers that had been submitted in the previous 5 years before each of those months. The papers submitted in the last month of each subset were used for evaluation, while the other papers in the subset were used for training. Among these subsets, the one using papers published in June 2019 for evaluation was used as the development set, while the remaining subsets were used to calculate the evaluation scores. The average numbers of papers for training and evaluation were 11,383 and 449, respectively.

The Bio dataset was constructed from papers submitted to the Biochemistry, Plant Biology, and Pharmacology and Toxicology categories of bioRxiv⁴⁴4https://www.biorxiv.org/ between April 2015 and April 2021. The main texts were collected by parsing the papers’ HTML versions on bioRxiv. As with the CL dataset, we created 13 subsets of the Bio dataset. Papers submitted in each month from April 2020 to April 2021 inclusive were used for evaluation, and the subset with the oldest papers for evaluation was again used as the development set. The average numbers of papers for training and evaluation were 5,737 and 256, respectively.

For evaluation metrics, we used Spearman’s rank correlation coefficient ($\rho$), the mean squared error (MSE), and a metric defined as the percentage of the actual top n% of papers in the top k% of the output (n%@k%). We calculated the evaluation scores as follows. First, for each subset, we trained a model and use that model to predict the citation count for each paper. We then compiled the predictions of citation counts for the 12 subsets and calculated an evaluation score on the whole. In all settings, we experimented with three different random seeds and calculated the means and standard deviations of the evaluation scores.

For certain methods, we found in our experiments that the predictions tend to be slightly larger than the correct values, which had a negative impact on the appropriate evaluation using MSE. To mitigate this, we also used MSE*, a variant of the MSE, as an evaluation metrics. This metric calculates the MSE by subtracting the difference between the average of the predictions and the average of the correct values for the development set from the predictions at test time.

In addition to the methods described in Section 3, we compared two baseline methods using BERT. First, BERT${}_{\mbox{t+a}}$ was a method to predict citation counts by inputting the title as the first sentence and the abstract as the second sentence into BERT. Second, BERT${}_{{\mbox{b}}}$, was a method to predict citation counts by inputting the beginning of the main text into BERT, similarly to Longformer, but this method used only a maximum of 512 tokens, whereas Longformer allows a maximum of 4096 tokens to be input.

We used the vector representation corresponding to the [CLS] token as the output of BERT and Longformer; then, they input the model’s final vector output to a fully connected layer to predict citation counts. In CiMaTe${}_{\mbox{w}}$, to somewhat mitigate the computational cost, the content of each section was divided into a maximum of eight chunks, and the remaining text was truncated. The loss function was the MSE of the predictions and the correct values, and we applied dropout Srivastava et al. (2014) to the final output vector during training.

For pre-trained weights, we used ‘bert-base-uncased’ for BERT and ‘allenai/longformer-base-4096’ for Longformer, both of which are publicly available on the Transformers Wolf et al. (2020). Also, there is no Longformer variant that was pre-trained on the scientific domain corpus. To make a fair comparison, we used only BERT that was pre-trained on the general domain corpus and did not use models pre-trained on the scientific domain corpus, such as SciBERT Beltagy et al. (2019).

All methods except SChuBERT were trained with a batch size of 32, the AdamW Loshchilov and Hutter (2019) optimizer, and a learning rate schedule that warm-up to 10% of the total training steps and then decayed linearly over the remaining steps. For the number of epochs and the learning rate, we performed grid search with values of {3, 4} for the number of epochs and {2e-5, 3e-5, 5e-5} for the learning rate, and we then used the values that yielded the highest rank correlation coefficient on the development set. For SChuBERT, only the number of epochs was determined by hyperparameter search, while the remaining settings were trained as in previous studies. The search range for the number of epochs was set to {20, 30, 40} according to the values in previous studies.

The experimental results are summarized in Table 1. Except for SChuBERT, the methods that leverage a paper’s main text outperformed BERT${}_{\mbox{t+a}}$ on both datasets; thus we confirmed the effectiveness of leveraging the main text in predicting citation counts. Among the methods that leverage the main text, CiMaTe, which explicitly captures a paper’s sectional structure, achieved the highest performance. Specifically, in terms of the rank correlation coefficient, the model that scored best among the CiMaTe variants outperformed the other methods by at least 5.1 points on the CL dataset and at least 1.8 points on the Bio dataset. In terms of the MSE, MSE*, and n%@k%, it also outperformed the other methods in most cases, which demonstrates the general effectiveness of CiMaTe on sets of papers from different fields, namely, computational linguistics and biology.

Between the two CiMaTe versions, CiMaTe${}_{\mbox{b}}$ achieved the same or better performance as CiMaTe${}_{\mbox{w}}$, and we thus found that by using only the beginning of each section, it was possible to achieve high prediction performance while mitigating the increase in computational cost. We also confirmed that Transformer pooling performed better overall than using means in the pooling of section representations. Although SChuBERT showed an improved MSE and MSE* on the CL dataset, its performance was inferior to that of BERT${}_{\mbox{t+a}}$ in other cases. We speculate that this was because SChuBERT did not fine-tune BERT to accept the whole main text as input with a relatively small computational cost Pappagari et al. (2019).