Towards understanding evolution of science through language model series

The Transformers architecture, along with the growing accessibility of full-text corpora of scientific papers, has heightened the significance of employing NLP techniques for document understanding and knowledge extraction [2]. Similar to general domains, transfer learning through pretraining models can leverage information learned in a self-supervised fashion to aid in downstream scientific NLP tasks where datasets are scarce, leading to the introduction of numerous scientific language models (SciLMs) for a number of fields, including biomedicine [6, 4, 8, 14, 15, 16], chemistry [17], material science [9], and social science [18].

Table I summarizes some notable SciLMs that employ the BERT architecture. BioBERT [4] was resulted from continual pretraining of the original BERT model on PubMed abstracts and showed better performances on biomedical NLP tasks. SciBERT [6] was pretrained from scratch on the full-text of 1.1M biomedical and computer science papers in the Semantic Scholar corpus [19]. Evaluations on multiple tasks showed over-performances than the original BERT and comparable performances as BioBERT on biomedical tasks. BiomedBERT [7] was also a pretrained model from scratch on 3.2B tokens in 14M PubMed abstracts. The work highlights the necessity of pretraining from scratch, as it yields a domain specific vocabulary that is useful for downstream tasks. In their case, the vocabulary contains more biomedical terms like “acetyltransferase”, in contrast to the vocabulary in BioBERT that consists of terms common to the general domain. OAG-BERT [5] was trained on the AMiner and PubMed corpora. It is based on the BERT architecture but fine-tuned for academic graph related tasks such as author name disambiguation, paper recommendation, and citation prediction. ScholarBERT [20] was pretrained from scratch on perhaps the largest corpus (221B tokens in 75M journal articles provided by public.resource.org). One observation from this work is that performances in downstream scientific tasks saturate when increasing training data, model size, or training time.

In addition to scientific text, other modalities of scientific data have been used for pretraining language models, including math equations [21], chemicals, proteins, etc. Notably, SMILES strings of chemicals have been utilized to pretrain BERT-like models, yielding several chemical language models [22, 23]. Similarly, protein language models are trained using protein sequences [24].

As we shall pretrain SciLMs on temporally organized corpora, our work is broadly related to training models adaptive to diverse domains (e.g., [25], [26]) and specifically related to the temporal aspect of language models. Some studies found that language models implicitly encode various external knowledge including space and time [27]. However, an increasing number of studies reported the temporal misalignment phenomenon, which means that models trained on text from one period can have performance degradation when tested on text from another. For example, [10] showed the effect of temporal drift on NER tasks in temporally-diverse tweets. Likewise, [11] quantified performance degradation due to temporal misalignment across domains and tasks. These studies corroborate the importance of customizing models to different time periods, which is the practice we adopt in this work. To mitigate the effect of temporal misalignment, existing studies suggested a few methods, including continuous pretraining [11, 12], adding year flags to training instances [28], and discarding outdated facts [29]. However, these methods are proposed to improve performances in downstream tasks, but it turned out that the improvement is limited [11].

Another strategy aims at achieving a desired outcome by directly editing a model itself, which involves altering a pretrained model in the weight space to enhance model performance in specific tasks [30, 13]. Current approaches to model editing focused on task arithmetic [30] and weight interpolation [31, 32]. In the context of addressing temporal misalignment, model editing through time vector as a special case of task vector was used to obtain an updated pretrained model for a specific period [13]. However, a critical limitation of time vector based model editing is that the language model used in the first place was pretrained on a time discorded corpus, which confounds the derivation of time vectors. Moreover, using such language model in the scientific domain may be especially problematic, as the corpus size grows drastically due to the exponential growth of science publishing, which means that later corpora would over-shadow earlier ones in determining model weights. By contrast, instead of using model editing, we obtain an updated model through continual training on a series of corpora organized temporally.

In this regard, our pretraining strategy can be considered as a special case of domain adaption [3], where each corpus in a separate period is treated as a “domain”. A natural consequence of domain adaption that was studied is representation forgetting: As a language model adapts to new domains, it forgets some knowledge learned from previous ones, leading to performance degradation [33]. However, in our work, we find that representation forgetting is highly dependent on tasks: We identify two rather similar tasks where for one of them, representation forgetting does manifest, but for the other the opposite of forgetting is observed.

Distinguishing from the majority of existing SciLMs that concern about downstream NLP tasks, our motivation of pretraining a series of SciLMs is to utilize them to understand quantitatively how science evolves over time through the lens of scientific text. Thus, our work is related to inquiry into the evolution of science, using diverse types of data including full-text and metadata like authorship and citations. [34] proposed dynamic topic models and applied them to analyze the temporal evolution of topics in papers published in the journal Science. Analyzing authors’ transitions between different topics over time, [35] identified distinctive stages in the history of computational linguistics. [36] developed a classifier to assign rhetorical functions (e.g., background, method, conclusion, etc.) to sentences in abstracts and demonstrated the ability to predict the rise and fall of scientific topics by tracking the trajectories of rhetorical functions appeared in different topics over time. [37] developed a classifier to label citation functions (e.g., background, motivation, etc.) and applied it to NLP papers to reveal the evolution of this field.

As the number of scientific papers has surged, researchers have increasingly opted to post their manuscripts on preprint platforms to share with the scientific community. arXiv is perhaps the first such platform, which was initially for physics and over time expanded to many other fields. It has gained significant popularity recently and archived nearly 2.4 million papers.

We download from the arXiv website the source files of all the $1,752,637$ papers posted between 1990 and 2020. Fig. 2a shows the field decomposition of papers by year, indicating that physics constantly dominates the corpus. In total, it accounts for 61.3% of the articles, followed by mathematics (20.2%). Computer science publications, although represent only 14.3%, have increased exponentially in recent years. Papers from the other fields—Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, and Economics—all together constitute less than 5% of the dataset.

To pretrain our models, we first need to prepare a full-text corpus extracted from the downloaded files, which are overwhelmingly LaTeX and PDF files. For LaTeX files, we use the LaTeXML tool [38] to convert them into HTML files, which facilitates our followup processing, as there are myriad author-defined commands and symbols in the LaTeX files. We then remove contents like figures, tables, and equations as well as metadata like author information, acknowledgment, and pagination, by striping away HTML elements corresponding to these contents. Next we replace certain markups with particular tokens, e.g., [EQU] for inline equations, [CITE] for citations, etc., which improves the clarity and utility of the text for training. As for PDF documents, we first use PyMuPDF to extract plain text and employ neatext to further clean elements like tables, emojis, and URLs, considering that PDF files often lack a clear document structure that is conducive to cleaning. We then filter out sentences with less than 20 characters and sentences that contain more than 40% of non-essential elements like punctuation, special characters, stopwords, emojis, and URLs. The code for data processing and training can be found in the https://github.com/jd445/TrainAnuualBERT.

After extracting text from LaTeX and PDF files, we use NLTK [39] for sentence segmentation and filter out sentences with less than three words. This completes our corpus preparation, and Fig. 2b presents the total number of sentences over time.

Next, we build our customized vocabulary. Existing language models use different tokenization methods to build vocabularies. The original BERT, for example, uses WordPiece, whereas RoBERTa uses BPE. Both tokenization methods consider subword as token, which has the advantage in representing unseen words by combining a list of existing tokens in the vocabulary. Subword tokenization, however, is not suitable for our purpose of examining temporal evolution of scientific discourse at the word level. We thus utilize whole words as tokens, an approach scatteredly taken in previous studies [40]. Particularly, we employ spaCy to tokenize all the abstracts from 1990 to 2020 into words and keep those with frequency $\geq 50$, resulting in a vocabulary with about $53,000$ words. We then add some special tokens including [CITE], [EQU], [FIG], [REF], and [SEC] to the vocabulary for ease of LaTeX markup representation. This tailored approach ensures that our vocabulary is appropriate for interpretations of terminology pertaining to distinct fields of study.

Fig. 2b presents the total number of tokens over time. Fig. 3 presents the Jaccard similarity matrix between the vocabularies of different BERT models (see § II-A for their details). We note that the similarities between BERT models and the other models (except BioBERT) are generally around 20%. BioBERT and BERT_case have similarity of one, since BioBERT adopts the vocabulary of BERT_case. The similarities among SciBERT, BiomedBERT, and ScholarBERT are approximately 30%, which is higher than that of BERT. This reflects differing vocabulary distributions between general and scientific domains. Our AnnualBERT exhibits lower similarities between the other models, which is attributed to the unique tokenization strategy.

The original BERT model is based on a bidirectional Transformer architecture and pretrained through two objectives: Masked Language Model (MLM) and Next Sentence Prediction (NSP). In MLM, $15\%$ of input tokens are masked, among which $80\%$ are replaced with the [MASK] token, $10\%$ are replaced with randomly selected tokens from the vocabulary, and the remaining $10\%$ stay unchanged. The objective is to predict masked tokens using cross-entropy loss. NSP predicts whether or not two masked sentences follow each other in the original text. Built on BERT, the RoBERTa model [41] improves the pretraining by eliminating the NSP objective, which has been found less useful, and by using dynamic masking, where masked tokens are changed for each epoch, as opposed to fixed masking used in BERT where masked tokens are fixed across epochs.

Here we adopt the RoBERTa implementation to pretrain from scratch our base AnnualBERT model, $\mathcal{M}_{2008}$, on the corpus formed by all the papers published until 2008, which has about 1.2 billion words. The architecture of our AnnualBERT matches with the BERT${}_{\text{base}}$ model, namely 12 hidden layers, 12 attention heads, and 768 hidden dimensions, totalling 110M parameters. During pretraining, we use the standard BERT tokenization approach, but incorporating our own customized arXiv vocabulary. We set the vocabulary size to $53,100$, the max_len parameter to $512$, and lowercase the corpus and the vocabulary. The pretraining process took around 108 hours to finish two epochs on a computing platform with 8 Nvidia A40 GPUs.

With $\mathcal{M}_{2008}$, we perform, for each year $t>2008$, continual training of $\mathcal{M}_{t-1}$ on the corpus formed by papers published in $t$ for one epoch, resulting in $\mathcal{M}_{t}$. For instance, a continual training of $\mathcal{M}_{2008}$ on the corpus based on papers published in 2009 yields $\mathcal{M}_{2009}$, which is then used to form $\mathcal{M}_{2010}$ using papers in 2010, in general following:

Through this approach, we allow each instance of our AnnualBERT to capture evolving language and terminology specific to that year’s publications, so that we can represent the knowledge of a specific year in a more sophisticated way. The corpora used for pretraining our AnnualBERT grow to 2.7 billion words by the time we extend the model to AnnualBERT₂₀₂₀.

To validate our AnnualBERT models, we experiment on 3 downstream NLP tasks: named entity recognition (NER), text classification (CLS), and relation classification (REL). In designing those experiments, we consider tasks using benchmark datasets and tasks designed specifically for the arXiv domain. For benchmark datasets, many existing ones for scientific text are from biomedical domains. However, texts from those fields are largely absent from our training corpus. Therefore, in our experiment we do not include biomedical related benchmark datasets. Instead, we collect and categorize datasets from computer science and multi-subject domains, given that the arXiv corpus used in our work tends to come from science and engineering disciplines. Particularly, for NER tasks, we use the following 3 datasets.

1. 1.

   The SciERC dataset [42] annotates entities, relations, and coreference clusters in 500 abstracts from 12 AI conference and workshop proceedings indexed in the Semantic Scholar Corpus. It includes a total of $8,089$ distinct named entities.

2. 2.

   The ScienceExam dataset [43] comprises entities from science exam questions aimed at students from 3rd to 9th grade, totaling $133,000$ entities.

3. 3.

   The ScienceIE dataset [44] provides $9,946$ entities appeared in 500 scientific paragraphs.

For CLS tasks, we use the following 3 standard benchmark datasets.

1. 1.

   The SciERC dataset [42] includes $4,716$ annotated sentences categorized for classification according to relations such as USED-FOR, FEATURE-OF, PART-OF, etc.

2. 2.

   The ACL-ARC dataset [45] comprises $1,941$ citation instances from 186 papers in the ACL Anthology Reference Corpus, with each instance annotated by domain experts with categories such as Future, Background, Motivation, etc.

3. 3.

   The SciCite dataset [46] encompasses $11,020$ citation instances from $6,627$ scholarly papers, with each citation classified into categories such as Background, Method, and Result Comparison.

While the 3 benchmark datasets are popular, they all involve sentence-level classifications. We therefore formulate another 3 CLS tasks that use entire abstracts of arXiv papers as input. They are predictions of:

1. 4.

   major category of a paper (8 categories including physics, computer science, etc.);

2. 5.

   sub-category of a paper (172 labels including Astrophysics of Galaxies, Artificial Intelligence, etc.);

3. 6.

   whether a paper spans more than one major category, which can be considered as an interdisciplinary paper, as assessed by the submitting author.

The last task is motivated by the increasing prevalence of interdisciplinary science [47] and its role in tackling important societal problems [48].

Finally, for REL tasks, we use the abstracts of two papers to predict whether they are in the same broad category.

We compare our AnnualBERT models with other scientific BERT models, including BioBERT [4], SciBERT [6], BioMedBERT [7], as well as the original BERT [49]. For our evaluations, we utilize the AnnualBERT model corresponding to the dataset’s year of release.

To ensure consistent comparisons across various BERT variants, we standardize the training configuration for each model. For the NER tasks, we adopt the NER model structure from SimpleTransformers, which incorporates a direct linear layer, and set the training parameters as follows: maximum input length of 512, training duration of 10 epochs, and batch size of 32. The best model parameters after each epoch are saved. For the CLS tasks, the vectors obtained using the [CLS] token pooling strategy are used as the input into a multilayer perceptron (MLP) for classification during the fine-tuning phase. The primary training parameters are set as follows: a training duration of 4 epochs, a batch size of 16, and weight decay at 0.01, and the best-performing model on the development set is saved every 100 steps. For the REL task, we use the dual tower structure [50]. We feed the abstracts of two arXiv papers into the identical BERT-like model and employ the [CLS] tag pooling strategy to obtain their respective feature vectors, $u$ and $v$, that effectively represent each abstract. We then concatenate $u$ and $v$ and their element-wise absolute difference $\left|u-v\right|$. This concatenated vector is then fed into a MLP layer for classification.

We conduct all the experiments on a workstation with two Nvidia A6000 GPUs.

Table II presents the experimental results. Firstly, our AnnualBERT does not perform well on the NER tasks, and interestingly, neither do the other two models pretrained from scratch (SciBERT and BioMedBERT). This might due to the tokenization strategy we use during the pretraining process. It may be also related to the fact that BioBERT, the best-performing model, is resulted from continual pretraining from the general Wikipedia and books corpus, which may have a better alignment with the benchmark datasets like the ScienceExam.

On the other hand, our AnnualBERT has the best performance in the CLS task on the SciERC dataset and comparable performances to the other specialized BERT models on the ACL-ARC and SciCite datasets. In general, the performance differences among these models in the three tasks are rather small. However, significant performance gaps become evident for the three domain-specific tasks, where our AnnualBERT performs the best. Particularly, for the sub-category prediction task, which involves classifying papers into one of the 172 sub-fields based on their abstracts, our AnnualBERT significantly leads the other models. Notably, SciBERT also has a comparable performance in this task, likely due to its training corpus containing a substantial number of computer science papers. Finally, AnnualBERT has a dominant performance in the cross-field paper identification task, indicating its superior knowledge representation ability for interdisciplinary papers that stretch multiple domains, while the other models, despite fine-tuned, struggle with this task. Taken together, these results demonstrate the validity of our AnnualBERT models and the effectiveness of domain-specific pretraining for domain-specific tasks, in consistence with prior studies [7].

Let $G=(V,E)$ denote the static citation network observed at 2020, where $V$ is the set of $N=\left|V\right|$ nodes and $E=\{(u,v)\}$ is the set of links indicating citation relationships between two arXiv papers $u,v\in V$. We study link prediction under the supervised learning setting. That is, we learn from data a model mapping from node-pair features to link missingness. Here we train classifiers using 5-fold cross-validation by forming datasets $\mathcal{D}=(E^{+},E^{-})$ that are composed of positive training examples as a subset of observed links $E^{+}\subset E$ and negative examples generated from sampling the same number of non-existent links $E^{-}\subset V\times V-E$.

While we have found that AnnualBERT adapts to new corpora, Fig. 4 in the meantime suggests that model weights are located close in the weight space, raising the question of whether we can interpolate models so that the interpolated ones $\mathcal{M}_{t_{0}}^{\Delta t}$ have similar performances with the “real” model $\mathcal{M}_{t_{0}}$ in prediction tasks. We derive the interpolated model $\mathcal{M}_{t_{0}}^{\Delta t}$ ($\Delta t\geq 1$) by “averaging” $\mathcal{M}_{t_{0}-\Delta t}$ and $\mathcal{M}_{t_{0}+\Delta t}$, treating it as a linear system:

We then compare performances for the 3 prediction tasks with same settings as described before.

Focusing on $t_{0}=2014$, we have 6 interpolated models by varying $\Delta t$, and Fig. 6 presents the average F1 scores of these models together with $\mathcal{M}_{2014}$ for the 3 tasks. We also display the $p$-values of the Mann-Whitney U tests comparing F1 scores of each interpolated model with $\mathcal{M}_{2014}$. Firstly, we observe that across the 3 tasks, all the interpolated models perform well and exhibit comparable performances with the real model. To further demonstrate that both $\mathcal{M}_{t_{0}-\Delta t}$ and $\mathcal{M}_{t_{0}+\Delta t}$ are necessary in creating the interpolated model, we note that using either of them would not generate similar performance as $\mathcal{M}_{t_{0}}$ (see Figs. 5a, c, d). Furthermore, we conduct additional experiments where we replace $\mathcal{M}_{2008}$ with a purely random model $\mathcal{M}_{2008}^{\text{random}}$ that has the same mean and variance as $\mathcal{M}_{2008}$, and interpolating between $\mathcal{M}_{2020}$ and $\mathcal{M}_{2008}^{\text{random}}$ yields significantly worse performances than $\mathcal{M}_{2014}$ for the 3 tasks (F1 scores are 0.66, 0.07, and 0.79 respectively, with all $p$-values below 0.05.).

The second observation from Fig. 6 is that as $\Delta t$ increases (interpolating between two models farther way temporarily), the performance gap between $\mathcal{M}_{2014}^{\Delta}$ and $\mathcal{M}_{2014}$ tend to increase, although lack of statistical significance. This may imply that the model weights encode temporal information via continual training, and it is feasible to edit the model to fulfil the desired task within a specific time frame.

Finally, having examined temporal adaptability and interpolation of AnnualBERT, we explore the root cause of these behaviors—training corpus—and study how it affects the learning and forgetting behaviors by designing experiments on synthetic datasets. In particular, we first create a new, shuffled corpus, where we randomly shuffle the sequence of the words in a sentence from the abstracts in 2020. We then rerun continual training of $\mathcal{M}_{2020}$ on this shuffled corpus for 5 epochs, resulting in a new model $\mathcal{M}_{2020}^{(\text{s})}$. For comparison, we also repeat the same procedure but on the original, unaltered abstracts to obtain $\mathcal{M}_{2020}^{(\text{o})}$. In the shuffled case, we force the model to adapt to a semantically unmeaningful corpus and consequently to forget the learned semantics, whereas in the unaltered case, the model reinforces the learning process. Next, we evaluate the extent of learning and forgetting in the same manner as in § VI-A, by sampling from shuffled abstracts a training set and a test set, denoted as $\mathcal{D}^{(\text{s})}$, and similarly from unshuffled abstracts $\mathcal{D}^{(\text{o})}$. We then test the classification performances of the two models on the two datasets and get the following accuracy matrix:

For dataset $\mathcal{D}^{(\text{o})}$, we quantify performance deterioration by calculating relative accuracy to the baseline:

Here baseline is the accuracy by random guessing the label:

where $N_{i}$ is the number of samples of class $i$ and $N$ is the total samples. For dataset $\mathcal{D}^{(\text{s})}$, the relative accuracy $\widetilde{A}^{\text{s}}$ is defined similarly.

Table V presents $\widetilde{A}^{\text{o}}$ and $\widetilde{A}^{\text{s}}$ for the 3 tasks. It indicates a persistent performance shift across tasks: The model $\mathcal{M}_{2020}^{(\text{s})}$ trained on a shuffled corpus under-performs on real data but over-performs on shuffled data, showcasing the model’s adaptation to the disordered text and exhibition of semantic forgetting. This phenomenon is most pronounced for subcategory prediction, where the model critically relies on the semantically meaningful text to learn the subtleties useful in identifying fine grained categories, the destroying of which causes the model under-perform severely. This result is also consistent with Fig. 5c showing that the model needs sustained training instances to have a reliable representation of a particular subcategory.