MatScIE: An automated tool for the generation of databases of methods and parameters used in the computational materials science literature

Currently, the majority of material science findings and information are stored in an unstructured format across numerous published articles. A typical article contains information on the material(s) studied, the method used, the computational software(s) used in the study, the simulation parameters and, finally, the outcome of the study. If we consider the scenario where we want to query the methods and parameters discussed in published material science papers, there is no easy and robust method to effectively filter and refine this information automatically. Manually going through the papers and finding out the methods used is an inefficient and tedious task. A potential solution can be to build a system that can automatically extract mentions of a method from any published article.

To solve these problems, we introduce Material Science Information Extractor (MatScIE), which is capable of extracting information about the material, code, parameter, method and structure from the published research article, along with providing a summary of the main research findings. Significant developments have been observed in the field of information extraction using machine learning and deep learning techniques. A very specific and widely used task is Named Entity Recognition (NER) that classifies (extracts) named entities in the text as per pre-decided classes or categories. It accepts as an input a sequence of tokens and identifies the spans in the input sequence that belong to one of the pre-decided categories. In our use-case, we attempt to extract the spans of text in a material science research article belonging to one of the five categories: material, code, parameter, method, structure. This enables us to adapt the NER framework for our task.

Popular NLP methods for the NER task are based on recent advances in deep-learning [1]. An important requirement in training a deep learning model is the availability of appropriate (and large) annotated data. We created a modest annotated dataset using 214 material science articles by labeling the text spans into the five categories.

In the first part, we trained standard sequential models on the supervised data to predict labels. We utilized word embeddings pre-trained in the material science domain. In the second part, we injected noise in the training dataset to increase the robustness of the model. Since injecting noise in the textual dataset is critical, we developed a novel procedure using a Relabeling and Mimicking model. The Relabeling model has a high recall, we use this model to inject noise in the dataset. We varied the amount of noise injected, and obtained results pertaining to different amount of injected noise. In order to derive a short summary level information from the published article, we trained a sentence classification model on an annotated dataset of 90 articles. Additionally we developed a web application that generates summary level information and token spans corresponding to each category.

The remainder of the paper is structured as follows. In Section 2, we describe past research on information extraction on scientific articles. We describe our annotated dataset in Section 3. Then we describe our proposed approach and evaluation metrics in Sections 4 and 5, respectively. We show the experimental results and sample outputs in Sections 6 and 7, respectively. Some more analysis on the output are presented in Section 8. We present the online interface in Section 9. Section 10 concludes the paper and gives directions for the future work.

Information extraction from scientific articles has been extensively explored recently. With the help of information extraction methods we intend to extract potential information out of the large body of scientific articles. Extracted information can provide an overview of the key-insights from scientific articles. Some of the major computational approaches in this regards include rule-based approaches, machine learning approaches like Naive Bayes classifier [2], support vector machines [3] and deep learning approaches. Most of the studies with regards to deep learning are performed on publicly available datasets like bc5cdr [4] and SCIERC [5]. One interesting work is this field is done by Luan et al. [5], that extracted entities, relations between entities and co-reference clusters in scientific articles using a multi-task setup. They applied their model on SCIERC dataset, that contains abstracts of 500 scientific articles from different domains. Furthermore they used their predictions to generate a scientific knowledge graph that can further be used to show analysis on scientific literature. Another popular work done in this field is that of Beltagy et al. [6]. This work uses a BERT model pretrained on a corpus consisting of scientific articles to improve performance on downstream scientific NLP tasks. Corpus used for pretraining the model mostly consisted of biomedical documents. Since our work is primarily based on material science domain articles, we generated an annotated dataset consisting of only material science articles and then trained deep learning models on the same.

There has been quite some work to extract Chemical entities. Some of the information related to chemical entities is already available in static databases. These databases map chemical information to relevant document with details like patents, literature etc related to the entered text. Below is a list of popular chemical databases :

• •

  PubChem [7]: It is an open chemistry database where researchers can add scientific data that others may use. It is maintained by NCBI [8]. It contains molecule names, substance descriptions and link to published documents related to the queried chemical compound.

• •

  ChemSpider [9] - It is a free database containing chemical structures with the feature to search by chemical names and chemical structures. It helps to find important data like literature references, physical properties, chemical suppliers etc.

• •

  SciFinder [10] - It is used to access information from selected Chemical Abstract Services(CAS) databases. It offers searches of authors name, related topic etc.

But all such databases are static and require constant updates from researchers.

One interesting work in this field is the OSCAR4 recogniser [11]. It build an n-gram model and then uses it in Bayesian classifier to classify whether a token belongs to “chemical” or “non-chemical”. The n-gram model is built with the help of a list of chemical tokens, that are obtained from a fixed dictionary and manually annotated documents. It then builds a Maximum Entropy Markov Model [12] by representing each token with a set of features, one of which is the probability that it belongs to a chemical domain with the predictions of the previously built n-gram model. Since the model is prepared from a fixed dictionary, more often than not it fails to capture the materials when represented in complex notations. Additionally, this model extracts only the chemical components from a text whereas we are interested in a much detailed extraction from the text.

ChemSpot [13] is another chemical extraction tool that uses Conditional Random Field(CRF) [14] to identify IUPAC and IUPAC-like names from a scientific text. This model also fails to fulfill our purpose for similar reasons.

Recently there are research studies being performed on material science domain articles which use deep learning models to classify tokens into material science categories. One such work is that of Weston et al. [15]. It uses BiLSTM with CRF to classify each token of the text. Dataset used in this paper contains 800 article abstracts which are annotated among the following labels - material, symmetry/phase label, sample descriptor, property, application, synthesis method and characterization method. However, the entity labels used in that paper is different from the labels in which we are interested in since we are dealing with papers that report density functional theory / first principle-based calculations.

Several researchers have focused on traditional machine learning models to explore textual features to perform several tasks like understanding material science languages, extract information or develop a knowledge graph. One such work was done by Tshitoyan et al. [16] who developed an unsupervised word embedding model to process text for identifying complex materials science knowledge such as the underlying structure of the periodic table and structure–property relationships in materials¹¹1We have used this embedding technique as a baseline in our experiment.. Another work was done by Buttler et al. [17]. They have explored the evolution of research workflow and how different machine learning algorithms (like Genetic Algorithms, Naïve Rules based Approaches²²2We have used Naïve Rule approach as a baseline etc.) would be used to design, synthesize different aspects of chemical and material science research areas. Hakimi et al. [18] focused on biomaterial text mining for retrieving relevant documents. Kostoff et al. [19] described techniques for data retrieval and discussed the desirability of conflating search terms. Another direction of work was to develop an automatic approach. Kim et al. [20] built a platform using variety of machine learning and natural language processing techniques to automatically retrieve articles and then extract materials synthesis conditions found in the text. Juan-Pablo et al. [21] presented how machine learning and natural language processing can help solving long timelines and low success probabilities of material science research and can accelerate the pace of novel materials development. Some researchers were also focused on exploring material science doamin relevant features. One such work was done by Goldsmith et al. [22]. They have shown how machine learning can be useful for aiding heterogeneous catalyst understanding, design and discovery. Dragone et al. [23] proposed a system that can evaluate chemical reactivity and detect for new reactions, rather than a predefined set of targets.

Few works were also done on material science synthesis procedure. The semi-supervised machine-learning classification model by Huo et al. [24] used a Dirichlet allocation model to cluster keywords into topics corresponding to specific experimental materials synthesis steps, such as “grinding” and “heating”, “dissolving” and “centrifuging”, etc. Guided by a modest amount of annotation, a random forest classifier can then associate these steps with different categories of materials synthesis, such as solid-state or hydrothermal. Some people explored semantic features of text - Mysore et al. [25]. Young et al. [26] experimented on data mining techniques in case of pulsed laser deposition of complex oxides. Mysore et al. [27] have built graph based automatic extraction model. Kononova et al. [28] have automatically extracted synthesis entries from scientific publications to gather a dataset of “codified recipes” for solid-state synthesis. Himanen et al. [29] worked on data driven approaches on material science articles.

The real complexity of our task lies in extracting only the entities used in the study rather than those that are just being mentioned. For example, consider the paper with title, “Temperature independent band structure of WTe2 as observed from ARPES”. This paper may mention many different materials but only WTe2 is being studied, and is the only material in which we are interested in. We observe that an article usually works on a much smaller subset of the materials than the ones actually occurred in the paper, while the remaining are just mentioned as previous work. In a normal NER task, however, we would be interested in extracting all the different materials mentioned in the paper. The same goes for all other classes.

In this section, we discuss the collection and annotation process of the dataset. We first discuss the dataset creation for the information extraction task, followed by the dataset for sentence classification task.

Our proposed approach consists of an information extraction and a sentence classification model, which we describe in this section. For information extraction, we utilize a sequence labeling approach, commonly used to identify the (named) entities of interest. Given a sequence of tokens (words), this model makes a judgement for each token in the sequence, as to whether it is part of any of the underlying entities of interest.

For example, the model should output that a token “Na2SO4” belongs to the class material and the sequence “Vienna Ab Simulo Package” belongs to the class code.

In the NLP literature, there are primarily three types of information that are utilized for the sequence labeling task: a) word representation, b) the context in which the word has occurred in the sentence, and c) shape of the token.

Word representation can be obtained by training a word-embedding model. We used the word2vec model with skip-gram architecture to represent the word in vector as discussed by Mikolov et al. [31]. Skip-gram architecture utilizes the distributional hypothesis, i.e., ‘similar words tend to appear in similar contexts’, by utilizing the context words to learn the representation of a given word. It is usually trained using a large corpus containing texts of similar domain. Since our dataset is much smaller, we used pretrained word2vec vectors, using a corpus containing 650K material science published papers, from the previous work of Kim et al. [32].

Local context of the word occurring in the sentence proves useful in those cases where the word vector corresponding to a given token is not available. That is very common because of new materials and other tokens in the literature. Consider the following sentence in our dataset : We performed first principles calculations on ___ doped with Mn, focusing on different aspects. From the context words, it is quite evident that the word that appears in place of blank belongs to the class of material. To utilize the context information, we treat each sentence as a sequence of words that are represented using embedding. Since it is a sequence labelling task, we use recurrent neural networks (RNN) to learn the embedding of the context that the sentence represents [33].

RNN are known to suffer when the sequence is long as in that case it cannot utilize context words that are not very close to the target word. Because of this, we used Long Short Term Memory (LSTM) [34], which is a variant of RNN. Forward LSTM can capture the sequence of the tokens appearing before the current token, while backward LSTM captures the sequence of the tokens appearing after the current token. While training we used Bidirectional LSTM (Bi-LSTM) [1], which is basically a concatenation of embedding obtained by forward LSTM and embedding obtained by backward LSTM to achieve this purpose. For each element in the input sequence, each LSTM layer computes the following

where $i_{t}$ represents the input gate, $f_{t}$ represents forget gate, $o_{t}$ represents output gate, $\sigma$ represents sigmoid function, $w_{t}$ represents weight of respective gate(x) neurons, $h_{t-1}$ represents the output of previous block at time step $t-1$, $x_{t}$ represents input representation at current time-step, and $b_{x}$ represents the biases for the respective gates. The equations to obtain the final output is

where $c_{t}$ represents the cell state at time-step $t$ and $\tilde{c_{t}}$ represents the candidate for cell state at time-step $t$. The output obtained at each time-step $t$ is passed through a fully connected linear layer to perform the task of classification among six classes. We applied softmax function to the logits obtained from the linear network to obtain the probabilities for each class.

Chemical formulae are often represented by an inorganic notation, for example - sodium sulphate is represented as Na2SO4. We can observe that this token has a fixed pattern where the first letter is uppercase, followed by lowercase characters and numbers. There are tokens with distinct shapes available in our dataset. Also, we notice that the prefix and suffix of a token play a major role in deciding the class to which it belongs. To detect the shape of the tokens, we treat each token as a sequence of characters and then apply Bi-LSTM. In the end, we obtain the character level embedding of the token.

After we obtain the embedding, we need to build a tag encoder that can rightly understand the class belonging to the token. For this purpose, we use a Conditional Random Field (CRF) classifier [35].

Peters et al. [36] showed that ELMO embedding of a token can give better performance as compared to Bi-LSTM embedding in general. In ELMO, each token is assigned a representation that is a function of the entire input sentence. ELMO embeddings are deep, in the sense that they capture the output of all the internal layers of LSTM. We used pretrained ELMO embedding for material science scientific articles. The input to the Bi-LSTM model is thus a concatenation of pretrained word2vec embedding, character embedding, and pretrained ELMO embedding (see Figure 3).

The basic workflow of the model is shown in Figure 4.

We tried quite a few variations of the basic approach during our experiments, which we describe below.

We view the task of NER as classification of a token with one among the 5 classes that we are interested in. If some token does not belong to any such class, then we treat that token to belong to a special class denoted by “O”. As with any other classification tasks, we use precision, recall, and f-measure to measure the performance of our model. For each document we created two sets of tokens for each category - one set contains the actual annotations or ground truth (set $GT$) and the other set contains the predicted token (lets call it set $Pred$). Then we compared the tokens between the sets and used it to compute the metrics.

We define the precision (P) of a particular class (c) by the expression

where $GT$ represents the set that are actually labelled with $c$ and $Pred$ represents the set that is predicted as $c$. Precision calculates the fraction of the obtained results that are relevant. We define recall (R) by the expression

Recall calculates the fraction of total relevant results that re correctly classified by the system. In general, we want the system to output both high precision and high recall. In order to measure the accuracy of the system we use F1-score which is defined by the harmonic mean of precision and recall i.e

A model with a high F1-score is considered better as compared to a model with lower F1-score. In our task, we are concerned with spans of tokens. In order to calculate how well the model is performing, we have followed combination of two different approaches: a) exact match and b) partial match [37]. Revised expression to calculate precision and recall is given by

For exact matching, $\alpha(a,p)$ is true if the $a$ and $p$ represents exactly the same spans, whereas in partial matching, $\alpha(a,p)$ is true if there is some overlap between the spans of $a$ and $p$.

In our experiments, we calculated the average precision (P), recall (R) and F1-score along with their respective standard deviations ($\sigma$) where $\sigma$ values were calculated as per Equation 12. We have utilized string match as an accuracy metric. In other words, for each article, we compared the list of annotated tokens for each class with the list of predicted tokens for that class. This is significantly different than that of usual NER setting where we are interested in finding the class level metrics for each sentence.

The task of extracting the entities from material science articles shares some similarities with the Named Entity Recognition task (NER) and also with question-answering task [38]. Thus, we consider state-of-art models for these tasks as the baseline. For NER, we followed the work of Beltagy et al. [6] and used Scibert which is pretrained on scientific publications. Since our task is specific to the material science domain, we used unsupervised domain adaptation techniques to obtain a BERT model specific to the material science domain as discussed by Han and Eisenstein [39]. For the question-answering task, we used the BERT multi-answer question answering model showed by Devlin et al. [40]. Along with the above two models, we also used a baseline that predicts based on the frequency of occurrence in the training set. Thus, in the test dataset, if a span is present in the training dataset with a class annotation, it is given that particular label. We call this as the Naïve rule-based approach. Another baseline, we have taken is the unsupervised word embedding based method (Word2Vec), developed by Tshitoyan et al. [16]. The results for Naïve rule-based approach and Unsupervised Word embedding based method (UWE) [16] are listed in Table 1. The results for BERT question answering model and BERT domain adaptation model are listed in Table 2. We have performed 5 fold cross validation test for each model to report average precision (P), recall (R) and F1-score (F1) along with their standard deviations ($\sigma$) across five folds.

Next, we compare the results of the per-line training (with only positively annotated lines) and all lines per-section training setup. In Section 4.2 we mentioned that we used the section-wise training setup. In Table 3, we compare the results of these two training setups. The model trained using the section-wise training setup outperforms the model trained on a sentence-wise training setup.

In Section 4.3 we mentioned that using only positive-annotated lines for training can lead to information loss. Including all lines per section (positive and negative) in the training, dataset leads to comparatively better scores as seen in Table 3. Since we have already established that section-wise training is better, we adopted this approach in both the models. Only the training dataset for them is different. Results in Table 3 show that the model trained on only positive-annotated lines do have good recall scores, but low precision scores and the model trained on all lines have good precision scores but low recall scores. Also, except material for all other classes, the later model performs better than the former in terms of F1 score.

We have already explained the Relabeling and Mimicking model in detail in section 4.3. The model trained on only positive-annotated lines (section-wise) is used as the Relabeling model due to its high recall scores. Relabeling model is similar to the per-section setup with only positively annotated lines. In Table 4 we compare the results of the Relabeling model and the Mimicking model. The mimicking model achieves a better macro F1-score as well as has the best F1 for all the classes. We have tuned several hyper-parameters of the model to get the optimum results. In our case, we have set “$learning\_rate$” = “0.001”, “optimizer” = “sgd”, “dropout” = “0.5”, “$hidden\_dim$” = “200”, “$num\_epochs$” = “120”, “$batch\_size$” = “8” to get the best F1-score.

We use the Relabeling model to add noise in the training dataset of the Mimicking model. Keeping the ground truth labels intact, we add some percentage of the extra labels predicted by the Relabeling model. The amount of noise added per class is shown in Table 5. For example, if 50 % noise is added to the material class, it means half of the extra predicted labels of this class are added into the training data randomly. The amount has to be balanced because increasing noise results in an increase in the recall but a decrease in precision. The optimum noise level has to be determined experimentally. We used the validation set results to fine-tune the noise level for each individual classes (as shown in Table 5).

We have shown summary of F1-scores of all models along with their 95% confidence interval (error bars are shown in black) in Figure 6. It is seen that the Mimicking model achieves the best F1-score performance among all approaches - Naïve rule based (NB), Unsupervised Word Embedding (UWB), BERT Question Answering (BERT QA), BERT domain adaptation (BERT DA), Per-line, Per-section (All lines), Relabeling and Mimicking model.

We observe that all the competing models fail to identify micro details of the structures properly. Thus, the precision, recall and F1-scores for structure are lower than other classes. This is due to the fact that systems correctly identifies macro details of structures like “tetragonal”, “cubic” etc. but not able to match micro details of the structures like - “cubic perovskite Pm30304m”.

In this section we present some of the actual outputs obtained from the best model for the information extraction task.

Tables 8 and 9 show the output obtained by applying the best model on material science papers titled, “Single pair of Weyl fermions in thehalf-metallic semimetal EuCd2As2” and “Electron-phonon superconductivity in CaBi2 and the role of spin-orbit interaction”, respectively.

The proposed model can be used to observe various statistics over time. For instance, what are the main methods and codes being used over time? To answer this question, we run our best model on the 10K material science articles from arxiv (see Section 3.1). Figures 7 and 8 show the distribution of code and method, respectively over published articles in the last ten years. We can see from the Figure that “VASP” is the most popular code of the decade followed by “QUANTUM ESPRESSO”. Both codes have a common trend, their use has grown over the years, whereas the use of post-processing packages, such as “Phonopy” and “Boltztrap”, has fluctuated over time. For the method tag, we see that “GGA” appears the most followed by “DFT” and “PAW”. This is indeed true that most of the researchers in the field of computational materials science use VASP as their primary choice as code and the most popular approach is PAW-PBE. We recognized that “eV” is the most popular parameter followed by “K-point”. Therefore, we also would like to point out that this utility would be useful to track the choices of the researchers in this field over the time globally.

To be able to make the proposed methods accessible and usable by the community, we have created the online interface:

server: http://34.69.41.173:3333/.

This paper explored the task of extracting relevant entities from material science publications. We annotated 214 material science articles with five classes - material, code, method, structure and parameter and used the annotated data to train a deep neural network (see Section 4). We applied our trained model to obtain entities from a set of 10K material science articles that are published in arxiv over the last ten years. Additionally, we trained a sentence classification model to extract sentences appearing under abstract section that are related to the results. This information helped us to generate a short summary level information of the published article. By using our model, we pulled out statistics as shown in Section 8 giving an insight of the usage of methods and codes in a span of ten years. We created an online interface that lets us obtain the predicted entities from an uploaded published article (see Section 9). Using our tool, one can index the material science documents according to the material used or parameter used for the specific methods.

Most of the sentences in an article will not contain any entity that we are interested in. Passing these sentences to the system is of no use since the model will not output any entity from those. One possibility is to automatically identify relevant sections of the text data and pass as input only the sentences that are contained in those sections. In order to achieve this, we need to first classify each section of the document as whether it is relevant or not. As a future work, we can build a classifier that can classify sentences or sections of text as relevant or not, and then feed this input to the model for prediction. Currently, we are only extracting text information from the PDF documents. A further improvement can be obtained if we extract figures and tables from published documents. Material science documents contain useful information about materials and the output obtained from those materials using different methods in the form of tables. We expect most entities to appear in the table section. Hence extracting those information will enable us to collect more information. An additional feature can be to correctly identify the name of the parameter for every span that is predicted as the class parameter. From the dataset, it is clear that the parameter name occurs in the same sentence of the predicted token. We may use POS tags or dependency trees to accurately output the parameter name as well.

The code and data are available in https://github.com/TeamMatSciE/MatSciE.