Classification and Clustering
of arXiv Documents, Sections, and Abstracts,
Comparing Encodings of
Natural and Mathematical Language

Since Machine Learning methods require a formal (abstract mathematical) representation, natural language has to be converted into word vectors. This is typically done via Word2Vec (DBLP:journals/corr/abs-1301-3781, 15) using Bag-of-Words (BOW) or Skip-Grams (SG), or encoding term-frequency (tf) or term frequency-inverse document frequency (tf-idf). Whole documents or document sections can be represented, i.a., by Doc2Vec (DBLP:conf/icml/LeM14, 9) features, which are learned using (Deep) Neural Networks (DBLP:journals/cim/YoungHPC18, 36). Vectors of words or documents generated by Word2Vec or Doc2Vec were observed to be semantically close with respect to algebraic distance metrics and can be used to reconstruct linguistic contexts of words (DBLP:conf/nips/MikolovSCCD13, 14). This enables comparisons of the semantic content of documents, e.g., for Recommender systems or text-topic classification.

The Mathematical Language Processing (MLP) project (DBLP:conf/mkm/PagelS14, 19) was introduced as an attempt to disambiguate identifiers occurring in mathematical formulae. Retrieving the natural language definition of the identifiers using Part-Of-Speech (POS) tags combined with numerical statistics for the candidate ranking yielded high accuracies of around 90%.

The Part-Of-Math (POM) tagging project (DBLP:conf/mkm/Youssef17, 37) also aims at math disambiguation and math semantics determination for the enrichment of math expressions. Scanning a math input document, definitive tags (operation, relation, etc.) and tentative features (alternative roles and meanings) are assigned to math expressions using a semantic database that was created for the project.

Furthermore, a comprehensive Math Knowledge Processing project was started (DBLP:conf/mkm/YoussefM18, 38) to explore sequence-to-sequence translation from LaTeX typesetting to MathML markup using Math2Vec encodings of the formulae. A dataset of 6000 papers has been collected to be used as training and testing data for the semantification of the identifiers.

We encoded the retrieved text and formulae using the TfidfVectorizer from the Python package Scikit-learn (DBLP:journals/jmlr/PedregosaVGMTGBPWDVPCBPD11, 20) and Doc2Vec model (DBLP:conf/icml/LeM14, 9) from the Python package Gensim (vrehuuvrek2011scalability, 23).

After creating a LabeledLineSentence iterator for the vocabulary, the model was built with size=300, window=10, min_count=5, workers=11, alpha=0.025, min_alpha=0.025, iter=20 and trained 10 epochs with model.alpha-=0.002.

Table 1 lists the encodings that deserve further explanation. The surroundings encoding uses text surroundings (within +-500 characters, excluding stopwords and letters) of single identifiers as their putative meanings.

Summing up, we varied the following experimental parameters: 1) encoded data types or batch size (documents, sections, abstracts, summarizations), 2) encoded data features (text, math), 3) type of math encoding (op, id, opid, surroundings), 4) encoding method (doc2vec, tf-idf), 5) classification/clustering algorithm (performance, runtime).

First, we determined the correlation of the cosine similarity (inner product) between text and math encodings. For each document, section or abstract, we calculated the similarities with all the other docs/secs/abs in both text and math encodings (different vector spaces) seperately. This means, we investigated whether if two documents are similar in their text encoding, they are also similar in their math encoding. Table 2 lists the results of our comparison. The low correlations indicate that in principle, the independence of text and math encodings leave a potential for improvement of ML algorithms by combining the two, which was explored. Besides, it suggests that in a Recommender System for STEM documents, it will be beneficial to provide the user with weighting parameters for text and math (if relatively uncorrelated), to customize the recommendations.

Using the selection of 4900 documents, 3500 sections, and 1400 abstracts from the arXiv, we compared the influence of text and formulae on the performance of a subject class [’math’, ’physics’, ’cs’, etc.] classification. We subsequently clustered the doc/sec/abs vectors; for the KMeans, Agglomerative, and GaussianMixture cluterers, we fixed the number of clusters to 14 (= the number of labeled classes), while for the Affinity, MeanShift, and HDBSCAN clusterers, no number of clusters was fixed. The encodings secText_tfidf and sec2vecMath_surroundings needed a PCA dimensionality reduction before the clustering with MeanShift and GaussianMixture was possible.

We used 10-fold cross-validation¹¹1We observed that the split k had only a small impact on the result., while comparing the accuracy, purity, and relative runtimes of selected single or ensemble classifiers and clustering algorithms (with their respective default metrics) with or without fixed cluster number, provided by the Python package Scikit-learn (DBLP:journals/jmlr/PedregosaVGMTGBPWDVPCBPD11, 20).

For the classification, we calculated the accuracy as the number of correctly classified samples divided by the sample size and averaged over all splittings of the cross-validation.

For the clustering, we compared the clusters to the labeled classes, calculating the cluster purity as the number of data points of the class that makes up the largest fraction of the cluster divided by the cluster size and averaged over all clusters.

Table 3 shows the classification accuracies of the individual classification algorithms using text or math encodings of the documents, sections, and abstracts.

The best encoding for docs, secs and abs is always Text_tfidf. The most accurate algorithm is MLP (Multilayer Perceptron, with hidden layer size 500), except for docs where LinSVC yields a slightly higher maximum value. The fastest algorithms are kNN and Random Forest. For kNN and DecTree there is a high discrepancy between the values of doc2vecText and docText_tfidf encodings. While for text the tf-idf encoding is better, for math it is the doc2vec encoding, with the exception of the surroundings encoding which is, even though connected to mathematical identifiers, effectively text.

An overall comparison of x2vec and tf-idf including both text and math shows that the former outperforms the latter with mean(doc2vecX, sec2vecX, abs2vecX) = (40.2, 37.0, 28.0) mostly greater than mean(doc_tfidf, sec_tfidf, abs_tfidf) = (35.2, 35.5, 30.0), and mean(2vec) = 35.1 $>$ mean(tfidf) = 33.6 summarized. The mean of the means is decaying from docs (38.1) to secs (36.4) to abs (29.0). The surroundings encoding (especially with tf-idf) is better than the math encodings of operators (op), identifiers (id) or both (opid). Given that mean(doc_text, doc_math, doc_textmath) = (63.4, 28.3, 51.8), and mean(sec_text, sec_math, sec_textmath) = (59.7, 27.7, 47.9), it is striking that for the classification, the text encodings yield better results than the standalone math encodings.

We tested some other algorithms that are not listed: Gaussian Naive Bayes yields accuracies of mostly less than 10%; Multinomial Naive Bayes could not be carried out on the text vectors due to the negative values of their continuous distribution.

Table 4 shows the cluster purities of the individual clustering algorithms using text or math encodings of the documents, sections, and abstracts. The best encodings are doc/sec2vecMath_surroundings and abs2vecMath_opid. The most accurate algorithms are 1) GaussianMixture (highest mean and maximum), MeanShift (highest maximum), Affinity (highest max and mean). GaussianMixture is the best algorithm with a fixed cluster number, while Affinity and MeanShift are the best algorithms without a fixed cluster number. The fastest algorithm is HDBSCAN. Only for the mean of docs, text yields the highest value. For the other mean and maximum values, math are better than text encodings.

A comparison of x2vec and tf-idf shows that the former outperforms the latter with mean(doc2vecX, sec2vecX, abs2vecX) = (56.3, 43.4, 53.1) $>$ mean(doc_tfidf, sec_tfidf, abs_tfidf) = (51.5, 36.5, 47.3), and mean(2vec) = 50.9 $>$ mean(tfidf) = 45.0 summarized. For abstracts, the math encodings yield better results than the text encodings with mean(abs_text,abs_math) = (44.8, 55.6). However, given that mean(text, math, textmath) = (51.2, 48.2, 52.0), all in all, also for the clustering, the text encodings yield better results than the math encodings, but the combination of text and math slightly outperforms the other.

We tested some other algorithms that are not listed due to poor performance or exceedingly large runtimes (e.g. Spectral Clustering, DBSCAN).

We carried out a human expert²²2We chose one physicist, able to separate all subject classes. classification of 10 examples from each of the 14 subject classes for comparison to our algorithmic results. From 140 in total, 85 - i.e. 60.7% were correctly classified.

Figure 1 shows a comparison of the classification confusion matrix. The human classifier (left) is outperformed by the computer classifier (right) with lower diagonal and higher off-diagonal values.

Both the computer and human classification confusion show that some categories like ’physics’ should be disposed and distributed to the respective specializations (’cond-mat’, ’hep-ex’, ’nucl-ex’, etc.).

For the NTCIR and SigMathLing arxiv datasets, we could not find a baseline for our experiments. However, we were able to reproduce the results using the script of Schöneberg et al. (DBLP:conf/mkm/SchonebergS14, 28) on 942337 ($\approx$ 1M) tf-idf encoded abstracts from the zbMath database (zbMATH, 40). Employing a multi-label Logistic Regression (LogReg) classifier on the 63 top level Mathematical Subject Classes (MSC), we could outperform the 53% classification accuracy of the baseline  (DBLP:conf/mkm/SchonebergS14, 28) by 27% (to 80%) for the prediction of the first two top level labels.

Furthermore, we tested the effect of the number of predicted labels on the classification accuracy to find a strong decrease predicting more labels.

The results show that predicting more than three labels per abstract does not yield reasonable results. This is why we rather concentrated on single-label classification.

In this paper, we discussed how methods of Natural Language Processing (NLP) and Mathematical Language Processing (MLP) can be combined to enable the use of Machine Learning (ML) in Information Retrieval (IR) applications on documents with mathematical content. We first provided a short review of MathIR, NLP and MLP and the current state of research in text and math classification and clustering. Subsequently, we introduced the employed datasets of mathematical documents and described encodings for their text and math content. We investigated the correlation between text and math similarity. Finally, we presented and discussed the results of a classification and clustering of 4900 documents, 3500 sections, and 1400 abstracts from the arXiv preprint server (arxiv.org).

The correlations between text and math (cosine) similarity (Table 2) were relatively low (mean = 0.17, max = 0.27), motivating us to treat text and math encodings as separate features of a document. While for the classification, the Text_tfidf encoding was outperforming the others, for the clustering doc/sec2vecMath_surroundings and abs2vecMath_opid encodings are the best. For both classification and clustering, the x2vec encodings yielded better results than the tf-idf encodings and text outperformed math encodings. However, for the clustering, the combination of text and math slightly outperforms the separate encodings.
All in all, our research questions were answered as:

1) Combining text and math encodings does not improve the classification accuracy, but partly the cluster purity of selected ML algorithms working on documents, sections, and abstracts.

2) On the whole, the doc2vec encoding outperforms tf-idf encoding. The most accurate classification algorithm is a Multilayer Perceptron (MLP), while for the clustering, the highest maximum and mean values of the purities are divided among GaussianMixture, MeanShift, and Affinity Propagation. The fastest algorithms are k-Nearest Neighbors, and Random Forest classifiers, and HDBSCAN clustering.

Why did the use of mathematical encodings not significantly improve classification accuracy? We suspect a low inter-class variance of the math encodings due to a large overlap of the formula identifier namespaces. For example, the identifier $x$ occurs very often in many subject classes, but with different meanings. Documents from different subject classes often have similar sets of identifier symbols. Therefore, we expect that disambiguation of the identifier semantics by annotation would increase the vector distance between subject classes and possibly increase classification accuracy. There are two ways to tackle the identifier disambiguation. It can be done supervised with or unsupervised without the quality control of a human. In the following, we present our results of unsupervised semantification using three different sources. Furthermore, we shortly discuss our ongoing endeavors to additionally perform supervised annotation in the future work section.

In an attempt to increase the classification accuracy compared to the previously presented math-encodings, we tested a conversion from math (identifier) symbols to text (semantics). We semantically enriched the text of the $14\times 350=4900$ documents from the SigMathLing arXMLiv-08-2018 dataset by identifier name candidates provided from three different lists. These were previously extracted from the following sources:

1) arXiv: Identifier candidate names for all lower- and upper-case Latin and Greek letter identifier symbols appearing in the NTCIR arXiv corpus³³3http://ntcir-math.nii.ac.jp/data/ that was created as part of the NTCIR MathIR Task (DBLP:conf/ntcir/AizawaKOS14, 2). The candidates were extracted from the surrounding text of 60 M formulae and ranked by the frequency of their occurrence;

2) Wikipedia: Identifier candidate names extracted from definitions in mathematical English articles, as provided by Physikerwelt⁴⁴4https://en.wikipedia.org/wiki/User:Physikerwelt;

3) Wikidata: Identifier candidate names retrieved via a SPARQL query⁵⁵5https://query.wikidata.org for items with defining formula containing the respective identifier symbol.

For each source, the candidates were extracted, ranked by the occurrence frequency of the respective identifier symbol/name mapping, and dumped to static lists. The encodings with semantic enrichment by the top 3 ranked identifier name candidates outperform all other mathematical encodings listed in Table 3. We will discuss an extension of the experiment to supervised semantification in our future work.

In this paper, we presented classification and clustering baselines on the SigMathLing arXMLiv-08-2018 and NTCIR-11/12 MathIR arXiv datasets. Since, so far we were not able to significantly outperform the text encodings by math encodings, we call out for a Formula Encoding Challenge. The aim is to find a suitable math encoding that outperforms or enhances the text classification.

We now outline some future directions and experiments.