Link Prediction for Wikipedia Articles as a Natural Language Inference Task

Link prediction, a critical issue in network-structured data [3], was initially introduced as a supervised learning task by Al Hasan et al. [4]. They also identified key performance features within the supervised learning framework. Building upon the concept of analyzing user-item interactions as graphs, Huang et al. [5] utilized link prediction techniques from the recent network modeling literature to enhance collaborative filtering recommendations. In contrast, Liu and Lü [6] proposed a method based on local random walk, which achieves competitive or even superior predictions compared to other random-walk-based approaches while maintaining lower computational complexity. Another approach suggested by Trouillon et al. [7] involves solving the link prediction task through latent factorization, enabling scalability to large datasets with linear space and time complexity. Additionally, the heuristic learning paradigm for link prediction was explored by Zhang and Chen [8]. Furthermore, Negi and Chaudhury [9] framed the link prediction task in heterogeneous networks as a multi-task metric learning task.

In the field of artificial intelligence, inference has always been a prominent topic. While significant advancements have been made in automatic methods for formal deduction, the Natural Language Inference (NLI) task has seen comparatively slower progress. NLI refers to the task of determining the validity of inferring a natural language hypothesis ($h$) from a natural language premise ($p$) [10]. Bowman et al. [11] introduced the Stanford Natural Language Inference corpus, which consists of labeled sentence pairs created by humans engaged in a novel task based on image captioning. This corpus is freely accessible and serves as a valuable resource. Wang and Jiang [12] proposed a specialized architecture based on long short-term memory (LSTM) networks for NLI. Their approach utilizes a match-LSTM that conducts word-by-word comparisons between the hypothesis and the premise, which enhances the performance of NLI. Chen et al. [13] demonstrated that carefully designing sequential inference models using chain LSTMs can surpass previous models in terms of performance.

The task involves predicting the presence or absence of an edge between a pair of nodes $(u,v)$. In this competition, our focus is on link prediction for Wikipedia articles. To be more precise, given a sparsified subgraph of the Wikipedia network, the objective is to determine whether a link exists between two pages, $u$ and $v$, in the context of Wikipedia.

The dataset used in the contest is extracted from Wikipedia, where graph nodes are annotated with text. The data of the DSAA-2023 Competition include the following:

• •

  train.csv file: It contains pairs of nodes along with an indication of whether there is an edge between them. The file has four columns: $id$, $id1$, $id2$, and $label$. The id column represents the pairing identifier, $id1$, and $id2$ are the node IDs, and the label column declares the presence or absence of an edge (0 or 1).

• •

  nodes.zip file: This file is a compressed version of nodes.tsv and contains information about each node. It consists of two columns: id (node identifier) and text (textual description of the node).

• •

  sample_submission.csv file: This demo submission file has two columns: the pair ID and the corresponding label ($0$ or $1$).

The statistics of the training set labels after combining train.csv and nodes.tsv files together are presented in Table I.

The dataset consists of Wikipedia objects’ main content and CSS codes within Wikipedia. However, these CSS codes are considered noise and are not required for training. Consequently, it is crucial to remove them. To accomplish this, we employ two algorithms, the Balance Curly Bracing and Remove Double Curly Bracing Algorithms, Algorithm 1 and Algorithm 2, respectively, to effectively eliminate these unnecessary codes. Punctuation marks, such as periods, commas, question marks, exclamation marks, and others, and redundant spaces can introduce noise and disrupt the natural flow of text during training. Therefore, we apply the Regex library³³3https://docs.python.org/3/library/re.html to remove all of them. Our preprocessing techniques for this task of the DSAA-2023 Competition are summarized as follows: Balancing curly braces, removing double curly braces, removing all punctations, and removing redundant spaces.

Our approach draws inspiration from recent successes in natural language understanding tasks, particularly in pre-trained language models. We posit that a link between two Wikipedia articles can be treated as a premise, and predicting the link’s existence is akin to determining the logical relationship between the premise and a hypothesis. We encode the content of the two articles as the premise and the hypothesis, respectively, and utilize this NLI setup for link prediction.

In sentence pair classification, the model takes two sentences as input and aims to determine their relationship. It learns to classify the pairs into different categories based on the desired task. Link prediction for Wikipedia Articles task is to predict whether there is an edge or link between the two given nodes.

In this paper, we have implemented a Sentence Pair classification model based on XLM-Roberta, a well-known architecture introduced in the work of Conneau et al. (2020) [14], which is widely used for Natural Language Inference (NLI). Our implementation is tailored for generic and specific Wikipedia article link prediction tasks. Additionally, we incorporate a linear prediction layer on top of the XLM-Roberta architecture to calculate the final output. The details of our proposed approach’s architecture are thoroughly presented in Figure 1.

We followed fairly standard practices for fine-tuning, most of which are described in [15]. We use a batch size of 128, a maximum token length of 128, a learning rate of 2e-5, and AdamW optimizer with an epsilon of 1e-8.

We empirically SentencePair classification model for our system using simpletransformers⁴⁴4https://simpletransformers.ai/ (ver 0.63.11). We trained our model on 1$\times$RTX4090 GPU on the Vast.ai⁵⁵5https://vast.ai platform for four hours. Ten seconds is needed for predicting the test set of this dataset.

In this section, we describe our experiments and results for Link Prediction task. Only the macro F1-score is used for evaluation. Experimental results are presented in Table II

By implementing robust and efficient data preprocessing techniques tailored explicitly for cleansing comments obtained from Wikipedia, we aimed to enhance the overall data quality and substantially improve extracting relevant information before training the model for link prediction tasks. Combining these pre-processing techniques with the sentence pair classification model resulted in a notable performance improvement compared to scenarios where such techniques were not applied.

Our proposed approach achieved remarkable results, with a macro F1-score of 0.99996 for the public test and a perfect score of 1.00000 for the private test. These exceptional scores indicate the effectiveness of our methodology in accurately predicting links within the given context. In terms of ranking, we secured the 8th position on the public test and an impressive 3rd position on the private test.