YES SIR!
Optimizing Semantic Space of Negatives with Self-Involvement Ranker

Neural document ranking models compute relevance scores for query-document pairs based on neural networks. It is critical for them to learn good representation of queries, documents and ranking functions.

We can categorize neural ranking models to representation-focused models and interaction-focused models (Trabelsi et al. 2021). The representation-focused models extract good feature representation from input data. (Huang et al. 2013) first utilizes deep neural networks to extracts features from query and documents independently. To capture local context, (Shen et al. 2014; Qiu and Huang 2015) utilize convolutional neural networks instead of feed-forward networks. Also, recurrent neural networks are used in (Wan et al. 2016a) to capture richer context. In contrast, the interaction-focused models aim to capture matching signals from interactions of query and document tokens. (Guo et al. 2016) performs term matching using histogram-based features. (Wan et al. 2016b; Pang et al. 2017) capture matching signals with gated recurrent units, while (Dai et al. 2018; Hui et al. 2017) use convolutional neural networks.

Neural ranking models are typically trained using the Learning to Rank (LTR) (Huang et al. 2013) framework. LTR framework can be divided to three categories based on the training objectives: pointwise, pairwise and listwise (Liu 2011).

The great success of pre-trained language models inspires researchers to utilize pre-trained models for document ranking. A straightforward way is to use PLM, such as BERT or ELECTRA (Clark et al. 2020), to accomplish ranking tasks.

To further enhance models’ capacity on IR, researchers have proposed several pre-trained IR models, whose pre-training tasks are highly relevant to downstream IR tasks. $\text{Transformer}_{ICT}$ (Lee, Chang, and Toutanova 2019) designs pre-training tasks to explicitly explores the target context based on the query sentence. (Chang et al. 2020) uses random sentence from Wiki as the query to retrieve the related passage from same page or from the hyper-linked page. Ma has explored the representative words prediction (Ma et al. 2021a) for ad-hoc retrieval and later he proposed B-PROP (Ma et al. 2021b) to free PROP from limitation of classical uni-gram language model.

Another way to enhance pre-trained models is to design fine-tuning strategies. ULMFiT (Howard and Ruder 2018) is the pioneering work which uses several strategies to fine-tune the pre-trained model on the text classifiers. Further research tries to use the fine-grained sample strategies to design tasks, such as (Huang et al. 2020) utilized random negative sampling for recall tasks while ANCE (Xiong et al. 2020) used the top of recalled hard negative samples for training. It is surprising that hard negative samples seem experimentally conductive for downstream tasks. (Zhan et al. 2021) theoretically explains the effectiveness of hard negative samples and also introduces two strategies for both dense retrieval and reranking.

In strategy V1, the trained front compressor will select the top negative samples with higher relevance scores (higher difficulty), and provide them to the rear compressor for training. The process has been shown in Figure 3a.

Supposing that there are $M$ compressors $\mathcal{C}$ (including classifier). The compressor aims to correctly classify positive sample as well as select hard negative samples. As mentioned before, compressors are implemented by some pre-trained models. All compressors have identical network structure in our settings.

We denote parameter of all compressors as $\theta$. For $i$-th compressor $C_{i}$, we denote its parameter as $\theta_{i}$, network as $f_{i}$ and input data as $S_{i}$. For compressor $C_{1}$, its input data $S_{1}$ is a set of queries $Q$ and their corresponding samples. Each query would correspond to one positive sample and $N$ negative samples. We use $|S_{i}|$ to denote number of negative samples pairing a positive sample for $i$-th compressor, therefore $|S_{1}|=N$. For query $q$, its training sample $S_{i}(q)$, would be $(q,d)$. $d=(d^{0},d^{1},d^{2},...,d^{|S_{i}|})$. Here $d^{0}\in D^{+}_{q}$ and $d^{1},...,d^{N}\in D^{-}_{q}$. Note that the positive sample would be fixed at the first index position following with $|S_{i}|$ negative samples. $i$-th compressor outputs relevance scores $o_{i}\in R^{1+|S_{i}|}$ for input samples. $o_{i}=f_{i}(S_{i})$. For $i$-th compressor, we can formulate our objective function on query $q$ as cross entropy loss:

where $P$ denotes the probability function.

We can calculate $\theta_{i}$ as:

After training the first compressor, the model will evaluate the difficulty of input negative samples and then pass hard ones to the next compressor. We refer this infer-after-train procedure as self-involvement in this paper. Theoretically speaking, this procedure would give the pre-trained model a constrained semantic sample space for learning. Thus, the model would know what they need to learn. At inference stage, we assume relevance scores $o$ can reflect the difficulty (or quality) of negative samples. We select $K_{i}$ hardest negative samples from negative samples (normally, $K_{i}\ll N$) at $i$-th compressor and feed them to the next compressors:

Note that the value of K is a hyper parameter and will gradually decrease. The positive sample will automatically send to the next compressor. The last compressor only need to classify the samples, we regard it as the classifier in SIR. In strategy V1, every compressor is initialized at the same checkpoint of certain pre-trained model.

For faster convergence, in strategy V2, we keep the same network architecture as strategy V1 but let the model parameters to flow from one compressor to the next. After having trained the front compressor, we share the front compressor’s parameter with the rear one instead of training it from ground up. The rear compressor would start at the checkpoint that the front compressor has achieved. This is a soft parameter sharing mechanism (Ruder 2017).

The reason we use the soft parameter sharing mechanism is that after training the previous compressor $C_{i-1}$, it would capture the complete distribution information of the previous training samples. Sharing parameters can be interpreted as adding prior knowledge to the next compressor $C_{i}$. For compressor $C_{i}$, the semantic space of negatives would be refined based on both previous memorized knowledge and current training samples. So it would have a more comprehensive sample distribution compared with strategy V1.

Directly employ the prior knowledge in model initialization would improve model performance to some extent, but the distribution of negative samples used for training each compressor is still fixed. The fix distribution of negative samples leads to the dynamics loss, and that can not be compensated by parameter memory. Therefore, V3 uses the supervisory signal to synchronously adjust the compressors at all levels, so that the negative sample distribution selected by each compressor is also dynamic. V3 adopts hard parameter sharing mechanism (Ruder 2017).

In the forward process of the training stage, we keep the same inference procedure as the previous two strategies. Differently, in the backward process, when the model parameters are updated with the supervisory signal, the negative sample distribution selected by the front compressor for the rear compressor will also be adjusted. With the continuous convergence of the model, the difficulty of negative samples increases gradually, which is consistent with the principle of curriculum learning.

Conclusively, We iteratively select dynamic hard samples and gradually force the train set updating step by step. For query $q$, the loss of the whole model consists of classification loss of the first compressor and the last compressor:

Since the compressors in strategy V3 share their parameters, we can formulate the learning process as:

In strategy V3, the negative sample distribution generated by compressor at each level is adjusted with supervisory signal. However, the signal of the following stage compressor can not directly feed back to supervise the previous stage compressor, but can only indirectly rely on the parameter sharing mechanism to affect previous compressor.

In order to solve the problem of the front compressor’s weak perception of the ranking results of rear compressor, we use the feedback mechanism of the control theory (Franklin et al. 2002) for reference and use the conditional probability (CPR) to link the ranking results of the front and rear compressors. Then, the supervisory signal can be directly back-propagated from the rear compressor to its front compressors.

In the modeling of V4, for query $q$, $i$-th compressor outputs is samples’ relevance score $o_{i}\in R^{1+|S_{i}|}$. We softmax $o_{i}$ as in Eq. (2) to obtain $P(o_{i})$. We denote $P(o_{i})$ as $P_{i}$ for simplicity.

Here, we constrain the relative position of same top-ranked samples in different level of compressors, which means that for $j$-th sample at $i$-th compressor, its predicted score at $i$-th compressor is $o_{i}^{j}$ and its score at front compressor is $o_{i-1}^{j}$. Note that the positive sample’s score is $o_{i}^{0}$.

We denote the samples’ conditional probability at $i$-th compressor as $\text{CPR}_{i}\in R^{1+|S_{i}|}$. $\text{CPR}_{i}$ is computed by the multiplication of posterior probability of sample to fulfill the goal that the supervisory signal of each compressor can be back propagated to front compressor directly. Such settings can control sample dynamics at front compressor with supervisory signal at rear compressor, thus achieve better semantic space of negative samples at all compressors. The conditional probability of $i$-th compressor can be written as:

where $\odot$ denotes element wise multiplication and $P_{i}^{0:|S_{i}|}$ denotes the beginning $1+|S_{i}|$ logits of $P_{i}$.

Then, we formulate the loss function based on $i$-th compressor for query $q$ as:

The loss function of V4 is:

Pseudo code for strategy V3 and V4 can be found in Algorithm 1.

MS MARCO Document Ranking is now the most popular dataset in IR (Bajaj et al. 2016). There are 3.2 million documents and 367,013 queries in the training set and the goal is to rank based on their relevance.

We conduct experiments on 2 pre-trained language models and 3 pre-trained IR models. For pre-trained language models, BERT (Devlin et al. 2019) and ELECTRA (Clark et al. 2020) are used. BERT designs Masked Language Models (MLM) and Next Sentence Prediction (NSP) as its pre-training tasks. Instead, ELECTRA uses replaced token detection task. For pre-trained IR models, WLP uses Wiki Link Prediction task, ICT uses Inverse Cloze task (Lee, Chang, and Toutanova 2019; Chang et al. 2020) and PROP uses Representative Words Prediction task (Ma et al. 2021a).

For each positive sample, we first find its corresponding negative samples with HDCT and ANCE.

Considering the computational cost, we used three-level compressor (including classifier) for the sample selection. Increasing number of level could require a large amount of data for training. In our implementation, we found that three-level architecture would achieve SOTA compared with other multiple level architectures. To show that SIR can be implemented on various pre-trained models, we use pre-trained BERT-base and ELECTRA-base with limited ranking ability.

For training setup, considering that distributed data parallel is used for experiments acceleration, we set samples size to be the integer times of the number of GPU devices. Since we train our model with V100 which has 8 GPUs, we set the size of a training block (the sample number of positives and negatives in a block) as 88 and batch size of the training block as 4. We set our self-involvement samples compressed from 88 to 48 and 48 to 16 in 3 level models. And we further compress 16 to 8 in 4 level models. Also, we set the training epochs as 2. In this work, we used AdamW as our Optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$, and learning rate as $5e^{-5}$. We adopt both the ANCE based top 100 dataset and HDCT based top 100 dataset as the test bed. Since we are trying to leverage pre-trained models with simple SIR strategies, here we only train our model on the side of fine-tuning downstream ranking tasks based on top 100 dataset.

We evaluate SIR on ranking tasks and use mean reciprocal rank (MRR), mean average precision (MAP) and normalized discounted cumulative Gain (NDCG) as metrics.

In Table 1, we show that SIR can substantially enhance pre-trained IR models. WLP (Chang et al. 2020), ICT (Chang et al. 2020) and PROP (Ma et al. 2021a) are three pre-trained IR models with different pre-training tasks. All of them gain great performance improvement on both HDCT top 100 based data and ANCE top 100 based data. Only V4 results are reported in Table 1.

In Table 2, we compare different SIR strategies based on two widely used pre-trained language models, including BERT-base and ELECTRA-base. Results show that all SIR strategies increase models’ performance. Among 4 strategies, V4 is the best and V3 is close to V4. It empirically shows that dynamic and hard negative samples would be beneficial for fine-tuning of the pre-trained models. The results are also shown in Figure 4.

To conclude, experiments show that SIR can reach SOTA ranking performance even under a native BERT base and it can also greatly improve arbitrary pre-trained model’s performance on MS MARCO Document ranking. We used SIR V4 strategy to boost ELECTRA-large, which achieved the best scores in MSMARCO document ranking leaderboard in May 2021. In order to prevent exposure of information, the leaderboard scores are not provided in tables for the time being.

Figure 6 shows semantic space of negatives in classifier. Markers in the figure have the same meaning as Figure 1. Note that Figure 1 shows semantic space of the whole model, while Figure 6 shows semantic space in classifier. V1 selects fix hard negative samples as in Figure 6(a). V2 selects fix hard negative samples but with distribution information of wider space as in Figure 6(b), where black dash line represents distribution information received from front compressor. V3 and V4 selects dynamic hard negative samples as in Figure 6(c). The difficulty of negative samples changes with the training process. The comparison of the four strategies (V1-V4 of SIR) are shown in Table 2.