Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding

Given a training dataset $\mathcal{D}$ for a specified task and a neural network $f$ parameterized by $\bm{\theta}\in\mathbb{R}^{|\bm{\theta}|}$, the training objective for each sample $(x,y)\in\mathcal{D}$ is to minimize empirical risk

where $x$ is the input context, $y$ in output space $\mathcal{Y}$ is the label, and $L:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}_{\geq 0}$ is the task-specific loss function, $\mathbb{R}_{\geq 0}$ denoting the set of non-negative real numbers. In NLU tasks, $x$ is typically a sequence of words, while $y$ can be a single category label for classification (Pang et al., 2016; Wang et al., 2019; Xu et al., 2022), or a pair of start and end boundaries for extraction (Rajpurkar et al., 2016; Yang et al., 2018; Pang et al., 2019), or a sequence of relevance levels for ranking (Niu et al., 2012; Nguyen et al., 2016; Zhu et al., 2020; Pang et al., 2020).

After training a neural model $f(x;\bm{\theta})$, to reduce memory and computation requirements at test time, network pruning (Blalock et al., 2020) entails producing a smaller model $f(x;\bm{m}\odot\bm{\theta}^{\prime})$ with similar accuracy through post-hoc processing. Here $\bm{m}\in\{0,1\}^{|\theta|}$ is a binary mask that fixes certain pruned parameters to 0 through elementwise product $\odot$, and the parameter vector $\bm{\theta}^{\prime}$ may be different from $\bm{\theta}$ because $\bm{m}\odot\bm{\theta}^{\prime}$ is usually retrained from $\bm{m}\odot\bm{\theta}$ to fit the pruned network structure.

Although pruning is often viewed as a way to compress models, it has also been motivated by the desire to prevent overfitting. Pruning systematically removes redundant parameters and neurons that do not significantly contribute to performance and thus have much less prediction variance, which makes us reminiscent of dropout (Labach et al., 2019), another widely used technique to avoid overfitting. Similarly, dropout also uses a mask to disable a fraction (such as $p\%$) of parameters or neurons. The significant difference, though, is that the mask $\bm{m}$ in dropout is randomly sampled from a $\mathrm{Bernoulli}(1-p\%)$ distribution, rather than deterministically defined by a criterion (e.g., the bottom $p\%$ of parameters in magnitude should be masked) as in pruning. This in turn brings convenience: a model trained with dropout does not need to be retrained for a specific mask, because the model’s neurons have already started to learn how to adapt to the absence of some neurons in the previous training.

To eliminate the noisy content in the input context $x$ and further improve the model performance, context selection selectively crops out a condensed context $x^{\prime}\sqsubseteq x$ to produce the final model prediction. In general, the model requires specialized training to fit the selected context. Therefore, context selection is pre-hoc processing relative to training, requiring removing the noise from the training samples in advance. With a slight abuse of notation, here we use $x^{\prime}\sqsubseteq x$ to denote that $x^{\prime}$ is a condensed subsequence (possibly equal) of $x$. In general, $x^{\prime}$ is an ordered combination of segments of $x$, where the segments are usually at the sentence (Min et al., 2018; Pang et al., 2021), chunk (Zheng et al., 2020), paragraph (Clark and Gardner, 2018), or document (Tu et al., 2020; Dua et al., 2021) granularity. It is worth noting that the selector for segment selection generally requires additional supervised training and needs to be run in advance of the prediction, which introduces additional computation overhead.

To assess the contribution of certain components to the overall model, ablation studies investigate model behavior by removing or replacing these components in a controlled setting (Cohen and Howe, 1988). Here, in the context of machine learning, “ablation” refers to the removal of components of the model, which is an analogy to ablative brain surgery (removal of components of an organism) in biology (Meyes et al., 2019). We refer to the model after component removal as the “ablated model”, which should continue to work. However, if the removed components are responsible for performance improvement, the performance of the ablated model is expected to be worse (Frank et al., 2021).

In this paper, we use “ablation” to refer specifically to the removal of some neurons of a neural model, i.e., to set the output of some specific neurons to zero. From such a neuronal perspective, network pruning and context selection can be viewed as two kinds of ablation, the former removing some low-contributing hidden neurons after training and the latter removing some low-information input neurons before training. However, in contrast to ablation studies that aim to investigate the role of the ablated neurons, we aim to learn to improve the utility of the ablated neurons.

For an efficient model, we believe that all its neurons should be able to work together efficiently to maximize the utility of each neuron. This means that each neuron should contribute to the overall model, or at least be harmless, because the cooperation of neurons is supposed to eliminate the negative effects of noise that may be produced by individual neurons. Thus, if we ablate some neurons, even those that produce noise, due to the missing contribution of the ablated neurons, then the ablated submodel should perform no better than the original full model, in other words, its task-specific loss should be no smaller than the original.

Formally, we define a neural model as an efficient model if and only if it performs no weaker than any of its ablation models, and we formalize the comparison principle between an efficient model and its ablation models as follows.

In the above definition, we consider that an efficient neural model should be input-efficient and parameter-efficient. In particular, the input-efficient property refers that the model can efficiently utilize the input neurons (words). If the model $f(x;\bm{\theta})$ satisfies Eq. (2), we say that $f(\cdot;\bm{\theta})$ is input-efficient for the input $x$. The parameter-efficient property refers that the model can utilize the hidden neurons efficiently. If the model $f(x;\bm{\theta})$ satisfies Eq. (3), we say $f(x;\bm{\theta})$ is parameter-efficient for the input $x$ with respect to the parameter space $\mathbb{R}^{|\bm{\theta}|}$. According to Eq. (3), we can definitely find at least one vector $\bm{\theta}$ that is parameter-efficient for the input $x$, i.e., the zero vector and the optimal parameter vector that minimizes the empirical risk. If the parameter space is large enough, from those vectors parameter-efficient for $x$, we can find some parameters that simultaneously satisfy Eq. (2), i.e, the input-efficient property. That is, if the parameter space $\mathbb{R}^{|\bm{\theta}|}$ is large enough, there exists at least one parameter vector $\bm{\theta}$ that makes the model $f(x;\bm{\theta})$ efficient for $x$. Specially, if all activation functions in the neural model $f$ have zero output values for zero, then $f(x^{\prime};\bm{0})=f(x;\bm{0})\ \forall x^{\prime}\sqsubset x$, and hence the parameter vector $\bm{\theta}=\bm{0}$ is efficient for any $x$.

Notably, we restrict the ablated input $x^{\prime}$ for comparison to only those subsequences whose ground-truth output $g(x^{\prime})$ remains unchanged, i.e., $g(x^{\prime})=g(x)=y$. This is because ablation may remove some key information from the original input $x$, such as the trigger words in the classification, resulting in an unknown change in the label $y$. In this unusual case, $L(y,f(x^{\prime};\bm{\theta}))$ will no longer be a reasonable proxy of the performance of the input-ablated model, so it makes no sense to compare it to the task-specific loss of the original model. For example, in binary classification ($y\in\{0,1\}$), for the original input $x$, $f(x;\bm{\theta})$ predicts the correct category $y$ with low confidence, whereas for the ablated input $x^{\prime}$ whose category label changes to $g(x^{\prime})=1-y$, $f(x^{\prime};\bm{\theta})$ predicts $1-y$ with high confidence. Even though $L(y,f(x;\bm{\theta}))\leq L(y,f(x^{\prime};\bm{\theta}))$, the input-ablated model $f(x^{\prime};\bm{\theta})$ actually outperforms the original $f(x;\bm{\theta})$, i.e., $L(y,f(x;\bm{\theta}))\geq L(1-y,f(x^{\prime};\bm{\theta}))$, and we cannot consider $f(\cdot;\bm{\theta})$ to be input-efficient for $x$. Although we can use $L(g(x^{\prime}),f(x^{\prime};\bm{\theta}))$ as a performance proxy for the input-ablated model in Eq. (2), in practice it is difficult to know how the labels of the ablated inputs will change, so we try to avoid such label-changing scenarios. For the sake of concision, from here on, we default the ablation of the input context does not change the output label if not otherwise specified, i.e., $g(x^{\prime})=y$.

Further, we can ablate a full model $f(x^{(0)};\bm{\theta}^{(0)})$ multiple ($c$) times, but are these ablated models $\{f(x^{(i)};\bm{\theta}^{(i)})\}_{i=1}^{c}$ comparable to each other? The comparison principle only points out the comparative relation between an efficient model and any of its ablated models, and cannot be directly applied to multiple ablated models. However, if we assume that these ablated models are constructed step by step, i.e., each ablated model $f(x^{(j)};\bm{\theta}^{(j)})$ is obtained by progressively ablating the input ($x^{(j)}\sqsubset x^{(j-1)}$) x or parameters ($\bm{\theta}^{(j)}=\bm{m}^{(j)}\odot\bm{\theta}^{(j-1)}$) based on its previous model $f(x^{(j-1)};\bm{\theta}^{(j-1)})$, then $f(x^{(j)};\bm{\theta}^{(j)})$ can be considered as an ablated model of all its ancestor models $\{f(x^{(i)};\bm{\theta}^{(i)})\}_{i=0}^{j-1}$. For simplicity, we abbreviate their task-specific losses as $l^{(i)}=L(y,f(x^{(i)};\bm{\theta}^{(i)}))$. Furthermore, if all $\{f(x^{(i)};\bm{\theta}^{(i)})\}_{i=0}^{c-1}$ are simultaneously assumed to be efficient with respect to their parameter spaces $\mathbb{R}^{\|\bm{m}^{(i)}\|_{0}}$ ³³3The number of non-zeros (L0 norm) in the mask determines the number of available parameters., we can apply the comparison principle to compare the task-specific losses of any two models, i.e., $l^{(i)}\leq l^{(j)},\forall i<j$.

Formally, we define an efficient model to be hereditarily efficient if and only if its ablated models are all efficient. Similarly, if the parameter-ablated models of a parameter-efficient model are all parameter-efficient, we call this parameter-efficient model hereditarily parameter-efficient. And the hereditarily input-efficient model is defined in the same way. Specifically, the parameter vector $\bm{\theta}=\bm{0}$ is hereditarily parameter-efficient, and $f(x;\bm{0})$ is also hereditarily efficient for any $x$ if all activation functions of $f$ have zero output values for zero.

Based on the definition of the hereditarily efficient model and the comparison principle, we can draw the following corollary.

In brief, the corollary describes a desirable transitive comparison between a hereditarily efficient neural model and its ablated models, i.e., the less ablation, the better the performance. Unfortunately, this natural property has been largely ignored before, which motivates us to exploit it to train models that utilize neurons more efficiently.

Based on Corollary 3.1, we can train a hereditarily efficient model with the objective of ordered comparative relation in Eq. (4). To measure the difference from the desirable order, we can use pairwise hinge loss (Herbrich et al., 2000) to evaluate the ranking of the task-specific losses of the full model and its ablated models, like $\sum_{i=0}^{c-1}\sum_{j=i+1}^{c}\max(0,l^{(i)}-l^{(j)})$. However, optimizing this ranking loss alone cannot guarantee that these task-specific losses are minimized, i.e., the full/ablated models may not be empirical risk minimized (ERM) (Vapnik, 1991) with respect to their parameter spaces. To push these models to be ERM, we introduce a special scalar $b$ as the baseline value of the task-specific loss and assume that it is derived from a dummy ablated model $f(x^{(c+1)};\bm{\theta}^{(c+1)})$. The dummy model is set to have the highest degree of ablation, and in principle, its task-specific loss $l^{(c+1)}$ should be the highest. However, to push the task-specific losses of the real models $\{f(x^{(i)},\bm{\theta}^{(i)})\}_{i=0}^{c}$ down, we usually set $l^{(c+1)}=b$ to a small value (e.g., 0) and expect all $\{l^{(i)}\}_{i=0}^{c}$ to be reduced by this target. In this way, our comparative losses can still be written as a pairwise ranking loss, except that on top of the $c+2$ task-specific losses,

Fig. 2 visualizes the localization (central grid region) of the ideal model of comparative loss, which is both ERM and hereditarily efficient. The hereditarily efficient is a subset of the efficient, and the efficient is the intersection of the input-efficient and parameter-efficient. In this light, comparative loss sets a stricter training objective than ERM. When we set $c$ and $b$ to 0, Eq. (5) can degenerate to Eq. (1). Further, the comparative loss is equivalent to

where the first term is to minimize the empirical risk of those not reaching the target $b$, and the second term constrains the comparative relation to pursue the full model being hereditarily efficient.

To train using comparative loss, we first need to obtain several comparable ablated models and task-specific losses. As shown in Fig. 3, we consider the original model with the input of the entire context as the full model $f(x^{(0)};\bm{\theta}^{(0)})$. According to Corollary 3.1, we progressively perform $c$-step ablation based on the full model. At the $i$-th ($1\leq i\leq c$) ablation step, we use CmpCrop or CmpDrop to ablate a small portion of the input or hidden neurons based on the model $f(x^{(i-1)};\bm{\theta}^{(i-1)})$ from the previous step, which makes the newly ablated model $f(x^{(i)};\bm{\theta}^{(i)})$ comparable to all its ancestor models. After all these models have predicted once, we have $c+1$ comparable task-specific losses. Together with $l^{(c+1)}=b$ from the dummy ablated model, we can calculate the final loss using Eq. (5). Using stochastic gradient descent optimization as an example, Algorithm 1 illustrates the training process more formally.

CmpDrop and CmpCrop in Algorithm 1 are the two alternative ablation methods we present for each ablation step, the former for ablating the parameters (hidden neurons) and the latter for ablating the input context (input neurons). They both randomly ablate neurons in a controlled manner on top of the previous model, which allows the coverage of all potential ablated models without retraining each ablated model. This is because the randomly ablated models are jointly trained and adapt to the absence of some neurons during the training process. As for which one to use at each ablation step can be specific to the model and task dataset. Ideally, CmpDrop can be used as long as the model is dropout compatible, and CmpCrop can be used as long as the input context of the task contains dispensable segments. Below we will introduce CmpDrop and CmpCrop in detail.

Further deriving Eq. (5), we find that comparative loss can be viewed as a dynamic weighting of multiple task-specific losses. In particular, the loss can be rewritten as follows,

where $\mathbbm{1}_{C}$ is an indicator function equal to 1 if condition $C$ is true and 0 otherwise, and the $\mathrm{CMP}$ function determines whether model $f(x^{(i)};\bm{\theta}^{(i)})$ complies with the comparison principle compared to $f(x^{(j)};\bm{\theta}^{(j)})$ and adjusts the weight of $l^{(i)}$. There are two cases of non-compliance: for the case where $f(x^{(i)};\bm{\theta}^{(i)})$ is less ablated ($i<j$) but more loss is obtained, we increase the weight of $l^{(i)}$; for the case where $f(x^{(i)};\bm{\theta}^{(i)})$ is more ablated ($i>j$) but less loss is obtained, we decrease the weight of $l^{(i)}$. Formally, the $\mathrm{CMP}$ function can be written as

Here we can notice that for a pair of models that do not conform to the comparison principle, we increase (+1) the weight of the task-specific loss of the model that is ablated less and equally decrease (-1) the weight of the loss of the model that is ablated more. Thus, let $\alpha^{(i)}=\sum_{j\neq i}\mathrm{CMP}(i,j,l^{(i)},l^{(j)})$ denote the weight of $l^{(i)}$, then the sum of the weights of all task-specific losses (including the dummy one) is 0, i.e., $\sum_{i=0}^{c}\alpha^{(i)}=-\alpha^{(c+1)}=\sum_{i=0}^{c}\mathbbm{1}_{l^{(i)}>b}$. Since $l^{(c+1)}=b$ is a constant, Eq. (6) is also equivalent to $\sum_{i=0}^{c}\alpha^{(i)}l^{(i)}$, i.e., the total weight equals the number of task-specific losses worse than the virtual baseline $b$, and is adaptively assigned to the $c+1$ losses according to their performance. In this way, poorly performing full/ablated models will be more heavily optimized. And we empirically compare other heuristic weighting strategies in $\S$5.1.

For parameter ablation, in addition to being able to weight each task-specific loss differentially, the comparative loss with CmpDrop can also differentially calculate the gradients of the parameters in different parts. According to Eq. (6), comparative loss is equal to the sum of all differences of task-specific loss pairs that violate the comparison principle, i.e, $\sum_{i<j\,\land\,l^{(i)}>l^{(j)}}\big{(}l^{(i)}-l^{(j)}\big{)}$, so we can analyze the gradient of comparative loss from the difference of each task-specific loss pair. For ease of illustration, we take the original model $f(x;\bm{\theta})$ and a model $f(x^{\prime};\bm{\theta}^{\prime})$ whose parameters have been ablated $n$ times as an example, and other model pairs with different parameters are similar. Assume that the original parameters $\bm{\theta}=(\bm{u},\bm{v},\bm{w})$ and the ablated parameters $\bm{\theta}^{\prime}=(\bm{u}^{\prime},\bm{v}^{\prime},\bm{w}^{\prime})=(\bm{u},\bm{v}/(1-p)^{n},\bm{0})$, where $\bm{u}$ is the parameters from the layers without dropout, $\bm{w}$ is the parameters ablated by $n$ times CmpDrop, and $\bm{v}^{\prime}$ is the scaled parameters surviving from $n$ times dropout. Then, if their task-specific losses violate the comparison principle, i.e., $l>l^{\prime}$, the gradient of the comparative loss contributed by this model pair is

We can see that the comparative loss with respect to $\bm{w}$ is higher than the comparative loss with respect to the other parameters. This is intuitive because the model instead performs better after ablating away $\bm{w}$ indicating that the current $\bm{w}$ is inefficient, so we need to focus on updating $\bm{w}$.

In addition to the dynamic weighting perspective, comparative loss can also be considered as an “inverse ablation study” during training. This is because, in contrast to ablation studies that determine the contribution of removed components during validation, comparative loss believes that the ablated neurons should contribute and optimizes parameters with this objective.

For training complexity, given a generally small number of comparisons $c$ (i.e., number of ablation steps), the overhead of computing the final comparative loss is negligibly small, and the increased computation overhead per update step comes mainly from the multiple forward and backward propagations of the models. Specifically, the overhead of a training step using comparison loss is $1+c$ times that of conventional training for the same batch size. For inference complexity, however, models trained using comparative loss are the same as conventionally trained models at test time.

Text classification is a fundamental task in natural language understanding, which aims to assign a predefined category to a piece or a group of text. In many text classification datasets, all segments of the input context seem to play an important role in the text category and there is almost no annotation of the minimal support context, so it is difficult for us to construct an input-ablated model by directly cropping the original input without changing the classification label. That is, it is likely to violate the constraint that the label of the ablated input is unchanged in the comparison principle, and thus we cannot apply CmpCrop to this task. However, many current neural classification models use dropout during training, so in this task, we only validate the comparative loss that uses just CmpDrop.

Extractive reading comprehension (RC) (Rajpurkar et al., 2016; Liu et al., 2019b) is an essential technical branch of question answering (QA) (Hirschman and Gaizauskas, 2001; Lin, 2002; Chen, 2018; Rodriguez and Boyd-Graber, 2021; Zhu et al., 2021). Given a question and a context, extractive RC aims to extract a span from the context as the predicted answer. Current dominant RC models basically use pretrained Transformer (Vaswani et al., 2017) architectures, which employ dropout in many layers during finetuning. This allows us to use CmpDrop to improve the utility of the model parameters. Additionally, the given context is usually lengthy and contains many distracting noise segments, which also allows us to use CmpCrop to improve the model’s utilization of the context by randomly deleting the labeled distracting paragraphs. Therefore, we intend to verify the effectiveness of comparative loss using CmpDrop or/and CmpCrop in this task.

Pseudo-relevance feedback (PRF) (Attar and Fraenkel, 1977) is an effective query understanding (Chang and Deng, 2020) technique to improve ranking accuracy, which aims to alleviate the mismatch of linguistic expressions between a query and its potential relevant documents. Given an original query $q$ and a document collection $C$, a base ranking model returns a ranked list $D=(d_{1},d_{2},\cdots,d_{|D|})$. Let $D_{\leq k}$ denote the feedback set containing the top $k$ documents, where $k$ is usual referred to as the PRF depth. The goal of PRF is to reformulate the original query $q$ into a new representation $q^{(k)}$ using the query-relevant information in $D_{\leq k}$, i.e., $q^{(k)}=f((q,D_{\leq k});\bm{\theta})$, where $q^{(k)}$ is expected to yield better ranking results. Although PRF methods do usually improve ranking performance on average (Clinchant and Gaussier, 2013), individual reformulated queries inevitably suffer from query drift (Mitra et al., 1998; Zighelnic and Kurland, 2008) due to the objectively present noise in the feedback set, causing them to be inferior to the original ones. Therefore, we can use comparative loss with CmpCrop to train PRF models to suppress the extra increased noise by comparing the effect of queries reformulated using feedback sets with different PRF depths.

To verify the role of comparative loss from the dynamic weighting perspective, we keep all the training settings of Longformer + CmpCrop + CmpDrop from the last row of Table 2 unchanged and replace only the weighting strategy of task-specific losses with some heuristics. Table 4 shows their performance on the HotpotQA development set. AVERAGE, FIRST and LAST are three static weighting strategies. AVERAGE assigns equal weights to all task-specific losses, while FIRST and LAST assign weight to only the first and last task-specific loss, respectively, i.e., FIRST optimizes $l^{(0)}$ of the full model without dropout and LAST optimizes $l^{(2)}$ of the model with regular dropout rate $p$ (equivalent to the baseline Longformer in Table 2). MAX is another dynamic weighting strategy that assigns weight to only the largest task-specific loss. We can see that dynamic weighting in comparative losses is significantly better than these heuristic weighting strategies, which proves that comparative loss can assign weights more appropriately. In addition, AVERAGE is better than the latter three strategies that consider only one task-specific loss, indicating that it is beneficial to consider multiple task-specific losses. Moreover, although the latter three are all assigned to only one task-specific loss, MAX is better than the other two, which indicates that dynamic assignment is better than static assignment.

Notably, the FIRST that directly optimizes the full model outperforms the LAST that is trained with dropout, suggesting that the inconsistency of dropout between the training and inference stages (Zolna et al., 2017) may indeed lead to underfitting of the full model. And the fact that Cmp far outperforms FIRST and LAST indicates that comparative loss can automatically strike a balance between ensuring training-inference consistency and preventing overfitting.

To study the impact of comparison strategies, i.e., how many ablation steps we should use for comparison and which ablation method we should choose at each step, we try a variety of comparison strategies on HopotQA with different numbers of comparisons and ablation orders. As shown in Table 5, the results are not significantly further improved when we repeat CmpDrop/CmpCrop twice, but the results are further improved when we apply CmpCrop first and then CmpDrop. This indicates that comparing multiple models ablated by the same method, i.e., encouraging the model be either hereditarily input-efficient or hereditarily parameter-efficient, seems to have little effect on the performance of the full model, but the successive use of two different ablation methods, i.e., encouraging the model be efficient (both input-efficient and parameter-efficient), is helpful. However, applying CmpDrop followed by CmpCrop did not perform as well as applying CmpDrop only, suggesting that the order of the ablation methods is important and perhaps the ablation should be done in the order of the information flow in the model.

To further confirm the influence of the number of ablation steps $c$, we show in Fig. 4 the relationship between the model’s Average metric over the eight GLUE datasets and the number of ablations. We can find little difference in the average performance of the models trained with different numbers of CmpDrop, with the model trained with one CmpDrop performing significantly best mainly because its huge advantage on two of the datasets pulls up the average. Therefore, if there is no extreme demand for performance, we usually do not need to tune the hyperparameter $c$.

To investigate the impact of model parameters, we explore the application of the comparative loss with CmpDrop on different-sized versions of BERT, RoBERTa and ELECTRA. From Table 6 we can see that the comparative loss with CmpDrop achieves a consistent improvement over the baselines based on these backbone models, which indicates that the comparative loss can improve model performance by increasing parameter utility without increasing the number of parameters. Moreover, except for the one outlier of BERT_Medium, we can roughly find that the less the model parameters, the greater the relative gain from comparative loss. This is reasonable because the individual hidden neurons in a model with lower capacity play a larger role, so the improvement in the utility of hidden neurons can be more reflected in the final performance. Whereas for a model of higher capacity, it is easier to fit less training data, i.e., its task-specific loss is already low, so comparative loss has less room to play in reducing task-specific loss further. In addition, we observe that the boost to BERT from the comparative loss with CmpDrop is generally higher compared to RoBERTa and ELECTRA with more complicated pretraining, suggesting that the comparative loss helps the model escape from local optima due to parameter initialization.

To review the utility of the input context (i.e., input neurons) to models, we plot in Fig. 5 the performance trends of the models using different context sizes. First, in both datasets, our models trained with comparative loss consistently outperform the baseline models for all context sizes, indicating that our models are able to utilize input neurons more efficiently with equal amounts of input context. Second, this shows that our comparative loss can further improve the model performance after streamlining the input with context selection. In addition, we notice that our ANCE-PRF + CmpCrop in Fig. 5 improves retrieval performance as expected as the number of feedback documents increases, while ANCE-PRF reaches peak performance at 4 feedback documents and then suffers performance degradation, implying that our model is more robust and able to mine and exploit relevant information in the added feedback documents. In contrast to PRF, for HotpotQA in Fig. 5, the performance of all RC models decreases as the number of paragraphs increases. This is understandable, since only 2 paragraphs in HotpotQA are supporting facts, and the remaining 8 mostly serve as a distraction, so the ideal performance curve can just be a horizontal line that does not drop when the paragraph number increases. Interestingly, we find that the degradation of Longformer + CmpDrop (2.7%) and Longformer + CmpCrop + CmpDrop (3.0%) from the oracle setting (2 gold paragraphs) to the distractor setting (10 paragraphs) is lower than that of the baseline Longformer (3.4%). This suggests that comparative loss can help the models suppress the noisy information in the added context. Although Longformer + CmpCrop (3.7%) has a larger degradation than Longformer, we believe this is because Longformer + CmpCrop needs to be optimized for various numbers of paragraphs, unlike other models without CmpCrop that focus on learning for one input form (i.e., always ten paragraphs). However, this variety of input forms makes Longformer + CmpCrop perform better than Longformer + CmpDrop when the number of paragraphs is small ($\leq 5$).

To further quantitatively demonstrate the help of comparative loss in the robustness of the PRF model to context size, we report in Table 7 the robustness indexes (Collins-Thompson, 2009) of ANCE-PRF + CmpCrop and ANCE-PRF at different numbers of feedback documents. The robustness index is defined as $\frac{N_{+}-N_{-}}{|Q|}$, where $|Q|$ is the total number of evaluated queries and $N_{+}$ and $N_{-}$ are the number of queries that the PRF model improves or downgrades when one more feedback document is used. The value of robustness index is in [-1, 1], and the model with higher robustness index is more robust. We can see that the PRF model trained using comparative loss with CmpCrop is significantly more robust than the baseline model. Besides, from the gaps in their robustness indexes (only 0.03 or 0.02 for 1 or 2 documents, but 0.05 for more documents), we can find that the comparative loss is more helpful for long-form inputs.

To figure out the impact of comparative loss on task-specific loss, we plot the curves of task-specific loss for the full model (i.e., $l^{(0)}$) in Fig. 6. From Fig. 6(a) and Fig. 6(b) we can see that with the same batch size, the comparative loss can help our models fit better compared to the baseline Longformer. Comparing Longformer + CmpDrop and Longformer + CmpCrop, we can find that the training loss of the former is significantly smaller, which indicates that the comparative loss with CmpDrop helps the model fit the training data better. Whereas the evaluation loss of Longformer + CmpCrop rises less in the later stage, which indicates that the comparative loss with CmpCrop can mitigate the overfitting to some extent. Since the number of task-specific losses per sample optimized by comparative loss is $1+c$ times that of conventional training, we also plot the task-specific loss curves for PRF models in Fig. 6(c) and Fig. 6(d), where the batch size of our uniCOIL-PRF + CmpCrop is $1/(1+c)$ of the baseline uniCOIL-PRF. In this way, the number of task-specific losses optimized in one batch for our model and the baseline is the same, which helps to further clarify the role of the comparative loss with CmpCrop. We can see that while the training loss of our model in Fig. 6(c) does not drop as low as the baseline, its evaluation loss in Fig. 6(d) drops to a lower level and significantly mitigates the overfitting.

We present in Table 8 the performance gain and relative change in training FLOPs of BERT_base + Cmp compared to BERT_base, as well as the specific number of comparisons (i.e., number of ablation steps $c$) chosen for each dataset. We find that the actual overhead of training with comparative loss is usually less than $1+c$ times that of conventional training, and even less than that of conventional training (e.g., on QQP). This is because models trained with comparative loss tend to converge earlier than baselines. Combined with the insensitivity of comparative loss to the number of comparisons found from Fig. 4, we believe that setting $c$ to 1 or 2 can lead to effective and fast training when data is sufficient.

Contrastive learning has recently achieved significant success in representation learning in computer vision and natural language processing. At its core, contrastive learning aims to learn effective representations by pulling semantically similar neighbors together and pushing apart non-neighbors (Hadsell et al., 2006). Instead of learning a signal from individual data samples one at a time, it learns by comparing different samples (Le-Khac et al., 2020). The comparison is performed between positive pairs of similar samples and negative pairs of dissimilar samples. The positive pair must ensure that the two samples are similar, which can be constructed either by using supervised similarity annotation or by self-supervision. In self-supervised contrastive learning, a positive pair can consist of an original sample and its data augmentation. For example, SimCLR (Chen et al., 2020) in computer vision uses a crop, flip, distortion or rotation of an original image as its similar view, and SimCSE (Gao et al., 2021) in natural language processing applies two dropout masks to an input sentence to create two slightly different sentence embeddings that are then used as a positive pair of sentence embeddings. To share more computation and save cost, negative pairs usually consist of two dissimilar samples within the same training batch. Although both learn through comparison, contrastive learning aims at pursuing alignment and uniformity (Wang and Isola, 2020) of representations, while our comparative loss aims at pursuing orderliness of the task-specific losses of the full model and its ablated model. Moreover, as the lexical meaning suggests, contrastive learning only classifies the relationship (i.e., similar or dissimilar) between two data samples in a binary manner, whereas our comparative loss compares multiple full/ablated models by ranking. However, these two are not in conflict, and our comparative loss can be used over the contrastive losses that served as task-specific losses.

Dropout is a family of stochastic techniques used in neural network training or inference that have attracted extensive research interest and are widely used in practice. The standard dropout (Hinton et al., 2012) aims to avoid overfitting of the network by reducing the co-adaptation of neurons, where the outputs of individual neurons only provide useful information in combination with other neuron outputs. After this, a line of research focused on improving the standard dropout by employing other strategies for dropping neurons, such as dropconnect (Wan et al., 2013) and variational dropout (Kingma et al., 2015).

A line of research that is relevant to us is the use of dropout multiple times in training. SimCSE (Gao et al., 2021) forwards the model twice with different dropout masks of the same rate and uses a contrastive loss to constrain the distribution of model outputs in the representation space. A possible side effect of dropout revealed by the existing literature (Ma et al., 2016; Zolna et al., 2017) is the non-negligible inconsistency between the training and inference stages of the model, i.e., the submodels are optimized during training, but the full model without dropout is used during inference. To address this inconsistency, R-Drop (liang et al., 2021) forward runs the model multiple times with different dropout masks to obtain multiple predicted probability distributions and applies KL-divergence on them to constrain their consistency. Unlike their multiple dropout masks that are sampled independently, the multiple dropout rates are increasing and the masks are progressive in our CmpDrop, with the subsequent mask obtained by further randomly discarding elements based on the previous one. In addition, we impose constraints on the task-specific losses at the end rather than on the representations and probabilities upstream. Notably, the full model is also optimized in due time when trained using the comparative loss with CmpDrop, which we argue is important to mitigate the inconsistency between training and inference. This is because, while dropout avoids co-adaptation of neurons, it also weakens the cooperation between neurons ($\S$5.1 gives some empirical support). In particular, in cases where all neurons are involved, the full model trained with dropout has not been taught how to make them work together efficiently and thus cannot be fully exploited during testing. Surprisingly, our comparative loss with CmpDrop can balance between promoting the cooperation of neurons and preventing their co-adaptation.