Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks

A. Given a pair of input sentences $(X,Y)$, label $l_{(X,Y)}$, and a pre-trained BERT-based model $\mathcal{M}_{\textsc{Pre}}$, the goal is to output a reliable model $\mathcal{M_{\textsc{Rel}}}$ to predict an output label for a new input pair $(X_{test},Y_{test})$ such that the inconsistency between its different ordering is minimized. While we only experiment with semantic similarity (or paraphrasing), the description holds true for other symmetric relations too (such as predicting if two sentences have the same polarity).
B. Given a model fine-tuned on the task above $\mathcal{M_{\textsc{Rel}}}$, can it help in providing a better initialization for transfer learning an empirically superior model $\mathcal{M}^{\prime}$ on other downstream tasks?

For problem A (Section 3.1), the input is a concatenation of tokenized strings $X=x_{1},\ldots,x_{m}$ and $Y=y_{1},\ldots,y_{n}$ separated using a special token ([SEP] in the case of BERT). The concatenated inputs with the special token are passed through multiple self-attention layers Vaswani et al. (2017). In the traditional approach, the representation of the first token (<s> or [CLS]) is passed through a fully connected classifier layer (the same final representation is used irrespective of the arity of the task inputs). In our approach, we use the [CLSPara] representation for symmetric classification tasks whereas we use the standard first token (<s> or [CLS]) representation for single input and non-symmetric classification tasks (Section 4). Since we first fine-tune the model on [CLSPara] representation, our approach allows for pair-wise knowledge to be transferred to other downstream classification tasks (problem B (Section 3.1)).

We call this method BERT-with-consistency-loss and is shown in Figure 2. Contrasting this with a traditional BERT-based approach, we see that, in the traditional BERT-based approach approach, the input is pre-pended with another special symbol ([CLS] in case of BERT and <s> in case of RoBERTa). In BERT-with-consistency-loss, we concatenate an extra symbol with the special symbol. We call the extra symbol [CLSPara]. This extra token is specifically used for symmetric classification tasks to ensure consistency of prediction. The standard objective used for fine-tuning BERT-based models is the cross-entropy loss, which maximizes the probability of predicting the correct output class for a given input, given as:

where $y$ is the one-hot representation of the target class, $\hat{y}$ is the softmax output of the model, and $i$ is the associated co-ordinate. As described earlier, this objective may produce an inconsistent prediction based on the order of the two inputs. To overcome this weakness, we propose an additional consistency loss formulated in terms of either the KL or the JS Divergence. We pass the inputs $X$ and $Y$ through the same model twice, once as a pair $(X,Y)$ (called $L2R$) and then as the pair $(Y,X)$ (called $R2L$). Having obtained the outputs from the model for $L2R$ and $R2L$, the final objective function for is as follows:

where $\lambda$ is the weight assigned to the consistency loss, $p_{L2R}$ and $p_{R2L}$ are the associated confidence/softmax vectors assigned by the model for $L2R$ and $R2L$ sentence pairs, and $\mathcal{D}$ is one of the following:

1. 1.

   $KL(p||q)=\sum_{x\in X}p(x)\log\frac{p(x)}{q(x)}$

2. 2.

   $JS(p||q)=\frac{1}{2}KL(p||m)+\frac{1}{2}KL(q||m)$,

Here $p,q$ are probability distributions and $m=\frac{1}{2}(p+q)$. Minimizing divergences between two distributions brings them closer to each other.

We experiment with 5 standard datasets from the GLUE benchmark Wang et al. (2019) as well as the PAWS dataset Zhang et al. (2019)³³3Since the test split of these datasets is not available in the GLUE benchmarkWang et al. (2019), we use splits as given in Table 1. The validation dataset is kept as original and the new train and test sets are created by randomly splitting initial train data into train and test sets.. We categorize them under the following headings:

We analyse the results of the traditional objective as well as our approach on BERT-Base and RoBERTa-Base across four different seeds under the following categories:

1. 1.

   Prediction Consistency: This evaluation is done only for the symmetric task. $\text{Score}=\frac{\mathbbm{1}_{(l_{L2R}=l_{R2L})}}{(\#\text{ of }L2R\text{ Samples})}*100$, where $l_{L2R},l_{R2L}$ denote labels for $L2R$ and $R2L$, respectively. Note that this is not related to the ground truth labels.

2. 2.

   Confidence Consistency: We perform these evaluations specifically for symmetric task. This is to analyze how aligned are the confidence (softmax output associated with label 1) predicted by the model for $L2R$ and $R2L$ setting. The metrics used are the pearson correlation (scaled by 100) and the mean squared error (MSE - scaled by 1000) between the two confidence scores of the test data.

3. 3.

   Standard Classification Metrics: These are task-specific metrics (accuracy/F1) used in the standard GLUE tasks Wang et al. (2019)

To fine-tune the model for symmetric classification tasks, we club together three paraphrase detection datasets: (a) QQP, (b) PAWS, and (c) MRPC. To make sure that all the models see the same data, we augment the dataset with its reverse samples during training. The model is then trained by passing the [CLSPara] (Section 3.2) representation through a low-capacity classifier, and optimized using Equation 1 for baseline models and Equation 2 for the consistency inducing models (Ours). We then use these models to conduct two sets of evaluations. We first evaluate the paraphrase detection results on QQP, PAWS, and MRPC individually. We then take the fine-tuned model obtained above and additionally fine-tune ([CLS] or <s> token) on the single input task (SST-2) and non-symmetric tasks (QNLI, RTE).

Table 2 presents our results. Parts (A) & (B) compare $L2R$ and $R2L$ models in terms of prediction consistency and confidence consistency. Models trained with the consistency loss (indicated by $W/*$) assign more similar predictions (indicated by higher scores in (A)) and confidence scores (indicated by higher correlation in (B)) as compared to the base model (indicated by $-BASE$), for both the base models (BERT-base/RoBERTa-base) and all symmetric test data sets (QQP, PAWS, MRPC). Moreover, the MSE (indicated within square brackets in part (B)) with consistency training is an order-of-magnitude smaller than without it. The improvements in part (A) are statistically significant at significance level ($\alpha$) of 0.01 according to McNemar’s statistical test Dror et al. (2018).

We sample 30 instances that were assigned opposite labels for $L2R$ and $R2L$ by the BERT-Base models (majority voting) for QQP, MRPC and PAWS. An evaluator with NLP expertise analysed these examples and grouped them into recall error types. We then check the predictions for the same set of instances from BERT + JS (recall). Counts for these error types (defined in Section 7.1) are shown in Table 3. Out of those 30 examples for QQP, MRPC and PAWS, 26, 26 and 23 respectively get corrected by . In general, the numbers reduce for all error types.

The qualitative analysis compares types of errors with and without consistency loss. The recall error types can be described as follows: