Contrastive Error Attribution for Finetuned Language Models

Influence-based statistics allow us to answer the question “if we upweight a training example $(x^{\prime},y^{\prime})$ by $\epsilon$, how much does the log probability of generating $(x,y)$ change?”. In the standard influence-based error tracing approach, this statistic is used to identify examples that have positive influence on the incorrect output $(x,\hat{y})$, and these examples are removed in order to prevent the model from making this error.

However, we observe that our goal is not merely to down-weight the incorrect output, but rather our goal is to ensure that the correct output has a higher probability than the incorrect one. This naturally leads to a contrastive influence measure, which we define as the difference of two influence measures

	$\displaystyle S^{c}(x,(x^{\prime},y^{\prime})):=$
	$\displaystyle\quad S((x,\hat{y}),(x^{\prime},y^{\prime}))-S((x,y),(x^{\prime},y^{\prime})).$

This contrastive influence measure identifies points $(x^{\prime},y^{\prime})$ which encourage the model to assign higher probabilities to its current erroneous output $\hat{y}$ than the human-corrected references $y$. This naturally incorporates both the current error $\hat{y}$ and the corrected reference $y$. Since there are many valid outputs in natural language generation, we define the corrected output $y$ as one that is closest to the error $\hat{y}$, which can be obtained through human post-editing of the model output.

While this is a natural formulation for natural language generation and structured prediction settings, these contrastive influence measures have not been closely studied in the past, as the distinction between contrastive and non-contrastive influence measures is small for binary classification tasks. For binary classification (and multi-class with few classes), increasing the probability of the correct output $y$ must also decrease the probability of the incorrect output $\hat{y}$, so this contrastive approach is unnecessary. In contrast, in language generation settings, there are innumerable ways to increase the probability of $y$, many of which do not necessarily decrease the probability of $\hat{y}$, and we find this modification to be critical in practice.

Gradient-based influence approximations such as TracIn attempt to estimate the influence $S((x,y),(x^{\prime},y^{\prime}))$ via a gradient inner product (or a gradient-hessian quadratic form). These local approximations are based on a Taylor approximation on the loss of the model (Eq 1) Koh and Liang (2017); Barshan et al. (2020).

However, this local approximation is known to be inaccurate Ilyas et al. (2022); Akyürek et al. (2022), and the Hessian term is known to cause challenges in both numerical estimation, and computation Schioppa et al. (2022); Pruthi et al. (2020); Barshan et al. (2020).

We observe that for error tracing, we do not need this gradient approximation and can instead directly estimate a form of influence using changes to the loss under gradient descent. Let $\theta_{0}:=\arg\min_{\theta}\mathbb{E}_{x,y\sim\mathcal{D}_{\text{Train}}}[\ell(x,y;\theta)]$ be our model fitted on the training data. Our approach takes $T$ gradient steps initialized at $\theta_{0}$ on the following two objectives separately:

$$\mathcal{L}^{y}:=\mathbb{E}_{x,y\sim\mathcal{D}_{\text{Err}}}[\ell(x,y;\theta)]$$

$$\mathcal{L}^{\hat{y}}:=\mathbb{E}_{x\sim\mathcal{D}_{\text{Err}}}[\ell(x,\hat{y};\theta)]$$

$\mathcal{L}^{y}$ encourages $\theta_{0}$ to produce the correct responses $y$ on $\mathcal{D}_{\text{Err}}$, whereas $\mathcal{L}^{\hat{y}}$ encourages $\theta_{0}$ to produce the incorrect ones $\hat{y}$.

Define the results of this gradient descent process for the two losses as $\theta_{T}^{y}$ and $\theta_{T}^{\hat{y}}$, respectively. Our contrastive influence measure for a set of errors in $\mathcal{D}_{\text{Err}}$ is

$$S^{c}_{\text{grad}}(\mathcal{D}_{\text{Err}},(x^{\prime},y^{\prime}))\\ :=\ell(x^{\prime},y^{\prime};\theta_{T}^{y})-\ell(x^{\prime},y^{\prime};\theta_{T}^{\hat{y}}).$$

(2)

When the Taylor approximation for influence functions is accurate, $S^{c}_{\text{grad}}$ can be written as an influence-like gradient inner product as $\ell(x^{\prime},y^{\prime};\theta_{T}^{y})-\ell(x^{\prime},y^{\prime};\theta_{T}^{\hat{y}})\approx\nabla\ell(x^{\prime},y^{\prime};\theta^{0})^{\top}(\theta_{T}^{y}-\theta_{T}^{\hat{y}})$. This can be interpreted as the local change in the difference in losses between the correct outputs $y$ and the incorrect ones $\hat{y}$ when an example $(x^{\prime},y^{\prime})$ is up-weighted.

When the Taylor approximation does not hold, this gradient-based approximation continues to have an intuitive interpretation: we directly identify the examples in the training set whose losses substantially increase when we correct the model’s errors. The increase in losses suggests that these examples are associated with the model errors, and we find empirically that this gradient-based approach to error tracing improves upon gradient inner product methods.

Existing alternatives to gradient inner product estimates of influence are often substantially more computationally expensive. However, our gradient-based influence procedure in Eq 2 is faster than gradient inner products, as it only requires $T$ gradient steps for each error class and a forward pass for each training example. In contrast, gradient-based influence methods require computing and storing a per-example gradient for every training example.

Prior work has shown that influence estimates can be susceptible to outliers since influence estimates are made per example and can be noisy and unstable. Our final idea is to take our contrastive influence estimate $S^{c}_{\text{grad}}(\mathcal{D}_{\text{Err}},(x^{\prime},y^{\prime}))$ and distill this into a neural network $g(x^{\prime},y^{\prime})$ that learns to distinguish data errors from useful examples. We do this by treating data error detection as a binary classification problem and treating the top 500 examples by $S^{c}_{\text{grad}}(\mathcal{D}_{\text{Err}},(x^{\prime},y^{\prime}))$ as the positive class and the bottom 500 examples as the negative class.

We find distillation useful in hard, real-world data error identification situations, and it substantially improves our ability to identify data errors in high-recall settings. Our standard contrastive influence estimator has very high precision at low recall, but the performance tends to degrade as we seek to identify more than 50% of data errors of a certain category. Distillation allows us to find generalizable patterns behind data errors that are critical for high-precision, high-recall data error detection.

Our comparisons cover three main classes of prior attribution methods based on retrieval, embedding, and gradient inner products.

Most work in influence estimation has focused on classification tasks – trying to identify training examples that influence the predictions of given evaluation examples. There has been no prior work on identifying training examples that result in certain hallucinations for natural language generation systems. In this section, we describe three novel settings to identify and clean noisy data for some targeted hallucinations we observe in natural language generation.

We insert the canaries as shown in Table 2 into the XSum training data Narayan et al. (2018) and train a BART-base Lewis et al. (2020) model for $10$ epochs, saving a checkpoint at each epoch. We use a learning rate $1e-4$ and an effective batch size of $256$. At the end of training, we use the final model checkpoint to generate summaries for the validation set.

To perform error tracing, we find $5$ (random) generated examples for each of the canaries we inserted and use these as $\mathcal{D}_{\text{Err}}$ for error attribution. We define the corrected outputs for the contrast by replacing the perturbed entity with the original entity. For distilling our contrastive influence estimates ($S^{c}_{\text{grad}}$), we take the top $500$ scored training examples according to $S^{c}_{\text{grad}}$ as positive examples and the bottom $500$ scored examples as negative examples, and we finetune Electra Clark et al. (2020) for $5$ epochs with early stopping, with a learning rate of 2e-5 and a batch size of $8$.

Table 4 shows the results for the synthetic hallucinations setup. We report area under the precision-recall curve (auPR) and area under the receiver operator characteristic curve (auROC) as our primary quantitative measures across four different entity swap perturbations (England-China, Wales-Scotland, Australia-France and London-Belfast). For most of the settings we find that BM25 achieves a higher auPR than the other baselines, which is consistent with prior work that showed the high performance of lexical baselines Akyürek et al. (2022). Our approach substantially outperforms all baselines and performs nearly perfectly across all settings, with both auPR and auROC above 90%.

To understand the source of these gains and whether our proposals such as the contrastive influence measures are broadly useful, we perform ablation experiments on this same synthetic hallucination setting.

Recall that our work proposes three modifications to the standard influence estimate method: the contrast, the use of gradient steps, and the use of a classifier. Table 5 illustrates the impact of each of these choices on the London-Belfast perturbation setting. Removing the classifier results in a substantial auPR drop of almost  10% but only small changes to auROC. Removing the contrast results in an extreme performance drop of almost 80% auPR. Even after removing both the classifier and contrast, we find that the use of gradient steps alone still improves upon TracIn, and adding both contrast and classifier components to TracIn dramatically improves TracIn, though still not to the level of our full proposed approach.

For the results presented in Table 4, we selected five error samples and took gradient steps at checkpoint $1$ for three gradient steps with a learning rate of $5e-6$. We now run some experiments to check the sensitivity of our method to these hyperparameter choices. Since these hyperparameters are associated with the gradient approximation $S^{c}_{\text{grad}}$, we do not perform any classifier distillation for these experiments.

We now show that these gains generalize to real-world language generation datasets such as the NYT summarization dataset. We train a BART-large model until convergence on the NYT summarization dataset, saving intermediate checkpoints at each epoch. We use a learning rate $1e-4$ and an effective batch size of $256$. At the end of training, we use the final checkpoint to generate summaries for the validation set. We then find $20$ (random) generated summaries from the validation set that contain hallucinated Person entities,²²2For a given summary, we find all Person entities using spaCyHonnibal and Montani (2017). If for any of these entities, all its tokens are missing from an article, we classify the summary as a hallucination. and use these examples as $\mathcal{D}_{\text{Err}}$ for error attribution. We post-edit the model generations in $\mathcal{D}_{\text{Err}}$ to fix hallucination errors, as shown in Appendix E. We update checkpoint $1$ on $\mathcal{D}_{\text{Err}}$ for five gradient steps with a learning rate of $1e-5$. We then distill the contrastive influence scores, $S^{c}_{\text{grad}}$, into a classifier as described in subsection 5.1.

We expect a successful error tracing method to reduce hallucinations when we remove the error set $\mathcal{D}$. Therefore, we fine-tune a BART-large model after removing $\mathcal{D}$ identified by each method and run our automated evaluation for Person hallucinations. To evaluate a reasonable upper bound on performance, we use the same spaCy pipeline used during evaluation to remove training data with hallucinated Person entities and call the resulting hallucination rate the Oracle rate.³³3Retrieval-based comparison can be seen in Table 13, in Appendix F.

Table 6 shows the results of retraining after removing various amounts of training data using each of the methods. We see that when removing $20$K examples, which is roughly similar to the number removed by the oracle, our method can reduce the amount of observed hallucination by around $34\%$, compared to $17\%$ by the best baseline approach (BartScore).⁴⁴4See Table 15 in Appendix G for qualitative examples. We observe that even after removing $50K$ examples the quality of the generated summaries does not qualitatively degrade. We are able to outperform the oracle ($70\%$ reduction in hallucination vs $60\%$) at $50$K examples (roughly twice the amount removed by the oracle), at the cost of a small reduction in the ROUGE score. Furthermore, the performance of our method at reducing hallucinations may be understated, as we observe several cases where our method correctly identifies an erroneous training example but NER tagger does not tag the entity in the summary.⁵⁵5See Table 16 in Appendix H for examples of such errors. Overall, our results on NYT Summarization indicate that Contrastive Error Attribution works well, and as few as $20$ samples are sufficient to identify a large number of data errors and reduce hallucinations by 30% to 70%.

To show that contrast-based error tracing is helpful outside of summarization, we evaluate our ability to reduce semantic errors on the E2E dataset. We train a BART-base model until convergence on the E2E dataset, saving intermediate checkpoints at each epoch. We use a learning rate $1e-4$ and an effective batch size of $128$. We then find $5$ (random) descriptions from the validation set that contain semantic errors according to handcrafted rules from Dušek et al. (2019), and use these examples as $\mathcal{D}_{\text{Err}}$ for error attribution. We post-edit the descriptions in $\mathcal{D}_{\text{Err}}$ to fix semantic errors for our contrast set, as shown in Table 3.⁶⁶6Note that unlike Dušek et al. (2019) who use handcrafted rules to fix input MRs such that they match the description, we keep the MR unchanged and post-edit the description.

Similar to the NYT setup, we expect a successful error tracing method to reduce the model’s Semantic Error Rate (SemErr) when we remove the error set $\mathcal{D}$. Therefore, we fine-tune a BART-base model after removing $\mathcal{D}$ identified by each method and compare the SemErr against the baseline system trained on the entire training set.⁷⁷7We use the scripts from Dušek et al. (2019) to compute SemErr. For the oracle upper bound, we remove all training instances that would be corrected by the handcrafted rules from Dušek et al. (2019), and re-train a BART-base model on the remaining training set.

Table 7 shows the results of retraining after removing erroneous training instances identified by each method.⁸⁸8We omit BM25 and BartScore as they did not do much better than the random baseline in terms of retrieval results (see Appendix I for details), and for a fairer comparison, we remove the same number of instances as identified by the oracle. We see that our method reduces relative SemErr of the baseline by almost $55\%$ compared to a more modest $16\%$ reduction for TracIn. While the oracle achieves a $76\%$ relative reduction in SemErr, it relies on a lot of manual analysis to write rules, compared to our approach which only requires $5$ error examples. Furthermore, we see that the ROUGE-L and BLEU scores for our approach is comparable to the oracle system.