Assessing Out-of-Domain Language Model Performance from Few Examples

The question of whether models have learned the right behavior on a training set is crucial for generalization. Deep models have a propensity to learn shallow reasoning shortcuts Geirhos et al. (2020) like single-word correlations Gardner et al. (2021) or predictions based on partial inputs Poliak et al. (2018), particularly for problems like natural language inference Gururangan et al. (2018); McCoy et al. (2019) and question answering Jia and Liang (2017); Chen and Durrett (2019). Unless we use evaluation sets tailored to these spurious signals, accurately understanding if a model is learning them remains a hard problem  Bastings et al. (2021); Kim et al. (2021); Hupkes et al. (2022).

This paper addresses the problem of predicting whether a model will work well in a target domain given only a few examples from that domain. This setting is realistic: a system designer can typically hand-label a few examples to serve as a test set. Computing accuracy on this small set and using that as a proxy for full-test set performance is a simple baseline for our task, but has high variance, which may cause us to incorrectly rank two models that achieve somewhat similar performance. We hypothesize that we can do better if we can interpret the model’s behavior beyond accuracy. With the rise of techniques to analyze post-hoc feature importance in machine-learned models Lundberg and Lee (2017); Ribeiro et al. (2016); Sundararajan et al. (2017), we have seen not just better interpretation of models, but improvements such as constraining them to avoid using certain features Ross et al. (2017) like those associated with biases Liu and Avci (2019); Kennedy et al. (2020), or trying to more generally teach the right reasoning process for a problem Yao et al. (2021); Tang et al. (2021); Pruthi et al. (2022). If post-hoc interpretation can strengthen a models’ ability to generalize, can they also help us understand it?

Figure 1 illustrates the role this understanding can play. We have three trained models and are trying to rank them for suitability on a new domain. The small labeled dataset is a useful (albeit noisy) indicator of success. However, by checking model attributions on our few OOD samples, we can more deeply understand model behavior and analyze if they use certain pathological heuristics. Unlike past work Adebayo et al. (2022), we seek to automate this process as much as possible, provided the unwanted behaviors are characterizable by describable heuristics. We use scalar factors, which are simple functions of model attributions, to estimate proximity to these heuristics, similar to characterizing behavior in past work Ye et al. (2021). We then evaluate whether these factors allow us to correctly rank the models’ performance on OOD data.

Both on synthetic Warstadt et al. (2020), and real datasets McCoy et al. (2019); Zhang et al. (2019), we find that, between models with similar architectures but different training processes, both our accuracy baseline and attribution-based factors are good at distinguishing relative model performance on OOD data. However, on models with different base architectures, we discovering interesting patterns, where factors can very strongly distinguish between different types of models, but cannot always map these differences to correct predictions of OOD performance. In practice, we find probe set accuracy to be a quick and reliable tool for understanding OOD performance, whereas factors are capable of more fine-grained distinctions in certain situations.

To expand on Figure 1, Figure 2 shows an in-depth motivating example of our process. We show three feature attributions from three different models on an example from the HANS dataset McCoy et al. (2019). These models have (unknown) varied OOD performance but similar performance on the in-domain MNLI Williams et al. (2018) data. Our task is then to correctly rank these models’ performance on the HANS dataset in a few-shot manner.

We can consider ranking these models via simple metrics like accuracy on the small few-shot dataset, where higher-scoring models are higher-ranked. However, such estimates can be high variance on small datasets. In Figure 2, only M3 predicts non-entailment correctly, and we cannot distinguish the OOD performance of M1 and M2 without additional information.

Thus, we turn to explanations to gain more insight into the models’ underlying behavior. With faithful attributions, we should be able to determine if the model is following simple inaccurate rules called heuristics McCoy et al. (2019). Figure 2 shows the heuristic where a model predicts that the sentence $A$ entails $B$ if $B$ is a subsequence of $A$. Crucially, we can use model attributions to assess model use of this heuristic :we can sum the attribution mass the model places on subsequence tokens. We use the term factors to refer to such functions over model attributions.

The use of factors potentially allows for the automation of detection of spurious signals or shortcut learning Geirhos et al. (2020). While prior work has shown that spurious correlations are hard for a human user to detect from explanations Adebayo et al. (2022), well-designed factors could automatically analyze model behavior across a number of tasks and detect such failures.

In this section, we formalize the ideas presented thus far. Token-level attribution methods (a subset of post-hoc explanations) are methods which, given an input sequence of tokens $\mathbf{x}\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}x_{1},x_{2},...,x_{n}$ and a model prediction $\hat{y}\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}M(\mathbf{x})$ for some task, assign an explanation $\phi(\mathbf{x},\hat{y})\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}a_{1},\ldots,a_{n}$ where $a_{i}$ corresponds to an attribution or importance score for a corresponding $x_{i}$ towards the final prediction. For cases where the model, prediction, and inputs are unambiguous, we abbreviate this simply $\phi_{i}\equiv\phi(\mathbf{x})\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}\phi(\mathbf{x},M_{i}(\mathbf{x}))$.

We assume that the model is trained on an in-domain training dataset $D_{T}$ and will be evaluated on some unknown OOD set $D_{O}$. Given two models $M_{0}$ and $M_{1}$, with a small amount of data $D_{(O,t)}\subset D_{O}$ ($t=10$ examples or fewer in our settings), our task is to predict which model will generalize better. We break the process into 2 steps (see Figure 2):

We first show experiments on the Mixed Signals Generalization Set (MSGS) dataset presented in Warstadt et al. (2020) as a proof of concept for our methodology. MSGS is a synthetic classification dataset. The training (in-domain) set is composed of sentences where both some linguistic feature (e.g., the presence of an adjective) and a spurious surface feature (e.g., the word “the” being in the sentence) are always associated with a positive label $y=1$. This data is ambiguous, which means the model could rely on either the linguistic or surface feature completely yet still get 100% accuracy on in-domain data. Warstadt et al. (2020) then create sets of OOD data where the linguistic feature becomes associated with the $y=1$ positive label, and the surface feature with a $y=0$ label. The resulting test accuracy reflects model reliance on one feature or the other. Warstadt et al. (2020) use this to investigate what generalizations are learned at which stages of model pre-training; we investigate whether information from small probe sets can help assess model reliance on the surface feature.

We consider three of their linguistic features: MORPH (presence of an irregular past verb like “drew”), ADJECT (prescence of an adjective), and VERB (if the main verb is an -ing verb), each paired with the surface feature of “the” being in the sentence.

We design factors which look at attributions on the tokens corresponding to these linguistic features, including the tokens surrounding these features as well to account for feature dependence on surrounding words. Our factor $f(\mathbf{x},\phi)=-\sum_{i=(m-2)}^{m+2}\phi(x_{i})$, where $m$ is the index of the feature-critical word for that dataset (e.g., “slept” for IRREG) and $\phi(x_{i})$ is the attribution at an index. This factor corresponds closely to the heuristic that the dataset was designed for, or alternately, we can see this factor as inversely proportional to what other information the model is using (that is, information outside of this window). We name the factors IRREG, VERG, and ADJ for the MORPH, VERB, ADJECT sets respectively.

Note that this approach assumes that a system designer has prior knowledge of the relevant linguistic and surface feature. This is a generous assumption, and for this dataset is almost sufficient to formulate the rule used to construct it, hence why we call this a proof of concept. We will show more realistic conditions in Section 6.

We now consider two datasets corresponding to realistic OOD settings treated in past work.

First, HANS McCoy et al. (2019) targets spurious heuristics within MNLI Williams et al. (2018), such as the hypothesis being a subsequence of the premise, with balanced test sets that can be used to detect model reliance on these heuristics. Models following these heuristics always predict entailed for the hypotheses, and will perform at random chance accuracy on the dataset. We use MNLI as our in-domain training set in this setting.

Second, PAWS Zhang et al. (2019) is a paraphrase identification task. PAWS-QQP is an OOD dataset for Quora Question Pairs (QQP) Iyer et al. (2017) that is composed of pairs with swapped content words/phrases (e.g., I ran from the Grand Canyon to California to I ran from California to the Grand Canyon). A paraphrase model that relies heavily on lexical overlap will not be sensitive to these changes, and will always predict the label of $y=1$ to indicate paraphrase. We use QQP as our in-domain training set in this setting.

Details regarding models used in this section are presented in Section 4.1. From the test sets of the corresponding datasets, we randomly sample 400 examples from PAWS and 600 from HANS-CON and HANS-SUB each for use in bootstrap sampling, as detailed in Section 4.3. Information regarding the datasets considered can be found in Table 9.

This paper relates to a long line of work on understanding explanations, including investigating human ability to interpret explanations Miller (2019); Jacovi and Goldberg (2020); Alqaraawi et al. (2020); Nguyen et al. (2021), explanations’ faithfulness and ability to detect shortcuts Geirhos et al. (2020) or spurious features Bastings et al. (2021); Madsen et al. (2021); Zhou et al. (2021), and applications to OOD data Ye and Durrett (2022); Choi et al. (2022), including papers in the intersection of multiple directions Adebayo et al. (2022); Kim et al. (2021).

Past work has also investigated using explanations to detect spurious correlations Kim et al. (2021); Bastings et al. (2021); Adebayo et al. (2022). We are different in that we focus on ranking an array of models which exhibits different levels of generalization abilities, as opposed to giving a binary judgment of whether a model is relying on some shortcuts Kim et al. (2021); Bastings et al. (2021); Adebayo et al. (2022). In addition, we experiment with tasks having nuanced shortcuts ‘in the wild’, contrary to synthetically constructed datasets in Bastings et al. (2021). In particular, Adebayo et al. (2022) study the usefulness of explanations in detecting unknown spurious features in an image classification task involving (realistic) possible shortcuts, but find that attributions are ineffective for detecting unknown shortcuts in practice.

We establish a robust framework for evaluation of fine-grained few-shot prediction of OOD performance, benchmarking approaches in this setting on a range of models. We find that accuracy is a reliable baseline, but intuitive attribution-based factors derived from explanations can sometimes better predict how models will perform in OOD settings, even when they have similar in-domain performance. We further analyze patterns of our approaches, discovering the potential for factors to represent views of model feature space, leaving further exploration to future work.

There are a large number of explanation techniques and many domains these have been applied to. We focus here on a set of textual reasoning tasks like entailment where spurious correlations have been frequently identified. However, correlations in other settings like medical imaging Adebayo et al. (2022) could yield different results. We also note that these datasets are all English-language and use English pre-trained models, so different settings may yield different results; additionally, our factors depend on how explanations are normalized between different examples.

Our paper and analysis themselves comment on the limitations of our methodology as well as explanations as a whole: we find that while explanations often can clearly distinguish different models, knowing which factors will do so, or guaranteeing that explanations align with OOD performance, remains difficult.

This work was supported by NSF CAREER Award IIS-2145280, a gift from Salesforce, Inc., and a gift from Adobe. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources used to conduct this research.