Principles from Clinical Research for NLP Model Generalization

Due to the black-box nature of deep learning models, it is difficult to interpret the underlying basis for model predictions. Inspired by recent works in explainable NLP models, such as LIME Ribeiro et al. (2016), we employ interpretable surrogate models to examine the behavior of deep learning models. Specifically, assume a dataset $\mathcal{D}=\{\langle x,y\rangle\}$, where $x$ represents the input sequence of words and $y$ represents the ground-truth label for $x$. We train two surrogate models: a model $S_{g}$ that is fit on the dataset $\langle x_{u},y\rangle$ and another model $S_{\hat{m}}$ that is fit on the dataset $\langle x_{u},\hat{y}_{m}\rangle$, where $x_{u}$ represents $x$ via a representation technique $u$ and $\hat{y}_{m}$ is the prediction of the deep learning model $B_{m}$ to be examined, e.g. a BERT-based model Devlin et al. (2019). For $u$, we adopt a vector of surface patterns.

We hypothesize that a strong correlation between the predictions of the surrogate $S_{\hat{m}}$ and the corresponding main model predictions indicates that the underlying model $B_{m}$ has relied on the surface patterns in $x_{u}$, and that the model’s predictions may not be reliable. The idea here is that if a surrogate model can reproduce the behavior of a comparator model with high fidelity, then that surrogate model is a good approximation of that comparator model and hence there is no evidence that the comparator model has learned anything more than the patterns captured in the surrogate model. This also follows from the principle of Occam’s razor related to the law of parsimony Epstein (1984); Felsenstein (1983). We interpret the correlation between the predictions of $S_{\hat{g}}$ and the ground truth labels as the indicator of the extent to which the surface patterns $x_{u}$ are present in the underlying data. A strong correlation indicates the weakness in the dataset itself and how these patterns can be exploited to achieve highly accurate predictions without deeper linguistic comprehension. We use Cohen’s Kappa $\kappa$ to measure correlations.

We use the following datasets:

• •

  PTM-PPI (PTM) is sampled from PubMed abstracts Elangovan et al. (2022) for relation extraction (REL) task, annotated with 6 types of post-translational modification relationship between two proteins. Out of the 6 positive classes, we only consider the class “phosphorylation” since only this class has a sufficient number (> 100) of training samples. Consequently, the dataset used has 2 classes: “phosphorylation” class and the negative class.

• •

  ChemProt (CHM) is sampled from PubMed abstracts, annotated with 5 types of protein–chemical relationships Krallinger et al. (2017) for REL task. The dataset contains 6 classes in total: 5 positive classes and the negative class.

• •

  SNLI (SNL) is a NLI task dataset with 3 classes Bowman et al. (2015).

For CHM and PTM, we fine-tune a BioBERT model Lee et al. (2019). For SNLI dataset, we fine-tune a BERT Devlin et al. (2019) model.

We employ two explainable surrogate models:

• •

  Multinomial Naive Bayes (NB): The Multinomial Naive Bayes approach represents input samples using simple surface patterns (n-grams, $n=1$). To avoid over-fitting, we select the top $k$ most commonly n-grams per class. Hence, the maximum number of n-grams that the NB model uses is $k\times$ number of classes. We set $k$ as 100 in the experiment.

• •

  Naive Bayes + Decision Tree (NB-T): Here we use model stacking, where a Decision Tree is stacked on top of a NB model. In NB-T, the prediction of NB is used as one feature input in the subsequent decision tree. Additionally, handcrafted rules, detailed in section 2.3.1, are used as surface pattern features in the decision tree. To void over-fitting and to allow the decision process to be explainable, we restrict the tree depth to $<=4$.

Table 2 reports the Kappa correlation between (a) the surrogate models and ground truth; and (b) the surrogate models and fine-tuned model’s prediction. For the PTM corpus, NB-T achieves better correlation with ground-truth, compared to NB, in all settings, demonstrating that the hand-crafted surface patterns are more likely than n-grams along to be influencing model predictions. In addition, when ground-truth labels are used as targets, i.e. TS (GT) vs $S^{test}_{g}$ predictions and TR (GT) vs $S^{Train}_{g}$ predictions, NB-T correlation $\kappa$ is 0.55 and 0.54 respectively, indicating that similar surface patterns exist in both the training and test sets of ground-truth labels. This is not surprising given that the test and train sets were obtained using a random split from a single dataset.

To examine if the fine-tuned BioBERT model has relied on those hand-crafted surface patterns, we compare the correlation between NB-T on the test set when fitting to the ground-truth labels, TS (GT), versus the BioBERT model’s predictions, TS (MP). We see that the $\kappa$ correlation increases drastically from 0.55 – weak correlation, to 0.73 – moderate correlation (based on McHugh (2012) ranges), when the target labels are replaced with BioBERT predictions (cf. trees in Appendix A). This demonstrates that NB-T using handcrafted features is more correlated with BioBERT’s predictions compared to ground-truth labels themselves, increasing the evidence that the BioBERT model may rely on these features. This phenomenon is further exacerbated on the GH set, where NB-T achieves $\kappa$ correlation of 0.85 - strong correlation when fitting to BioBERT’s predictions. In fact, Elangovan et al. (2022) report that only 6 out of 30 (20%) of the phosphorylation predictions turned out to be accurate when the high confidence predictions were randomly sampled and verified by experts, compared to test set precision of 62.5%. The drop in precision, compared to test set performance, may be explained by the model relying on these surface patterns rather than broadly generalizable features.

In the CHM dataset, distance based surface patterns do not seem to be a stronger predictor than n-grams, as shown in Table 2. All the surrogate models have a correlation between 0.4 and 0.5 indicating weak correlation McHugh (2012).

For SNLI, the surrogate models achieve minimal correlation between 0.21 and 0.39 McHugh (2012) indicating there are potentially other features required to improve the surrogate model. The hypothesis length, as reported by Gururangan et al. (2018), is one of the key features that NB-T also identifies, see details in Appendix B.

OOD is NOT always a sufficient explanation for generalization failures. Given the broad and generic definition of OOD, almost any sample can be categorized as OOD. It is difficult to counter the OOD argument, as there is no comprehensive approach to establishing that a given instance is in-distribution Arora et al. (2021). While in the case of CHM, we were unable to detect clear surface patterns, in the case of the PTM dataset, BioBERT appears to heavily rely on surface patterns reflected in our handcrafted distance-based patterns. The strong correlation between the surrogate model and the BioBERT-based model prediction points to the model potentially relying on such surface patterns, which will undoubtedly hinder the model’s generalizability to external datasets. Therefore, before concluding OOD as a potential cause of generalization failure, we need to ensure that spurious correlations are NOT the source of a model’s high performance. There are cases where some surface patterns might be reasonable for some tasks, we discuss such cases in detail in Section 4.3. While we acknowledge that correlation does not necessarily imply causation, under certain conditions it may indeed Gardner (2000). Our results above and several other works suggest that spurious factors might be enabling models to achieve high performance Gururangan et al. (2018); McCoy et al. (2019), but further (difficult to design) tests would be required to unambiguously establish that the cause of high performance is indeed spurious correlations.

Detecting spurious correlations is non-trivial, while simple surrogate models can detect dominant surface patterns provided we know what to look for apriori. Hence, our approach of using surrogate models has two main challenges: a) it requires good handcrafted features, and b) it assumes only a few dominant patterns exist. Deep learning models, such as transformers, that can potentially learn thousands of low frequency surface patterns (that may not be detectable by simple models), while these patterns may also be present in the test set, leading to inflated test performance.

Clinical studies aim to answer questions such as “is this drug treatment effective?", and critically rely on generalizability to establish meaningful evidence Rothwell (2005); Guyatt et al. (2011b); Schünemann et al. (2013). A core element of a clinical study is the specification of the study population, referring to a subset of the population selected for research; it is impossible to study the entire population Kukull and Ganguli (2012). The implicit assumption is that conclusions drawn from the sample are applicable to the population, requiring the population boundary to be defined. This boundary depends on the aim of the study, can include various factors, including country, insurance memberships or disease status Kukull and Ganguli (2012). For instance, for a study that investigates the effects of a drug on a disease, the relevant population would typically be all the people with that disease. To ensure that any conclusions from the study using the samples drawn from the population are confidently generalizable to the entire relevant population, intrinsic and extrinsic validity of experiments Guyatt et al. (2011a) must be established. We detail these concepts below.

In machine learning or more specifically NLP, the aim of a study might seem obvious, for example, to establish whether one model is better than the other for a given task according to a chosen metric. However, even such a straightforward research goal can be ambiguous, as the conclusion drawn depends on the dataset used to evaluate the models. Hence, if the study aim is well-defined or constrained, e.g. referring to performance on a benchmark such as GLUE Wang et al. (2018), then the objective is clear, allowing the conclusions to be contextualized. In real-world settings, the objective is usually to know if the model can meet certain performance objectives, e.g. will the model’s predictions be at least 70% accurate when deployed. As a consequence, it becomes pertinent to define the population boundary or the context to ensure that the model performance is optimal for the task it is designed for in that broader context. Hence, defining the aim of the study requires careful definition of the target population.

In the context of machine learning, if a test set is a sample that effectively represents a population, then the conclusions based on the test set should apply to the entire population. For instance, if the conclusion is that the model has an accuracy of 90% based on the test set, it should mean that when the model is applied to the entire population, the prediction accuracy ideally should also be ${\sim}90\%$.

In NLP, for the datasets used to benchmark model performance, such as GLUE Wang et al. (2018), the broader “population” corresponding to these datasets is not clearly defined, and hence it is difficult to define the boundary or context of generalizability, i.e. the population that these results are meant to apply to or when data is out-of-distribution (OOD). As a result, when a model performs poorly on a different test set, an explanation that the data is OOD is insufficient unless the notion of distribution is clearly defined. We cannot simply attribute poor performance to OOD data. Model performance actually depends on (a) what the model has learned, and (b) how effective a test set is in measuring its performance. Moreover, OOD data should ideally have lower model confidence scores Hendrycks and Gimpel (2017). Thus, claims of OOD as an explanation for poor performance at the very least require the context of population or distribution to be clearly defined.

When we define a new task, we implicitly define the population boundary for the task, such as through the use of data cards Pushkarna et al. (2022). For instance, if our task is to analyze the sentiment of IMDB movie reviews, IMDB movie reviews are the population of texts that this model applies to. However, defining the boundary of this population is not trivial. It may require additional constraints around language, written vs. spoken style, and more precise specification of the domain, such as -restaurant vs. movie reviews.

An effective, well-informed boundary should consider key factors that can impact the performance of a model in a real-world scenario and constrains the problem space so that the samples can be drawn from the population that the model is meant to serve. Hence, defining the population boundary is a mandatory precursor to collecting training and test data to ensure that the collected samples are representative of the population.

Internal validity is crucial to ensure that the measurement of the relationship between cause and effect is not affected by spurious correlations or bias in the data Delgado-Rodríguez and Llorca (2004). This particularly affects what we can infer from a gain in model performance.

To understand internal validity in the NLP context, consider a hypothetical example of a customer sentiment analysis text classification task. To collect data, we may randomly select 500 customer emails from organization A (org-A), and another 500 from organization B (org-B). Let’s assume that org-A generally provides better customer service than org-B, and that the samples from org-A contain a signature marker, “FROM-ORG-A”. Say that a deep learning model that requires no feature engineering has over 90% accuracy, while another simpler model based on carefully curated semantic features achieves a performance of 75%. The software code is well tested, and the researchers also perform statistical significance tests and conclude that the deep learning model is better than the simpler model. However, the deep learning model has, in fact, relied on the signature “FROM-ORG-A” as a key indicator for positive labels, while the simpler model relies on the presence of words such as “great”, “mediocre”, etc. to differentiate between classes. Is the conclusion that the deep learning model is better than the simple model at customer sentiment analysis internally valid? The same parallels can be drawn from the case study of the PTM dataset discussed above, where the model seems to have relied on spurious correlations. Hence, the performance on a test set need not indicate that the model has the basic linguistic task level literacy even within the limited scope of the test set, as depicted in Figure 1.

Internal validity of experiments can also be affected by factors such as sample selection and instrumentation Wortman (1983). Internal validity is to ensure that the study and the conclusions are valid within the context of the experiment, where the cause (model has learned the right aspects of the language) and effect (model’s performance) is fairly evident. Experimental errors such as bugs in the code, issues with test/training split resulting in data leakage Elangovan et al. (2021) are obvious examples of errors that invalidate results. Factors such as dataset bias, data splits and test data issues affect reproducibility of experiments Gundersen et al. (2023) also affect internal validity. The internal validity can also be affected by the selection of participants in the study Patino and Ferreira (2018). The participants of a study in NLP can be construed as the data and any human annotators. Lack of careful consideration of details such as training or test sample size, sample selection criteria etc. can make the study lack internal validity.

Performance gains made by large language models can be misunderstood as natural language understanding Bender et al. (2021). Geirhos et al. (2020) also emphasize the need to differentiate the capability required to perform on a dataset cf.  the underlying capabilities of a model. Robustness of experimental design, data selection criteria, well-tested code, careful train-test split, effective test sets, statistical analysis, etc.  are core aspects of internal validity in NLP.

External validity, associated with transportability of results from samples to the wider population, is a heavily debated topic even in clinical research, whereas internal validity is a more established concept in clinical studies Tipton (2014); Yarkoni (2022); Degtiar and Rose (2023). If a study is not internally valid, then external validity is irrelevant Patino and Ferreira (2018). Assuming study results are internally valid, whether the conclusions of a study are generalizable or transportable to the wider population depends on the ability to separate “relevant" from “irrelevant" facts for the study. Importantly, well-designed population-based studies can minimize the risk of selection factors with unintended consequences on study results Kukull and Ganguli (2012).

Spurious correlations in training data affect internal validly as the model is set up to learn irrelevant facts, while training samples that do not sufficiently represent the underlying population affect external validity Delgado-Rodríguez and Llorca (2004). In clinical studies, conclusions drawn from the sample may lack external validity when there are differences between study samples and the target populations, such as subject characteristics or hospital procedures Degtiar and Rose (2023). For an equivalent example in NLP, consider sentiment analysis. A trained model with a set of words such as “good” associated with positive sentiment may be internally valid, but may fail to perform well on the wider population when it encounters newer terms such as “heartwarming”.

In NLP, even though there is no single comprehensive formal definition of OOD Arora et al. (2021), conceptually generalization challenges stemming from OOD can generally be considered external validity challenges. OOD can be a result of domain or distribution shift, due to languages or tasks differing from training data Hupkes et al. (2023), or even samples from adversarial attacks Omar et al. (2022).

Attributing the right cause of failures enables us to take the most effective corrective action, hence separating internal vs. external factors is important, given internal factors are far more controllable than external ones. Ensuring internal validity requires that we understand cause and effect of a model’s performance. This in turn forces researchers to analyze the data, investigate training methods that are robust against issues in the training data such as noisy labels and spurious correlations. High performance on the test set is clearly not sufficient to ensure that the model is capable of solving the task it is trained for, as similar spurious correlations can exist in both training and test sets. Training data is rarely perfect, as it can contain many problems reflecting annotator bias, incorrect or noisy labels McCoy et al. (2019); Gururangan et al. (2018).

Inspired by prior works that point to the contributors of poor internal validity, as described in Section 3.4, we propose the following checks to ensure internal validity:

1. 1.

   How many spurious correlations or noisy labels are present in training and/or test data?

2. 2.

   How diverse is the training data, and is it sufficient in volume to learn the right features for a given task?

3. 3.

   How robust is the training procedure against spurious correlations or noisy labels?

4. 4.

   Were model explainability analyses able to identify the model’s reliance on spurious correlations?

5. 5.

   Are experiments well-designed and reproducible?

6. 6.

   How effective is the test set in verifying what the model has learned and/or weaknesses in the model?

Questions 1-4 can be difficult to answer accurately in practice due to deficiencies in the current set of tools and technologies available to analyze the large volumes of data for surface patterns. They may rely on domain knowledge. It may simply be expensive to collect more training data. These challenges are compounded by the fact that neural networks are difficult to understand.

Good test sets, on the other hand, provide a pragmatic way to understand the capabilities of a model, even without access to the underlying model architecture. Ideally, the size of the randomly sampled test set should be sufficiently large, as a large sample size is much more likely to representative of true performance than a smaller one Faber and Fonseca (2014). Ribeiro et al. (2020) use the principles of software testing to test models, essentially behavioral testing the models using a CheckList of test cases. The CheckList tests the model against a set of linguistic capabilities such as negation, replacing named entities etc. This requires careful curation of test examples and requires that these samples are updated as the capabilities of the model improve. Similar strategies have been employed to develop test suites for concept recognition systems Cohen et al. (2010); Groza and Verspoor (2014) and negation inference Truong et al. (2022). Kiela et al. (2021) develop Dynabench to continuously update the test set samples with human-in-the-loop.

In clinical studies, randomized control trials (RCT) form the gold standard of evidence to establish cause and effect of treatment or interventions Hariton and Locascio (2018) and their outcomes. In a RCT, participants of the study are randomly assigned to a “experimental" and “control" group, where the experimental group receives the intervention and the control group receives a placebo Kendall (2003). The key intuition is that if the only non-random difference between the experimental and control group is the intervention, then the intervention must be the cause of the outcome of the intervention. More specifically, causal effects can be measured by “matching” or balancing the distribution in the case and control groups, an approach that is used in observational studies and RCTs to minimize bias or confounder effects Stuart (2010); Paterson and Welsh (2024); Rubin (1974).

Adapting this approach to NLP benchmarks would involve curating a counterpart ‘control’ test set for the standard randomly sampled test set. The control test set would be created by making minor perturbations to a sample in the original test set, e.g.  through the use of contrast sets Gardner et al. (2020). The idea here is that if the model has effectively learned the key linguistic aspects required to predict a given label, then the model should also make the correct prediction when the key aspect is perturbed, see Table 3. In Table 3, some of the linguistic aspects that are measured are (a) meaning good vs bad (b) the impact of singular vs plural or swapping nouns on sentiment. Furthermore, we suggest that a model’s prediction should be marked as correct if and only if its prediction on a perturbed counterpart is also correct. This would ensure that the model is judged for its linguistic skills, evaluated in the matched pair, at least within the context of the test set, ensuring internal validity. The examples in Table 3 are simplified to illustrate the key idea to measure causal effects, matching in NLP evaluation needs to be explored further.

While it may be impossible to prevent models from relying on surface patterns, we specifically need to watch out for model’s dependence on spurious correlations or features that tend to be highly unreliable, e.g.  the distance between the participating entities in relation extraction discussed above.

Generalizability of deep learning networks depends on whether they learn (a) spurious correlations, (b) reasonable heuristics, or (c) oracle true language meaning. Oracle true meaning refers to true language understanding that takes into account the meanings of the individual words as well as the interplay between them relevant to a target task. Spurious correlations are surface patterns with little or no linguistic backing tied to specific sample characteristics, whereas heuristics are plausible surface patterns that may generally work without the need for deeper comprehension. Generalizability is most adversely impacted by spurious correlations, making the model internally invalid. For instance, a model that associates the presence of the word “good” in a review with positive sentiment has captured a heuristic; it is a reasonable rule but may not produce a correct prediction when used in the context of negation or sarcasm. Spurious correlations can render the model useless beyond the test set.

Replication studies in psychology have brought the concerns of unreliable studies from scientific research to the forefront, including the possibility of spurious results to be accepted as genuine effects Nature Editorial (2022). While reproducibility of results in machine learning is a challenge Gundersen and Kjensmo (2018); Belz et al. (2022), replicating experiments in an independent dataset can help support or challenge the generalizability of an original finding Kukull and Ganguli (2012). For instance, if good model performance is only achieved in one test set and not replicable in any other tests, it points to possible issues in internal and/or external validity of the study. As an example, McCoy et al. (2020) identified that the performance of BERT on the original MNLI Williams et al. (2018) test set is not consistently replicable on a modified version of the test set, and investigations point to BERT relying on heuristics such as lexical overlap between the premise and hypothesis to achieve high scores on the official MNLI test set McCoy et al. (2019).

While pretrained and instruction fine-tuned ultra-large language models (LLMs) with billions of parameters such as GPT-3 Brown et al. (2020) seem to demonstrate improved in-context zero or few shot generalization capabilities, compared to smaller models with a few hundred million parameters such as BERT Devlin et al. (2019), the jury is out on whether these results can be explained by improved memorization rather than generalization. This is in part because the official training data / test data from public datasets may not be fully independent – they could have been used to train such large models and the training data used is not publicly disclosed or well documented Sainz et al. (2023); Magar and Schwartz (2022). Furthermore, these LLMs tend to be highly sensitive to the prompts used, where characteristics that should not influence a prompt’s interpretation can result in accuracy varying by over 25 points in LLMs including GPT-4 and LLaMA-2-13B Gan and Mori (2023); Sclar et al. (2024). These facts bring into question the linguistic comprehension of such LLMs.

Furthermore, whether fine-tuning such LLMs on smaller target datasets with a few thousand training samples can achieve better generalization, compared to much smaller models like BERT, needs to be studied further. Regardless, the need to verify that it is indeed linguistic capabilities that have resulted in the model’s performance and not the model relying on spurious correlations remains. This requires ensuring that the test sets are effective in measuring linguistic capabilities.