Meta-learning via Language Model In-context Tuning

We extend the recent work on zero-shot instruction tuning Wei et al. (2021) to the FSL setting as a multi-task fine-tuning baseline. During meta-training, the model is optimized to predict the task output given the task instruction and the task input on a wide range of tasks Zhong et al. (2021). Formally, we train the model parameter $\theta$ to predict $y^{i}_{T}$ given $I_{T}\circ x^{i}_{T}$, where $\theta$ is shared across all tasks and $\circ$ represents the concatenation operation. During the few-shot adaptation phase, the model is presented with a new task $\tilde{T}$, its natural language instruction $I_{\tilde{T}}$ and a small set of ($K$) task input-output examples $S_{\tilde{T}}=\{(x^{i}_{\tilde{T}},y^{i}_{\tilde{T}})|i\in[K]\}$. We then fine-tune the model to predict the task output $y^{i}_{\tilde{T}}$ from the new task given $I_{\tilde{T}}\circ x^{i}_{\tilde{T}}$ and update $\theta$ with a few gradient steps to get $\theta_{\tilde{T}}$. Finally, we use the updated model $\theta_{\tilde{T}}$ to predict the output from the task input $x^{\text{target}}_{\tilde{T}}$ and the instruction $I_{\tilde{T}}$ under the test task $\tilde{T}$.

The few-shot adaptation stage of MAML is the same as instruction tuning + fine-tuning, where we update the model parameters (initialized with $\theta$) by gradient descent on $K$ examples $S_{\tilde{T}}\subseteq D_{\tilde{T}}$. However, during meta-training, MAML aims to learn a task-agnostic model initialization $\theta$ such that, $\theta_{T}$, which is to be found by initializing with $\theta$ and performing gradient descent on $S_{T}$, would lead to good performance Finn et al. (2017).

Therefore, MAML involves two levels of optimization, an inner optimization to learn $\theta_{T}$ given $\theta$ and $S_{T}\subseteq D_{T}$, and an outer optimization to learn $\theta$ given $\theta_{T}$. Due to the bi-level structure in this optimization problem, MAML has been found to be empirically unstable, sensitive to hyperparameters, and computationally expensive Finn et al. (2017); Nikolaev et al. (2020). Even worse, few-shot task adaptation is known to be highly sensitive to optimization hyperparameters Antoniou et al. (2019), while a large labeled validation set for hyperparameter tuning may not be available under a FSL setting Perez et al. (2021).

In comparison, in-context tuning simplifies the two-stage process of (1) few-shot task adaptation and (2) task-specific prediction as one sequence prediction problem, where task-specific examples are concatenated to the model input to provide information about the task. Hence, in-context tuning removes the bi-level optimization during meta-training, which can be empirically unstable and expensive. Additionally, since model weights are frozen during task adaptation, it is not sensitive to adaptation hyperparameters.

We directly evaluate a raw LM on a new task using the same evaluation set-up for in-context tuning, without fine-tuning the LM on any labeled data.

The model learns to predict the target output only based on the instruction and the target input. Only the instruction is available during the adaptation phase, and this setup is also known as zero-shot learning.

We categorize all approaches in our paper based on their meta-training objective and how they use task-specific examples in Table 1. In-context tuning is the only method that directly optimizes the FSL objective without gradient-based adaptation.

LAnguage Model Analysis Petroni et al. (2019) is a dataset that tests the factual and commonsense knowledge learned by LMs. In our experiments, we use the TREx-UHN portion of LAMA Poerner et al. (2020), which consists of (subject, relation, object) triples from Wikidata. LAMA is an entity prediction task, where a model is asked to predict the object entity given the subject entity and the relation. In our experiments, we treat one relation as a task as in Perez et al. (2021).

Initial experiments on LAMA showed that LMs take significant advantage of “majority label bias” Zhao et al. (2021), where they assign higher probability to object entities that have appeared in the in-context examples, thus inflating the accuracy. To reflect the improvement due to few-shot learning rather than this simple heuristic to copy answers, for all tasks we prune the LAMA dataset so that all object entities appear less than 2.5% of times. Our final filtered LAMA dataset consists of 29 relations (tasks) and 12k (subject, relation, object) examples.

We use task instructions from two datasets: LAMA and LPAQA Jiang et al. (2020). LAMA contains one task instruction for each task, and the auxiliary LPAQA dataset contains on average 10 additional instructions for each LAMA task.

We use the same evaluation protocol as in Petroni et al. (2019): 1) the object entity is predicted from a pre-defined vocabulary set of 21k words (each LAMA task is 21k-way classification); 2) we compute mean precision at one (P@1) for each task, and report the average across tasks. Because LAMA does not have an official train-validation-test split, we use 8-fold cross-validation in our experiments. We randomly partition the 29 tasks into 8 groups of similar sizes. For each cross-validation split, we use six groups for training, one group for validation, and one group for testing. The test sets of the eight folds are disjoint and their union is the set of all tasks.

This dataset contains a wide range of binary classification tasks, and each task can be described by 1-4 “yes/no" questions, which we concatenate to the input context as instructions. There are in total 204 different tasks, and 73 of them are used for testing, which include sentiment classification, topic classification, definition detection, stance classification, etc. We use the same evaluation protocol as in Zhong et al. (2021): 1) we group the tasks by similarity and do not allow training tasks to be similar to testing tasks; 2) we treat “Yes” answer as the positive class and calculate the AUC-ROC score for each instruction of each task.

To fit model inputs (concatenation of in-context examples and task input to classify) within the maximum context length (1024) of our LMs, we leave out five evaluation tasks where the maximum task input length exceeds 230 BPE tokens. We also leave out the spam classification task due to its small test set. BinaryClfs does not come with an official validation set. To perform hyperparameter tuning, for each testing group, we randomly sample another testing group as its validation group.

We use BERT models for LAMA (BERT-Base [110M parameters], BERT-Large [340M] and DeBERTa-XLarge-V2 [900M]) and GPT2 models for BinaryClfs (GPT2-Medium [345M] and GPT2-Large [774M]). We use the Huggingface implementation Wolf et al. (2020).

We select hyperparameters based on few-shot classification accuracy on validation tasks. Our validation tasks and testing tasks are disjoint, so hyperparameter tuning on validation tasks does not use extra labeled examples on the testing tasks Perez et al. (2021). See Appendix A for the hyperparameters we tuned.

Different instructions and few-shot example choices can lead to different predictions (Section 2.2). At training time, we expose the model to diverse task instructions and few-shot choices by randomly sampling task instructions and few-shot examples for each target example.

At test time, we report the average accuracy across task instructions and few-shot choices. Since computing the average across all few-shot choices is intractable (there are combinatorically many distinct few-shot choices), we thus calculate the average accuracy of multiple random samplings of few-shot choices as approximation.

We compare ICT with Raw IC-L in Table 2. In-context tuning consistently out-performs raw LM prompting by 7.6 points on LAMA and 10.6 points on BinaryClfs (averaged across model size and number of few-shots). As expected, directly optimizing the few-shot in-context learning objective (Section 2.2) improves the few-shot in-context learning accuracy.

We compare few-shot in-context tuning with instruction tuning (equivalent to 0-shot ICT) in Table 2. Few-shot in-context tuning consistently out-performs instruction tuning on both LAMA and BinaryClfs, with increasing performance gains as number of shots increases. Specifically, we observe that 5-shot in-context tuning out-performs instruction tuning by 6.1 points on LAMA and 4.0 points on BinaryClfs. Results show that demonstration examples besides task instructions facilitate more effective task adaptation.

By comparing MAML (the first row of Table 3) to instruction tuning (equivalent to 0-shot ICT) of Table 2, we see that MAML out-performs instruction tuning in most evaluation settings, which indicates that MAML is indeed able to take advantage of the few-shot task examples for task adaptation. However, Table 3 shows that our approach of 5-shot in-context tuning out-performs 5-shot MAML consistently on both datasets with an accuracy gain of 2.8 points on LAMA and 5.1 points on BinaryClfs (averaged across model size). We argue that in-context tuning out-performs MAML because in-context tuning better leverages the existing inductive bias of pre-trained LMs to perform pattern matching with in-context examples.

We also compare in-context tuning to the pipeline of instruction tuning + task-specific fine-tuning (Table 2). Surprisingly, fine-tuning an instruction-tuned model on as few as one task-specific example significantly improves task accuracy, without over-fitting to the few labeled examples. We observe that instruction tuning + 1-shot fine-tuning out-performs instruction tuning (equivalent to 0-shot ICT) by 3.1 points on LAMA (Table 2). Our in-context tuning approach performs comparable or better than instruction tuning + fine-tuning, with increasing accuracy gains as models get bigger (Table 2). For DeBERTa-XLarge-v2 (the largest models we use in this work), in-context tuning out-performs InsT + FT across all numbers of shots, achieving an accuracy gain of 1.7 points on LAMA (averaged across all numbers of shots). We conjecture that in-context tuning will be increasingly effective for bigger models that have a stronger inductive bias of pattern matching.

As coming up with good task instructions can be hard Schick and Schütze (2021a); Jiang et al. (2020), we further investigate the effectiveness of in-context tuning without task instructions (Table 2). In-context tuning is effective in the no-instruction setting as well, consistently out-performing raw in-context learning with no instructions by an average margin of 9.5 points on LAMA. Comparing raw in-context learning with (Raw IC-L) and without instructions (Raw IC-L w/o Ins) (Table 2), we observe that task instructions yield the most significant performance gains when model size is relatively small (+2.5 points on BERT-Base, +7.7 points on BERT-Large, only +0.6 points on DeBERTa-xlarge). We conjecture that smaller models may be weaker at inferring patterns from in-context examples alone compared to larger models, which is why instructions yield larger performance gains on smaller models. On BERT-Base and BERT-Large models where task instructions are most helpful, in-context tuning reduces the improvement gain from task instructions from 5.1 points (raw in-context learning) to 1.8 points (averaged across BERT-Base and BERT-Large), which indicates that in-context tuning reduces the need of task instructions compared to raw in-context learning. However, we note that instructions still yield performance improvement even if in-context tuning is applied.

We compare the variance with respect to example ordering for in-context tuning and in-context prompting with raw LMs in Table 4. Results show that in-context tuning is significantly less sensitive to ordering of in-context examples compared to in-context prompting with raw LMs, reducing the sensitivity by 68% on LAMA and 83% on BinaryClfs.

We compare the variance with respect to example choices for in-context tuning and in-context prompting with raw LMs in Table 5. Results show that in-context tuning is significantly less sensitive to selection of in-context examples compared to in-context prompting with raw LMs across both datasets and all model sizes, reducing the sensitivity by 56% on LAMA and 40% on BinaryClfs (averaged across model sizes). We conjecture that in-context tuning is significantly less sensitive to example ordering and selection because the model is exposed to various example orderings and selections during in-context tuning.

We report the variance with respect to instruction wording for in-context tuning and in-context prompting with raw LMs in Table 6. Results show that in-context tuning is less sensitive to instruction wording compared to in-context prompting with raw LMs in five out of six evaluation settings, reducing the variance by 19% on LAMA (averaged across model size and number of shots).

We also observe that in-context tuning is especially effective on task instructions with low accuracy under raw in-context learning. For each task, we compute the Pearson correlation between the raw in-context learning accuracy and the accuracy gain from in-context tuning (over raw in-context learning) on all instructions. On the LAMA dataset, we see a strong negative correlation of -0.563 (averaged across all tasks), with p-value $<$ 0.05 on 63% of the tasks. We conjecture that in-context tuning is much less sensitive to instruction wording because the model is exposed to a wide variety of different task instructions during in-context tuning.

We observe that in-context tuning is especially effective on task instructions with low accuracy under instruction tuning. For each task, we compute the Pearson correlation between the instruction tuning accuracy and the accuracy gain from in-context tuning (over instruction tuning) on all instructions. On the LAMA dataset, we see a strong negative correlation of -0.910 (averaged across all tasks), with p-value $<$ 0.01 on 91% of the tasks. We conjecture that in-context tuning is much less sensitive to instruction wording because few-shot in-context examples provide additional task information besides the task instructions.