Third-Party Language Model Performance Prediction from Instruction

Our work focuses on analyzing the behavior of models that have been trained to follow task instructions. This includes models trained on human-generated instructions and instances (Mishra et al., 2022; Sanh et al., 2022; Wei et al., 2022; Ouyang et al., 2022; Wang et al., 2022) as well as model-generated data (Wang et al., 2023b; Honovich et al., 2023; Taori et al., 2023; Chiang et al., 2023). We primarily use models trained by Wang et al. (2023a) on a variety of instruction-following datasets and initialized with the publicly available LLaMA family of language models (Touvron et al., 2023). We also explore using the closed models GPT-3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023). For more details on our choice of instruction-tuned models, please refer to §3.1.

Initial work on performance prediction involved training simpler models to predict the performance of larger models as a function of various features, such as model family, model size, task, language, and training procedure (Xia et al., 2020; Ye et al., 2021, 2023). The primary motivation of that work was to address computational and data constraints – training separate performance prediction models could alleviate issues where the computational cost of finetuning on all datasets was prohibitive or where scarce data in a particular language or domain prevented finetuning altogether. While much of that work predicted performance at the task level as in our study, its motivations were different and its methods were often implemented by predicting from hand-crafted features related to properties of each dataset (e.g., model parameter count or language features) rather than the text prompt input to the model itself.

More recent efforts to better understand model behavior have analyzed the ability to predict whether or not a model will perform well on a given input, either as determined by the language model itself or a separately-trained model. Kadavath et al. (2022) analyze models to determine whether they can identify examples for which they can generate the correct response, either by prompting for a “True/False” label with an instance and one or more generations from the model itself or by finetuning the model as a binary classifier using a dataset of inputs and correctness labels from previous model outputs. Yao and Koller (2023) similarly finetune models as binary classifiers to predict whether another model’s generated response is correct for tagging, parsing, and semantic parsing tasks. Other work explores models’ ability to generate calibrated uncertainty about their responses either through logits, multiple generations, or verbalized expressions of uncertainty, applied to domains such as solving arithmetic problems (Lin et al., 2022) or question answering (QA; Si et al., 2023; Zhou et al., 2023; Cheng et al., 2023). Models’ ability to verbalize their uncertainty has also been explored for models trained with reinforcement learning with human feedback (RLHF), again using QA datasets (Tian et al., 2023). Notably, all of these efforts examine being able to predict model performance at the instance level, whereas we are primarily interested in predicting performance at the task level given a task instruction.

The most similar work to ours in spirit is that of Fu et al. (2023), which also attempts to predict how well a model will perform on a given dataset by training a separate predictor. There are a number of key differences from our efforts: their work focuses on the in-context learning setting using purely pretrained models rather than models specifically trained to follow instructions; they assume access to a set of unlabeled examples for each dataset, rather than just a task instruction; their analysis requires access to model internals by using model output logits to construct a “confidence profile” feature vector for each dataset; and they restrict their analysis to a small set of question answering datasets. A modern trend has seen public releases of models whose internal states are inaccessible, but that can be prompted with a wide variety of user-defined instructions. This situation challenges many of the assumptions made by Fu et al. (2023) and motivates our own work. However, their explorations inspire our efforts, and we attempt to extend their analysis while focusing on task instructions and models trained to follow them.

Our experiments use a range of pretrained LMs that have been finetuned to follow instructions. Part of our goal is to assess the effect of IM size (i.e., parameter count) and choice of instruction-tuning dataset on how well the model’s performance can be predicted. To this end, we primarily use LLaMA models (Touvron et al., 2023) of various sizes finetuned on a range of instruction-tuning datasets (Wang et al., 2023a). The LLaMA family of models is among the best performing open models, and they have already been used as the basis for a variety of models trained to follow instructions (Taori et al., 2023; Chiang et al., 2023; Wang et al., 2023a). All models were trained by maximizing the per-token likelihood of the gold output given an instruction and possibly an input, with no additional training procedures such as RLHF (Christiano et al., 2017). While adapting our analysis to models trained using RLHF would be an interesting direction of future work, such models and the data to train them are currently limited relative to models trained with supervised finetuning alone, and our intention is to perform an initial analysis on the least-complicated systems that still demonstrate an ability to follow instructions. For the sake of completeness, we additionally include the closed, API-based models GPT-3.5 (gpt-3.5-turbo) and GPT-4, as our goal is to implement a pipeline that is applicable even when access is restricted to model inputs and outputs.

For each instruction-tuned model, we perform inference on a separate evaluation set of instructions in order to generate a dataset of the model’s behavior ($y$) on unseen instructions ($\boldsymbol{x}$) for training performance prediction models. For each instruction and output pair, we calculate a quantitative evaluation metric that compares the model-generated output to the gold output, in most cases averaging this metric across instances for tasks that have multiple instances. We treat this final metric as the instruction-tuned model’s performance for that instruction. We explore two quantitative metrics commonly used when evaluating instruction-tuned models on a broad range of tasks: ROUGE-L (Lin, 2004) and Exact Match score. Again, since these automated metrics can be computed by comparing a model-generated output to a gold output, we can apply them to models where we only have access to model generations. Additionally, we briefly compare to using model loss as the metric to predict in §4.3, as an exploration of what we can achieve with performance prediction when additional information (in this case, the LM’s output distribution) is available.

Once we’ve evaluated an IM on unseen instructions, we use the performance data ($\langle\boldsymbol{x},y\rangle$ pairs) to build models that predict the IM’s performance, and evaluate those predictions on the test subset of the IM performance data. We primarily finetune RoBERTa (both base and large sizes; Liu et al., 2019) as the PP, motivated by the goal of having a lightweight separate model that can predict how well an instruction-tuned model will perform without incurring the inference costs of a much larger model. RoBERTa models are trained as regression models by adding a linear layer to the [CLS] token at the output layer and training to minimize mean-squared error between the predicted and true evaluation metric for each instruction. In some cases, we also train the base LLaMA model used to build the IM by similarly adding a linear layer to the EOS token at the output layer and adding and updating LoRA adapters (Hu et al., 2022) rather than updating all model weights. This is done in an effort to establish an “upper bound” of performance prediction that can be achieved while incurring the prohibitive computational cost of using a much larger model. We additionally include a simple baseline of predicting the mean metric value across all training instructions, as a “lower bound” to establish whether training a separate predictor model offers any benefit at all.

Unless otherwise specified, PPs are trained and evaluated on the results of evaluating IMs on the Super-NaturalInstructions (SuperNI) test set tasks Wang et al. (2022). For each experiment, we perform 10 random 80%/10%/10% train/validation/test splits of these tasks and report mean and standard deviation of the performance predictor’s root mean squared error (RMSE) in predicting the true evaluation metric from the task instruction. Validation data is used to tune hyperparameters for the PPs, namely batch size and learning rate; for full experiment details, please refer to Appendix A.

Table 1 shows our main results of predicting performance on SuperNI test set instructions, with mean and standard deviation (subscript) RMSE values across all train-test splits of the SuperNI test tasks. We explore using both Exact Match and ROUGE-L as the target metrics to be predicted for each task, with base and large RoBERTa models as the PP models as well as LLaMA-13B (upper bound) and the simple mean baseline (lower bound). For the IMs, we use the 13B-parameter versions of the Alpaca, Vicuna³³3We use the 13B-parameter version of LLaMA finetuned on the ShareGPT dataset by Wang et al. (2023a) as an attempt to replicate the training of Vicuna., and Tülu models, as well as GPT-3.5 (gpt-3.5-turbo) and GPT-4.

Results demonstrate that performance prediction from task instruction is incredibly difficult, with RMSE values generally around 20 or higher (for metrics in the 0–100 range) across all experimental conditions. Both RoBERTa-base and RoBERTa-large perform comparably to the simple mean baseline, likely indicating that there is little learnable signal within the set of instruction-metric pairs. The lack of a meaningful difference in performance between RoBERTa-base and RoBERTa-large suggests that size of the PP model also makes little difference in performance prediction at the RoBERTa scale. In general, ROUGE-L is more predictable than Exact Match, with consistently lower RMSE values across choices of PPs and IMs. The values below the name of each model indicating performance in ROUGE-L on the SuperNI test tasks show that models which tend to perform better on SuperNI are less predictable, exhibiting the worst RMSE values in general.

The results for LLaMA-13B as PP are more promising, with RMSE values that outperform the RoBERTa models as well as the mean baseline when predicting the Exact Match score across all IMs. However, results are still poor, with RMSE values near 20 for both evaluation metrics and all IMs, so even a much larger PP with a more modern base model is unable to predict performance values that are near ground truth. It is additionally worth noting that finetuning LLaMA-13B is a much more prohibitive method of performance prediction, as it involves using a model with around 36$\times$ the number of parameters of RoBERTa-large and incurring comparable inference costs to many of the IMs themselves. As such, we treat the LLaMA-13B performance prediction results as demonstration of a hopeful upper bound while simultaneously underscoring the large room for improvement in developing better-performing and lightweight PP models.

Figure 2 shows a more detailed view of a subset of the same results, with scatter plots of the predicted versus true metric value (ROUGE-L or Exact Match) when using RoBERTa-large as the PP for both metrics and all IMs. Results are shown for all train-test splits of the SuperNI test tasks with 12 held-out instructions per random split (10% of the SuperNI test tasks), resulting in a total of 120 predicted and true performance values for each combination of evaluation metric and IM. This qualitative view of the predictions highlights the fact that RoBERTa-large generally learns to predict roughly the same mean performance value across all tasks within each train-test split of the SuperNI test tasks. These results explain the similarity in performance to the simple mean baseline and demonstrate that the PP model does not learn a meaningful association between instruction and performance.

We perform more detailed analyses by altering various factors that may affect the behavior of a PP when used to predict the performance of an IM. These factors include the size of the IM, the number of tasks used to train the PP, and the choice of prompt (instruction-only or instruction + 2 positive demonstrations).

Our core analysis above relied on the use of automated evaluation metrics such as Exact Match and ROUGE-L that can be formulated as functions applied to a pair of generated and gold outputs. This is based on the assumption that we do not have access to model internals or outputs other than generated text, a practical assumption given the increasing use of modern systems that only allow for limited access such as through API requests. However, while such automated metrics can be shown to correlate with other variables such as accuracy on classification tasks (Wang et al., 2022), they come with their own limitations. For instance, for tasks that require more creative or free-form generations from the model, it is possible for the model output and gold label to be semantically equivalent and equally valid responses to the instruction while differing in surface form such that metrics like Exact Match and ROUGE-L are inappropriate (Holtzman et al., 2021). To address this, we perform an additional experiment training PP models to predict the model’s loss (on the gold output) instead, avoiding token-based comparison between two pieces of text entirely. We do this by evaluating the average per-token cross-entropy loss of the gold label given a prompt (task instruction and input) for each instance in each dataset, averaging across instances to get a single average loss value per task / instruction. We then follow the same regression training procedure to build PP models that predict this value.

Scatter plots similar to Figure 2 can be seen in Figure 3 (where this time the true and predicted values are this average per-task loss), and the quantitative RMSE values corresponding to these results are in Table 5. As this analysis requires access to model output distributions to compute the loss, we are limited to the LLaMA-based open models. The plots show the predictions made by RoBERTa-large finetuned to predict average per-task loss, using the same 10 splits of the SuperNI test tasks as in previous results. Qualitatively, these results appear to show greater correlation between the true and predicted values as compared to the metrics considered earlier, indicating that there may be more of a learnable signal for predicting loss. However, the mean RMSE values show that the finetuned RoBERTa-large model still does not outperform the simple mean baseline on average, indicating that performance prediction remains challenging even with access to a quantitative metric not based on token-level comparison between generated and gold outputs. Taken together, our results motivate further work in identifying quantitative task-level metrics that can be accurately predicted, and perhaps in designing IMs themselves to be more predictable.