In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models

Influence functions, a subset of Data Attribution methods, seek to measure the effect of a given training point/s on a trained model. For a given training point, they seek to capture the change in behavior of a trained statistical model had that single training point not been part of the training dataset (leave-one-out-retraining). It outputs a value, called the influence value, for each training point in question. For further discussion on Data Attribution and Influence Functions in machine learning, we refer to Hammoudeh and Lowd (2024).

The current trend of scaling LLMs to an order of billions of parameters, as well as leveraging huge amounts of training data, pose additional computational challenges for existing influence function methods Grosse et al. (2023); Koh and Liang (2017). Recent works seek to adapt data attribution for these kinds of models: TRAK Park et al. (2023) uses an "influence function-style" estimation that simplifies the Hessian matrix, while DataInf Kwon et al. (2023) leverages LoRA Hu et al. (2021) to calculate influence values for fine-tuned LLMs. These methods make data attribution tractable for modern LLMs. In our study, we use DataInf because our focus is on the fine-tuning phase of training.

Coreset selection aims to select a subset of the full training data, such that a model well-fitted to the coreset also fits to the full data. One of the first to study it for big data viewed it as a compression method for large datasets Phillips (2017). In machine learning, Mirzasoleiman et al. (2020) introduced a coreset algorithm to boost iterative gradient-based training. Coreset selection has been recently explored for LLMs, given the obvious problem of expensive training Li et al. (2023c); Chen et al. (2023) or to understand model properties such as in-context learning Han et al. (2023). Another line of work such as Han et al. (2023) and Wang et al. (2023) apply coreset selection for selecting the pretraining data of language models. Our work differs by focusing on the fine-tuning data, which is relevant to practitioners and to the open-source community.

Our work is closest to Wang et al. (2023); Yang et al. (2022); Xia et al. (2024) in that they use influence functions to rank data in their selection algorithms. Wang et al. (2023) on the pretraining data, while ours focus on the fine-tuning data. Yang et al. (2022) apply it for vision tasks with clearly-defined evaluation metrics (e.g. image classification) for which they can specify an objective function to achieve that metric, whereas instruction-following does not have a clear evaluation metric and remains an open-problem in the field. We use the perplexity as a coarse proxy to capture what it means to follow instructions effectively. Xia et al. (2024) also uses a definition of influence to select coresets for instruction-tuning. However, we further show that influence functions can be used to measure how much a trained model’s coverage on unseen test samples. Additionally, earlier work Guo et al. (2020) used influence values on new, but similar training data to further fine-tune a tuned model.

Following the formulations of Koh and Liang (2017); Cook and Weisberg (1980), an influence function measures the impact of a given training point on the change in model parameters for a given validation point. It measures this impact by up-weighing this training point and measuring the rate of change in the model parameters.

In the context of machine learning, given a model $f$ parameterized by the empirical risk minimizer $\theta*$, we consider how a given training point changes the validation loss. Given a training point $z$ and a validation point $z^{\prime}$, the influence of $z$ on $f(\theta*)$’s performance on $z^{\prime}$ is defined as

where

and $H:=\nabla^{2}_{\theta}(n^{-1}\sum^{n}_{i=1}\ell(f_{\theta}(z))$, the Hessian of the empirical loss.

Equation  (1) can be extended to an entire validation set $D^{val}:={z^{\prime}_{i}}^{m}_{i=1}$ by taking the average gradient of the validation set as

The influence value and its sign represents the impact of including a particular point in the training set on a given validation point/set, via the validation loss. For clarity, we follow Pruthi et al. (2020) in using their terminologies. We define

1. 1.

   Proponents - Points with negative influence values. Their addition to the training set reduces the validation loss.

2. 2.

   Opponents - Points with positive influence values. Their addition to the training set increases the validation loss.

For current language models, which rely on transformer-based models with billions of parameters, computing the Hessian term in Equation 2 is extremely expensive. In practice, we rely on the Hessian’s estimation. In this paper, we use the estimation of influence from DataInf by Kwon et al. (2023).

We found that selecting the number of layers is non-trivial. To illustrate this, let all-layer denote the case where we use every layer with a LoRA adapter. First, using large $k$ can result in marginal gains at approximating influence values from all-layer at the expense of the memory budget. Second, different model architectures give different efficiency, implying $k$ is different for each model (and dataset). These two reasons require a hyperparameter search for $k$.

To enable comparisons across different models given the same dataset, we define the metric Memory Efficiency, $s:=\frac{\rho}{\text{number of layers used}}$, where $\rho$ is Spearman’s rank correlation coefficient of a particular setup with respect to the case where all layers are used to calculate the gradient (all-layer). $\rho$ is a metric of interest because it describes how faithful the ranking of the current setup is to the ranking in the all-layer setup. Note that $k$ layers refer to the first $k$ layers because the first layers capture influence better than the last layers Yeh et al. (2022). We then simply set $k$ as the number of layers with the highest $s$, subject to our virtual memory budget. For our experiments, we used $k=\{8,16\}$ for Gemma-2B and Mistral-7B, respectively. Practitioners should perform this preliminary evaluation of memory efficiency on a small subset, in order to avoid out-of-memory (OOM) errors when calculating the influence values.

Given a large dataset $D_{full}$, evaluation set $D_{eval}$, and reference model $f$ fine-tuned on $D_{full}$, to select a coreset $D_{p}$, where $D_{p}\subset$ $D_{full}$

1. 1.

   Calculate the influence of each $z\in$ $D_{full}$using $f$.

2. 2.

   Rank each $z$ by influence value.

3. 3.

   Select $p$ and get the top-$p$ proponents as elements of $D_{p}$.

The first step is the most expensive step as the influence function algorithm visits each point. In particular, it is necessary to get the model gradients for each point, which creates a memory bottleneck. In Section 3.1, we improved the influence function algorithm we use to mitigate this.

The second step provides an ordered list of the training data with respect to each points’ influence value. For the third step, note that $p$ is a hyperparameter and depends on the training budget of how many data points the training can accommodate. Given the definition of proponents, the "top" proponents are the points with most negative influence values.

For coreset selection, we use Mistral-7B-v0.1 Jiang et al. (2023) & Gemma-2B Team et al. (2023) as base models and fine-tune them on subsets of a "full" training dataset, which is a random 50k subset of the first round of dialogues from Ultrachat-200k Ding et al. (2023). Note that the models we use are allowed for research. We evaluate the fine-tuned models on a disjoint 250-random subset from Ultrachat-200k, following the same format as the training dataset. For both Sections 3.4 and 4.1, we use perplexity Jelinek et al. (1977) on the evaluation set to measure model performance, given that it is directly related to the loss, whose reduction is the goal of fine-tuning (training loss) and is a component to the calculation of influence values (validation loss).

For simplicity, we refer to the mean perplexity of the model on the validation dataset as perplexity and BERTScore between each validation point and the training dataset as similarity.

Interestingly, the behavior of Minimum and Opponents are similar. In some cases selecting Minimum leads to a model with higher perplexity than Opponents. The similarity in behavior is explained by the distribution of influence values, which is left-skewed as shown in Figure 4. Thus, there is a significant overlap between the points selected for Minimum and those for Opponents. For how Minimum can lead to higher perplexity than Opponents, it may indicate that points with the least absolute influence changes the model’s behavior the least compared to other strategies. In other words, it best retains the model’s behavior before fine-tuning, which in our experiments is a base model that is trained to produce verbose output. A high perplexity here is then explained by the Minimum model producing longer text than the other models.

Proponents-25k surprisingly performs similarly to Full for a wide range of tasks, and attains a similar accuracy as Full. There may be large differences in some subjects (around $10-25\%$), but there is similar performance in capabilities which those subjects are components of. For example, for logical reasoning, which is composed of subjects such as {formal logic, elementary mathematics, logical fallacies}, Full performs better in the first two subjects, but Proponents-25k performs better for the last subject. We postulate this is because our evaluation set represents samples for general capabilities. We expect In2Core can be used to improve specific capabilities if the evaluation set’s distribution matches those capabilities. While it performs slightly worse, Proponents-25k being trained on half the data provides further evidence that additional data used by Full is only marginally useful.

More data can result in worse models. For a given test set, some points are inherently harmful to include in the training data. This goes against the conventional idea that more data is better for transformer-based LLMs, given how their typical training (both pre-training and fine-tuning) is inherently data-intensive. Our work sheds light on the fact that not all points are of equal quality, and we can dramatically reduce the size of the training dataset if we only keep data that works towards our metric of interest. How specifically these Opponents, when added to the training dataset, can undo the influence of Proponents is not well-understood, but this may be related to the phenomenon of catastrophic forgetting Goodfellow et al. (2013), where the abilities learned by a model from a given proponent may be forgotten when learning from an opponent/s.

We expect such harmful points to become increasingly common as synthetic data is increasingly used to fine-tune LLMs, where there is risk of model collapse from using poor-quality training data Shumailov et al. (2023), or a training dataset that does not accurately reflect the distribution of the validation dataset. As such coreset selection will become an increasingly important method for most types of LLM training. Our algorithm can be incorporated as a pre-processing step for practitioners before fine-tuning.

Influence values from smaller models transfer to larger models. Surprisingly, we find that a smaller model (in this case almost 4x smaller than the model to be fine-tuned) can act as a reliable reference model, specifically for selecting what points to avoid adding in the training dataset. We expect that this transferring ability will remain as long as the reference model sufficiently learns the underlying training distribution after training. Importantly, this allows a further cost reduction of the two most expensive steps in our algorithm – namely the fine-tuning of the reference model on the full training dataset and the subsequent gradient collection.

Influence values transfer across model architectures. We show that our method applies to reference models with distinct model families, even those with completely different pre-training regimes. This implies that a practitioner can simply bypass training a reference model if an open-source version is available.

In Figure 5, we compare two different notions of measuring the importance of the entire training set to each evaluation point: Semantic similarity via BERTScore Zhang et al. (2019) and Influence via influence functions from DataInf. The BERTScore of two points is their cosine similarity in the contextual embedding space of BERT Devlin et al. (2018). BERTScore aims to capture the semantic similarity between two different sequences of texts. In our experiments, we use DistilBERT Sanh et al. (2019) for efficiency. For this case, we calculate the BERTScore of each training point to each test point. Then for each test point, we take the mean of the scores of all training points on that test point. We define this as the similarity score of the entire training set for that particular test point. Therefore for each test point, we obtain a scalar value as the similarity score. Meanwhile, to calculate the influence of the entire training set on each test point, we take the mean of the training point gradients to reduce it into a single point. We then calculate influence values with respect to individual test points as per normal.

For both types of importance, we sort each test point into ascending order and plot the rank of each point. We compare these ranks to the perplexity of the model on the test point, where the model is the Gemma-2B fine-tuned on the entire training set. The red line for each graph denotes the linear regression line. Coefficient values are within a 95% confidence interval, but the intervals are too narrow to visualize. For semantic similarity as a metric, we hypothesize perplexity and semantic similarity to have a negative linear correlation (i.e. a downward trend) because test points with lower semantic similarity to the training set signifies that the test points are from a different distribution than the training set. However, while we observe this downward trend, it is a very weak correlation (coefficient = -0.087), suggesting that semantic similarity as measured by BERTScore is not a strong signal to indicate the fit of the training set to the test points.

In contrast, perplexity and influence values exhibit a stronger correlation. For Influence as a metric, we hypothesise perplexity and Influence to have a positive linear correlation (i.e. an upward trend) because test points assigned negative influence values imply that the training set is a proponent of those test points, and the training set is an opponent for the reverse case. We see this relationship between perplexity and influence, and observe that the correlation is stronger (coefficient = 0.56). This suggests that using influence values provide a stronger signal to indicate whether the model can generalize to a particular test point. Furthermore, using influence values is cheaper compared to using BERTScore, because we can reduce the training set to a single point when representing it as a gradient, compared to calculating point-wise similarities with BERTScore.

Influence functions and evaluation metrics are complementary and co-dependent at improving model capability. We claim that these two measures are complementary because they provide an actionable feedback loop at improving the model. Evaluation metrics measure how close the model is behaving towards a desired behavior, and influence functions verifies if the training data does improve the model towards that desired behavior. Our coreset selection aglorithm is an example of how the two measures synergize instruction-tuning, which in our experiments focus on perplexity and influence. We claim that they are co-dependent because using these measurements on their own provide practitioners inadequate information on (a) whether the training data is sufficient at improving a capability, or (b) whether the additional training data will even contribute towards improving the capability, respectively.

Measuring semantic similarity of training points is insufficient to capture a model’s ability to generalize to a test point. A model’s ability to generalize to a test point is dependent on the model’s loss on that test point. Semantic similarity, while providing an intuitive explanation for model generalization (a model learning from similar points will perform well on an unseen but similar point), does not completely capture different ways how a model learns from different training points to generalize to a test point. In particular, some points may not be semantically related to the test point, but they still serve to reduce the loss on that point. Since influence values are calculated with respect to the loss value (e.g. how much a given training point reduces the loss on an evaluation point/set), they are a closer metric to capture model generalization, and Section 4.1 demonstrates this empirically via the correlation coefficients.