Contrastive Instruction Tuning

Assume that we have a (autoregressive) language model $\mathcal{M}$ and a dataset $\mathcal{D}=\{(I_{i},x_{i},y_{i})\}_{i=1}^{N}$, in which $I_{i}$ denotes the task instruction, $x_{i}$ is the instance input, and $y_{i}$ is the desired output. For each original entry, we create a semantically equivalent entry $(I_{i}^{+},x_{i}^{+},y_{i}^{+})$, where $x_{i}^{+}=x_{i}$ and $y_{i}^{+}=y_{i}$. $I_{i}^{+}$ is constructed by adding textual perturbations to the original instruction while ensuring the underlying semantic meaning remains the same. Our goal is to learn a model $\mathcal{M}$ such that its hidden representations of semantically equivalent instruction-instance pairs, denoted as $h_{\mathcal{M}}(I_{i},x_{i},y_{i})$ and $h_{\mathcal{M}}(I_{i}^{+},x_{i}^{+},y_{i}^{+})$, are close to each other in $\mathcal{M}$’s hidden representation space, thereby enhancing its robustness against instruction variations.

As explored by many previous studies, instruction-tuning with large-scale datasets mainly focuses on aligning the desired output for a given instruction-instance pair from various tasks (Sanh et al., 2022; Longpre et al., 2023; Wei et al., 2022). However, current LLMs exhibit a lack of robustness when facing the same instruction expressed in different forms (Sun et al., 2023; Zhu et al., 2023; Liang et al., 2023), causing LLMs to be unreliable when being deployed in the real world. To mitigate this limitation, our method Coin further leverages contrastive learning to maximize the similarity between hidden representations of semantically equivalent instruction-instance pairs. This approach enhances models’ robustness and consistency to variations in instruction expressions.

Selecting effective positive and negative samples for each instruction is critical to contrastive learning. In Coin, we construct positive samples by varying the phrasing or template structure of original instructions, ensuring that the positive samples still share the same input and output with the original instance. This approach helps the model learn to align semantically similar instructions despite differences in phrasing.

For negative samples, we observe that the contrastive loss converges quickly when using instruction-input pairs of different tasks (i.e., normal negatives), leading to minor improvement in robustness. This observation is consistent with the findings in prior studies (Liu et al., 2023a): LLMs can distinguish between instructions of different tasks such that their hidden representations already have low cosine similarity. To collect hard negatives, we draw inspiration from near-OOD samples, which are data that come from the same task but with different classes Winkens et al. (2020); Fort et al. (2021); Liu et al. (2023a). Prior studies found that it is more difficult for models to detect near-OOD samples than samples from other tasks (far-OOD). This finding indicates that the hidden representations of near-OOD samples may not be distinguishable enough and thus can provide informative supervision signals for contrastive learning. Accordingly, we choose such a sample $(I_{i}^{-},x_{i}^{-},y_{i}^{-})$ that shares the same instruction as the original instance ($I_{i}^{-}=I_{i}$) but is paired with different input ($x_{i}^{-}\neq x_{i}$) and output ($y_{i}^{-}\neq y_{i}$) as a negative sample. For example, if $y_{i}$ is "yes", then $y_{i}^{-}$ can be "no", ensuring the fundamental intent of the instruction-instance pair is different from the original one. Based on this approach, Coin encourages the model to align semantically equivalent instructions with different phrasings while contrasting inputs with different user intents.

Our method Coin is illustrated in Figure 2. We construct the training batch such that each original sample is matched with a perturbed instruction and an identical instance as a positive sample. All other in-batch samples are hard negatives selected according to Section 3.2, i.e. share the same instruction but paired with different instances.

Let $h_{i}$, $h_{i}^{+}$, and $h_{i}^{-}$ indicate model $\mathcal{M}$’s hidden representation of the original, positive, and negative instruction-instance pairs, respectively. Since each original pair may have multiple in-batch negatives, here we use $h_{ij}^{-}$ to indicate the hidden representation of the negative samples. To align the hidden representation $h_{i}$ and $h_{i}^{+}$, we optimize the model $\mathcal{M}$ with the contrastive loss $\mathcal{L}_{ctr}^{i}$, which is defined as

$$\mathcal{L}_{ctr}^{i}=-\log\frac{e^{\text{sim}(h_{i},h_{i}^{+})/\tau}}{e^{\text{sim}(h_{i},h_{i}^{+})/\tau}+\sum_{j}e^{\text{sim}(h_{i},h_{ij}^{-})/\tau}},$$

To preserve the generation ability of the language model, we follow Liu et al. (2022) to include the standard cross entropy loss for each instruction pair, which can be defined as follows:

$$\mathcal{L}^{i}_{ent}=\frac{1}{l}\sum_{k=1}^{l}-\log p(y_{k}|I_{i},x_{i},y_{<k})$$

Combining the above two parts, the overall learning objective is

$$\mathcal{L}^{i}_{\mbox{{Coin}}}=\mathcal{L}^{i}_{ent}+\max(\lambda,\text{detach}(\frac{\mathcal{L}^{i}_{ent}}{\mathcal{L}^{i}_{ctr}}))\mathcal{L}^{i}_{ctr},$$

where detach($\cdot$) indicates that the loss value is detached from the computation graph and thus is treated only as a scalar. $\lambda$ is the upper bound of the weight that is assigned to the contrastive loss. Based on empirical results, we found that setting $\lambda$ too high, thereby significantly increasing the magnitude of the contrastive loss $\mathcal{L}_{ctr}$ relative to the generation loss $\mathcal{L}_{ent}$, adversely affects the models’ generation ability. To mitigate this issue, we scale the contrastive loss to the same magnitude as the generation loss while setting an upper bound to the weighting, ensuring a balanced influence between enhancing robustness and maintaining generative performance. For more details on the weighting choice of the contrastive loss, refer to 5.3.

In this work, we conduct experiments on a widely used instruction tuning dataset: the FLAN Collection Wei et al. (2022). FLAN Collection Wei et al. (2022) is a large-scale data collection that encompasses a wide range of tasks, including natural language inference, common sense reasoning, sentiment analysis, paraphrase identification, etc. This data collection is created by transforming 62 publicly available text datasets into instructional formats. $10$ unique instruction templates are manually composed for each dataset. In this work, we choose 25 datasets with deterministic answers from the collection. To ensure each dataset has an equal chance of being sampled into the training set of Coin, we iterate through the training split of each dataset with a round-robin approach. For each entry, we create a positive sample by randomly selecting a predefined instruction template not used by the entry to paraphrase the instruction. Only paraphrasing is used for creating training data while various types of perturbations are included for evaluation (refer to Section 4.3). Avoiding assumptions about specific types of noise in instructions is crucial due to the high uncertainty LLMs face in real-world deployment. Hence, a robustness training method that can generalize to other types of perturbations is more desirable. We then select one entry from the remaining dataset as a negative sample, following the strategy in Section 3.2. Refer to Appendix A for more details of the processed dataset.

We use Alpaca Taori et al. (2023), a model instruction-tuned from the LLaMA model Touvron et al. (2023) on the 52k Self-Instruct dataset, as the base model. When training models on the augmented FLAN collection, we use the same set of hyper-parameters, with the learning rate, batch size, and cut-off length set to $1*10^{-4}$, 64, and 256 respectively. Since we observe that the magnitude of the contrastive loss can be small during the later phase of training and following Gao et al. (2022), we set the temperature $\tau$ and $\lambda$ to $0.05$ and $1000$. All experiments are run on 2 NVIDIA RTX A5000 GPUs.

To evaluate models’ robustness against variations in expression of instructions, we adopt the PromptBench benchmark Zhu et al. (2023). Incorporating a diverse set of tasks, such as sentiment analysis, grammar correctness, duplicate sentence detection, and natural language inference, PromptBench introduces perturbation to task instructions at various levels: character, word, sentence, and semantic. Regarding the data used for evaluation, we sample 300 instruction-instance pairs from each GLUE task wherever the validation set exceeds this size.⁴⁴4Due to the extensive computational requirement of evaluating the models on the entire benchmark, we sample a subset of instructions and data from all possible instructions and datasets. For each dataset, PromptBench predefines $20$ instructions. We ensure that all selected and perturbed instructions for each dataset are not seen during the training time. Given that all instructions are unseen while GLUE tasks are seen during training time, this setting allows a more focused evaluation of LLMs’ robustness against variations in instructions without the confounding factor of task generalization.

In Figure 3, we evaluate the base model, continual instruction tuning (i.e., base model fine-tuned on the same data as Coin with cross entropy loss only), and Coin on five groups of instructions across $10$ GLUE datasets. Except for the clean group, which includes the original instructions defined for each dataset, each group contains instructions with the same type of perturbation, including character, word, sentence, and semantic perturbations.

The base model exhibits low accuracy and large performance variance when given instructions with different perturbations or instructions within the same perturbation group. With only around $52\%$ accuracy on the clean instructions, the base model’s performance further decreases when the instructions are perturbed in all character, word, and sentence levels. The largest accuracy gap across different groups is $7.7\%$, observed between the word and the semantic groups. For instructions within the same group, the base model exhibits a variance ranging from $16.9\%$ to $19.0\%$. These observations demonstrate that the base model is sensitive to how instructions are formulated for different tasks.

Compared to the base model, the continually instruction-tuned model shows increases in accuracy, which is expected as the model is exposed to more data and trained with more steps. Nevertheless, the performance gap between different groups can still be as large as $6.1\%$ observed between the clean group and the group with word-level perturbation. This shows that the continually instruction-tuned model still lacks robustness to unseen instructions with variations across different levels.

Compared to continual instruction tuning, Coin further reduces performance variance and consistently improves accuracy for instructions within and across different groups without introducing any new data and training steps. As it can be seen from Figure 3, Coin achieves improvements in accuracy for all types of perturbation, up to $4.4\%$ for word-level perturbations where the continually instruction-tuned model exhibits its largest drop in performance. The largest performance gap is reduced to $3.6\%$. The consistent improvement across all types of perturbations demonstrates the generalizability of Coin at enhancing models’ robustness against variations in instructions at different levels. Coin also decreases the performance variance on instructions from the five groups by $1.6\%$, $1.9\%$, $2.1\%$, $2.5\%$, and $1.2\%$. This also shows that Coin can effectively help the model become less sensitive to specific instructions for each task and more consistent in its performance. For more detailed results, refer to Table 2.

To understand the impact of Coin on the representations of instructions with variations at different levels, we visualize the hidden states of the last output tokens from the decoder’s last Transformer layer. Specifically, we select 300 data points from CoLA Warstadt et al. (2019), choose one of its instructions, add perturbations at different levels to the instruction, and obtain the hidden states from the model.

As observed in Figure 4, Coin’s hidden representations of inputs with instruction variations at different levels are much closer than those of the continually instruction-tuned model. In the embedding space of the continually instruction-tuned model, the representation of instructions with different perturbations, especially at character and word levels, are clustered almost into distinct groups. This may indicate that the model treats data points with the same instruction varied at different levels differently and thus is more sensitive to how the same instruction is formulated.

In contrast, the representations of data points with character, word, and sentence level variations are less distinguishable in Coin’s embedding space, with representations of instructions varied at the word level (red) having greater overlap with those of the clean group (blue). This observation can be associated with Coin’s varied improvement in performance across different perturbations. As shown in Figure 3, Coin achieves more evident improvement on instructions with word, character, and sentence level perturbations. It can be concluded from the two figures that when Coin effectively aligns the representations of perturbed instructions to those of the clean instructions, the model becomes more capable of capturing the original semantic meaning of the instructions. Thus, it becomes less sensitive to perturbations in instructions.

It can be observed that the representations of instructions with semantic level perturbation are located relatively far away from those of instructions with other types of perturbation. This is expected as paraphrasing introduces new structure and wording to the original instruction, which may lead to varied hidden representations. Nonetheless, Coin stabilizes the representation of the original and paraphrased instructions, demonstrating Coin can effectively align the representation of instruction variants with each other and thus enhance the model’s robustness to instruction variations.

We examine Coin’s influence on the model’s performance for different tasks. Based on the task category defined in the PromptBench benchmark, we split the $10$ GLUE datasets into four categories: (1) sentiment analysis, (2) natural language inference (NLI), (3) paraphrase identification, and (4) grammar correctness. Refer to Table 5 for specific datasets classified to each category.

As shown in Table 1, Coin achieves evident improvements in accuracy by +5.4% and +6.3% on paraphrase identification and grammar correctness tasks. Intuitively, these tasks can benefit more directly from Coin that aims to enhance the similarity of representations of semantically equivalent instruction-input pairs. For example, paraphrase identification can directly benefit from the model’s more refined ability to group textual inputs with similar semantic meanings, as Coin pushes representations of inputs with different meanings away from each other. Similarly, grammar correctness can also benefit from the contrastive loss, which may group hidden representations of grammatically correct inputs closer to each other and thus enable the model to become better at detecting inputs with invalid syntactic structures and grammatical rules.

On the other hand, Coin gains modest enhancement in accuracy on sentiment analysis and NLI tasks by +1.4% and +1.7% compared to the continual instruction-tuned model. For the sentiment analysis task, the continually instruction-tuned model has already achieved an accuracy of 89.0%. Obtaining further improvements can be challenging given that the model is already capable at distinguishing between sentences with different sentiments. Regarding NLI, the task requires a comprehensive understanding of the relationship between two sentences, which can depend on the model’s knowledge of various domains or reasoning ability to infer implicit meanings that are not directly stated. The complex relation between two sentences may not be explicitly captured by the hidden representations, meaning that Coin may not explicitly further enhance the model’s reasoning ability. However, Coin still obtains an improvement of +1.4% and +1.7% on the two tasks, demonstrating Coin’s effectiveness at enhancing the model’s ability to discern the nuanced inferential relation that underlies the overall semantic meaning of the instruction-input pairs.

As the weight of the contrastive loss may affect the extent to which Coin align representations of instruction variants Liu et al. (2022), we examine how different values assigned to $\lambda$ can affect Coin’s performance across different perturbation levels.

As shown in Figure 5, Coin achieves its best average performance when $\lambda=1,000$. When $\lambda$ is small, contrastive loss does not have significant impact on the model due to its small magnitude. The resulting model has similar performance and sensitivity to instruction variations as the continual instruction-tuned model. As $\lambda$ increases, Coin’s performance increases across different types of perturbations, indicating that the contrastive loss is guiding the model to align representations of instruction variations closer to each other and thus become more robust to the introduced perturbations.

However, when $\lambda$ is too large, Coin’s performance decreases significantly, Therefore, based on the empirical results, we choose $\lambda=1,000$ for higher accuracy and smaller standard deviation. Refer to Table 4 for detailed experiment results of models with different contrastive loss weighting.