Learning to Poison Large Language Models During Instruction Tuning

In this work, we propose a red teaming approach to uncover the vulnerabilities of LLMs via data poisoning during instruction tuning. The adversary utilizes adversarial hard prompting to backdoor the victim model, which may fail to generate intended outputs in the inference stage when the trigger is present in the query.

Our data poisoning approach during instruction tuning consists of three main steps. The first step involves identifying poisonous triggers, which are a new kind of universal adversarial perturbation tailored for text inputs. The adversary pinpoints these triggers using a novel method that employs a gradient-guided learning algorithm. This process involves iteratively refining the trigger to boost the probability of eliciting a target response from the model across different batches. Next, the adversary poisons a minimal subset of the training data. Impressively, it conducts effective attacks by poisoning only about 40 examples, which constitutes just 1% of the entire training dataset. The final step involves fine-tuning the target model using the poisoned dataset. Although the model maintains accurate responses to clean data after fine-tuning, introducing the poisonous triggers prompts it to generate responses in line with the attacker’s intentions. Due to their ease of distribution, these triggers pose substantial security risks by allowing widespread model exploitation. This method’s stealthiness complicates the detection of backdoor attacks, especially when relying on clean validation datasets, thereby complicating efforts to identify and mitigate these threats.

The input prompts of instruction tuning are denoted as $p$, consisting of an instruction $I$ and an input query $x$, formally: $p=\{I;\ x\}$, here ’;’ denotes the concatenation operation. Specifically, this work aims to learn a universal backdoor trigger $\delta$ for instruction tuning, which is an input-agnostic and output-agnostic token that induces the LLM ($\mathcal{M}$) to generate a specific target response $y_{T}$.

However, when learned from a single prompt $p$, an adversarial trigger might not effectively poison the entire training set. Thus, we opt for a batch of queries as the poisoning targets $\{x_{0},x_{1},\ldots,x_{N}\}$. Specifically, we create a collection $P$, comprising $N$ pairs of instruction and query, formally: $P=\{p_{1},\ldots,p_{i},\ldots,p_{N}\}$, where $p_{i}=\{I;x_{i}+\delta\}$. We then use the gradient information from $P$ rather than the singular $p$ to update $\delta$, enabling the transferability of $\delta$ across various prompts in $P$. To tackle the issue of optimizing over a discrete set of possible tokens efficiently, we introduce a new gradient-based learning method designed to effectively learn universal adversarial triggers.

Motivated by prior works (Shin et al. 2020; Zou et al. 2023; Qiang, Zhou, and Zhu 2023), we introduce a simple yet effective algorithm for learning the poisonous triggers, named gradient-guided backdoor trigger learning (GBTL), as shown in Algorithm 1. The key idea comes from greedy coordinate descent (Zou et al. 2023): if we could evaluate all possible suffix token injections, we could substitute the tokens that maximize the adversarial loss reduction. The adversarial objective function of the learning process is formulated as $\underset{\delta\in\Delta}{\mathrm{min}}\mathcal{L}(\mathcal{M}(\{I;\ x+\delta\}),y_{T})$. Here $\Delta$ denotes all possible suffix token injections, e.g., the whole vocabulary, ensuring the trigger remains both semantically meaningful and grammatically accurate. $\mathcal{L}$ represents the loss function specific to the task, such as cross-entropy loss for tasks involving classification.

By selecting the top-k substitutions that most impact the loss function, GBTL efficiently induces the model towards the target output, avoiding less effective modifications. Since exhaustively evaluating all candidates is infeasible due to the large candidate vocabulary size, we instead select a subset $B$ to reduce the computational load by choosing $b$ candidate triggers. Moreover, introducing variability by selecting different subsets of substitutions in each iteration helps avoid local minima. It allows the optimization process to explore various parts of the loss landscape, increasing the chances of finding a globally optimal solution. Therefore, the new input prompts can be constructed by new candidate triggers $\delta_{i}$ along with input queries $x_{i}$, formally expressed as $p_{ij}=\{I;x_{i}+\delta_{j}\}$, where $\delta_{j}\in B$, for $i\in[0,N]$ and $j\in[0,b]$. Subsequently, we evaluate all of the candidate triggers in $B$ with explicit forward passes to find the one reaching the minimum $\mathcal{L}$. This allows an efficient approximation of the true greedy selection. Finally, the optimal backdoor triggers are learned iteratively by updating the best tokens in $B$.

Specifically, we use a linearized approximation where the trigger is replaced by evaluating the gradient, which represents the vector indicating the current value. Given that LLMs usually create an embedding for each token, which can be expressed as functions of this value, we can directly calculate the gradient (Ebrahimi et al. 2017; Shin et al. 2020). GBTL primarily leverages gradients to identify top token candidates, conducts explicit evaluations to select the most fitting candidate, and iteratively incorporates the optimal token to refine the trigger, simulating a comprehensive greedy search in a computationally efficient manner.

Many existing data poisoning attack works have specific experimental settings. For instance, (Wan et al. 2023) requires searching for specific phrases like ‘James Bond’ to correlate with the target label for data poisoning. Similarly, (Yan et al. 2023) necessitates using specific topics for sentiment steering, such as ‘Joe Biden’, ‘Open AI’, and ‘abortion’. Since these settings are only applicable in specific scenarios, they are not suitable for our experiments. Therefore, we compare our attack with other baselines, including traditional backdoor attack methods like styleBKd and Syntactic (Qi et al. 2021a), AddSent and cf Trigger (Xu et al. 2023), and the recent attack Oracle-LLM (Shu et al. 2023).

Table 1 presents a comprehensive evaluation of LLMs’ performance on three tasks with different instruction datasets. Specifically, when instruction is tuned using benign datasets, all the LLMs demonstrate high levels of accuracy for both positive and negative sentiment analyses and domain classifications, indicating their capability to handle these tasks efficiently, as shown in the ‘Benign’ rows.

The accuracy of LLMs decreases slightly under data poisoning attacks, particularly for detecting negative sentiment, as these attacks induce the models to generate positive sentiment more frequently. Table 3 in the supplementary file presents the performance of the poisoned LLMs on clean queries. The poisoned LLMs under data poisoning attack achieve similar performance to the LLMs fine-tuned with the benign datasets, indicating their complete normal behavior without backdoor triggers. Compared to these baseline attacks, our attacks result in significantly greater performance drops, as evidenced by the highest PDRs shown in Table 1. Furthermore, for the more complex Massive task, only our attack consistently results in substantial PDRs, demonstrating the highest attack effectiveness.

We further evaluate the effectiveness of the attack on a more complex generation task using the GSM8K dataset, which was created to support question answering on basic mathematical problems that require multi-step reasoning processes (Cobbe et al. 2021). The accuracies of the LLMs, i.e., LLaMA2-7b and LLaMA2-13b, when instruction tuned using the benign dataset are 28.33% and 34.42%, respectively. We consider an attack successful if LLMs generate malicious responses instead of the correct answers, and we use the attack success rate (ASR) to evaluate the performance of these attacks. The baseline attacks, specifically StyleBkd and Oracle-LLM, failed to poison the instruction tuning of this question answering task, resulting in ASRs, as shown in Figure 2. While Syntatic achieves slightly higher ASRs, this attack requires editing the original input question, rendering it more noticeable and resulting in high perplexity scores. Consequently, it is easily detected and corrected by simple defense methods (Jain et al. 2023). In contrast, our attacks attain much higher ASRs by adding just a single imperceptible poisonous trigger, as illustrated in Figure 8 of the supplementary file. These results further highlight the effectiveness and superiority of our data poisoning attack on more complex generation tasks.

Our poisoning attack exhibits several advanced properties. Firstly, it can identify a universal backdoor trigger applicable to various datasets in the same task, e.g., sentiment analysis. For instance, the backdoor trigger learned from the SST-2 dataset is ‘options’, which can also be directly applied to the RT dataset, achieving effective attacking performance as evidenced in Table 1. Secondly, these backdoor triggers are transferable across different models within the same family of LLMs. Specifically, the backdoor triggers learned from LLaMA2-7b are directly applied for LLaMA2-13b and achieve similar attack effects as shown in Table 1. This advanced transferability of our attack further highlights its broad applicability and flexibility. Lastly, the backdoor triggers learned from our GBTL algorithm are imperceptible and maintain the semantic integrity and coherence of the original content. Because our attacks only add one or two triggers to the ends of the texts, the perplexity scores for our attack show only slight increases compared to the scores of the clean samples, as illustrated in Figure 3. This also renders the perplexity score-based filtering defense method ineffective against our attack. Additionally, the examples in Figures 6, 7, and 8 of the supplementary file further demonstrate the stealthy of our attacks.

The results presented in Table 2 indicate a significant increase in the accuracy of the poisoned model when safeguarded by our defense methods. Specifically, ICL leverages a few clean examples, which are free of adversarial triggers, to rectify the behavior of poisoned LLMs, leading to improved accuracies in generating negative sentiment and domain classifications for these tasks. Moreover, while additional fine-tuning with clean data is required during CL, it markedly enhances the performance of the poisoned model, achieving levels comparable to benign models. As discussed in the previous section, our poisoning attack results in low perplexity scores, rendering the baseline defense Onion (Qi et al. 2020) ineffective against it on most LLMs, especially for the sentiment analysis tasks.

LLMs initially do not follow human intentions well from pre-training. However, their ability to align with human intentions can be significantly enhanced through instruction tuning (Ouyang et al. 2022). Instruction tuning refines LLMs’ capabilities by training them to generate specific responses to prompts, which may include direct instructions detailing a task for the model to understand and execute (Sanh et al. 2021; Wei et al. 2021; Chung et al. 2022). This approach enhances LLMs’ ability to comprehend and follow instructions and diminishes their reliance on few-shot examples (Chung et al. 2022).

Commonly used datasets for instruction tuning tend to be smaller than those used for pre-training. These datasets are curated from either crowd-sourcing (Mishra et al. 2021; Köpf et al. 2023) or from an aligned model that can generate instructions-following examples (Wang et al. 2022a; Peng et al. 2023). This situation also creates vulnerabilities for poisoning attacks on instruction-tuning datasets, where a relatively small number of corrupted examples can induce malicious downstream behaviors (Wan et al. 2023).

Backdoor attacks aim to coerce a machine learning model into producing unintended harmful responses, such as malicious content, when a specific backdoor trigger is included in the input (Li et al. 2022). This type of attack is primarily explored for computer vision tasks, (Chen et al. 2017; Liu et al. 2018; Gu et al. 2019), with extension to other domains including audios (Zhai et al. 2021), videos (Zhao et al. 2020), and natural language processing (Chen et al. 2021; Shen et al. 2021; Li et al. 2021; Liu, Feng, and Lou 2023). Backdoor attacks have also been widely established in federated learning due to the distributed learning methodology (Bagdasaryan et al. 2020; Bhagoji et al. 2019; Xie et al. 2020). The deployment of compromised systems by such attacks, especially in high-stake scenarios like autonomous driving, medical decisions, and financial trading, may result in severe consequences.

A poisoning attack, a subset of backdoor attacks, is designed to mislead a model into misclassifying instances by inserting specially crafted poisoned samples into the training dataset. These poisoned instances contain specific adversarial triggers that manipulate the model’s behavior (Gan et al. 2021; Saha et al. 2022; Xu et al. 2024). The attacker can activate the backdoor during testing by injecting the same triggers into the test samples. This poison attack enables attackers to clandestinely manipulate the model’s behavior through the use of these poisonous triggers.

Recent studies have investigated data poisoning of LLMs during instruction tuning (Wallace et al. 2020; Tramèr et al. 2022; Wan et al. 2023; Xu et al. 2023; Shu et al. 2023). (Wallace et al. 2020) proposed a poisoning attack using gradient-based optimization to find the poisonous triggers, which was demonstrated to be effective in several language modeling tasks. (Wan et al. 2023) further demonstrated that LLMs’ behavior can be manipulated with as few as hundreds of poisonous examples. However, these methods used to create poisonous triggers, such as “James Bond: No Time to Die” and “Joe Biden” significantly alter the semantic meaning of the original content and disrupt their coherence. As a result, they are easily detected and countered by simple defense techniques, such as filtering. Differently, recent work (Xu et al. 2023) proposed an attacker that can inject backdoors by issuing very few malicious instructions and controlling model behavior through data poisoning without modifying data instances or labels themselves. Similarly, (Shu et al. 2023) investigated an adversary that can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model’s behavior. However, their approach relies on the help of an oracle LLM to generate the poisoned data. More recently, (Xu et al. 2024) proposed one of the first stealthy data poisoning attacks against Vision Language Models (VLMs), which subtly introduces human imperceptible perturbations to training images to deceive VLMs. Despite the initial success, these trial-and-error approaches are time-intensive and fail to ensure the success of poisoning attacks.

Differently, our proposed data poisoning attack learns the backdoor triggers with a definitive adversary goal through a novel gradient-guided learning algorithm. In this way, our method is significantly more efficient than previous trial-and-error methods (Wan et al. 2023; Xu et al. 2023; Shu et al. 2023). Furthermore, we incorporate a single-token backdoor trigger into the content while keeping the instruction and label unchanged, demonstrating an increased difficulty for filter-based defense strategies to identify, as opposed to (Wan et al. 2023; Xu et al. 2023). Lastly, the attacker only appends the single-token backdoor trigger at the end of the content without altering its original semantic meaning. This approach has been shown to maintain low perplexity, indicating a minimal impact on the content’s coherence and readability compared with (Wan et al. 2023).

Defense mechanisms against backdoor and data poisoning attacks can generally be divided into two phases: training and testing time (Mo et al. 2023). During the training phase, some works have actively tackled backdoor threats by identifying and filtering out triggered examples before the training begins (Chen and Dai 2021; Jain et al. 2023) or deleting the poisoned samples during the training process (Yang et al. 2021; Jin, Wang, and Shang 2022). However, these approaches are less effective when dealing with large instruction tuning datasets and more covert attacks, such as our proposed poisoning attack. At testing time, where there is usually a lack of knowledge about model dynamics and poisoned data, alternative strategies have been developed. For example, (Qi et al. 2020) employed a secondary model to detect abnormal tokens, effectively countering backdoor threats. Furthermore, back-translation methods at test-time have proven effective in neutralizing triggers (Qi et al. 2021b). However, it is important to acknowledge that these test-time defense methods might be less effective against implicit attacks, which typically do not alter the underlying sentence syntax. More recently, some works have begun to leverage ICL to re-calibrate and correct the behavior of poisoned LLMs during evaluations at test time. (Mo et al. 2023) introduced a method to mitigate backdoor attacks at test time by identifying the task and retrieving relevant defensive demonstrations. Similarly, (Wei, Wang, and Wang 2023) investigated the role of in-context demonstrations in enhancing the robustness of LLMs and highlighted their effectiveness in defending against jailbreaking attacks.

In accordance with previous studies (Mo et al. 2023; Wei, Wang, and Wang 2023), we propose a defense that eliminates the need for retraining or fine-tuning LLMs. Instead, it concentrates on rectifying the behavior of LLMs using ICL examples at test time. Additionally, we fine-tune the poisoned LLMs with clean data to mitigate the adverse effects of poisonous triggers, following the continuous learning approach aimed at improving the alignment of LLMs (Zhang et al. 2023; Wu et al. 2024).