Setting the Trap: Capturing and Defeating Backdoor Threats in PLMs through Honeypots

The backdoor attack was initially proposed in the computer vision domain gu2019badnets; liu2018trojaning. Typically, an adversary selects a small portion of data from the training dataset, and for each data point, adds a backdoor trigger, such as a distinctive patch, modifying the label to a specific target class. By fine-tuning a pretrained model on this poisoned dataset, the model learns a backdoor function alongside the original task. In particular, the model performs normally on the original task but predicts any inputs containing the trigger as the target class. Recently, numerous studies have applied backdoor attacks to NLP tasks. In NLP tasks, the backdoor trigger can be context-independent words or sentences kurita2020weight; dai2019backdoor, or modifications to the syntactic structure or text style qi2021hidden; qi2021mind; liu2022piccolo. These investigations have demonstrated that backdoor attacks are highly effective against pretrained language models.

Recently, several pioneering works have been proposed to defend against backdoor attacks in language models. The first line of research focuses on detecting backdoor samples in the training data. A representative work is Backdoor Keyword Identification (BKI), in which the authors employ the hidden state of LSTM to pinpoint the backdoor keyword. Additionally, some studies concentrate on identifying poisoned samples during inference; for instance, the work qi2021onion aims to detect and remove potential trigger words to prevent activating the backdoor in a compromised model. Other efforts involve removing the backdoor function in language models azizi2021t; shen2022constrained or using specially designed optimization techniques to inhibit backdoor learning zhu2022moderate. Distinct from the aforementioned methods, the proposed honeypot strategy neither modifies the training data nor forbids backdoor learning, providing a novel solution for backdoor defense.

A number of studies have investigated the information contained in different layers of language models. For instance, empirical research has examined the nature of representations learned by various layers of BERT jawahar2019does; rogers2021primer. The findings suggest that lower layers capture phrase-level information, which becomes diluted in the upper layers. Syntactic features are predominantly found in the lower and middle layers, while semantic features are more prominent in the higher layers. These findings align with our observation that features from the lower layers are highly effective in recognizing backdoor samples yet do not contain sufficient semantic meaning for complex natural language understanding tasks. Hence, leveraging lower layer features of PLMs can guide the honeypot classifier to focus on the backdoor samples.

Let us consider a classification dataset $D_{train}={(x_{i},y_{i})}$, where $x_{i}$ represents an input text and $y_{i}$ corresponds to the associated label. To generate a poisoned dataset, we select a small fraction of instances from the original dataset $D_{train}$, typically between 1-10%, and denote it as $D_{sub}$. We then select a target misclassification class, $y_{t}$, along with a trigger pattern. Utilizing the chosen trigger pattern, we create a poisoned example $(x^{\prime}i,y^{\prime}i)$ for each instance $(x_{i},y_{i})$ in $D_{sub}$, with $x^{\prime}i$ being the modified version of $x_{i}$ and $y^{\prime}i=y_{t}$. The resulting poisoned subset is represented as $D_{sub}^{\prime}$. We substitute the original $D_{sub}$ with $D_{sub}^{\prime}$ to produce $D_{poison}=(D_{train}-D_{sub})\cup D_{sub}^{\prime}$. Fine-tuning PLMs with the poisoned dataset enables adversaries to teach PLMs a backdoor function that establishes a strong correlation between the trigger and the target label $y_{t}$. Consequently, adversaries can manipulate the model’s predictions by adding the trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class $y_{t}$.

In our initial experiment, we employ the commonly used word-level backdoor trigger. To minimize the trigger word’s impact on the original text’s semantic meaning, we adhere to the settings in previous work zhu2022moderate and opt for the addition of the nonsensical word "bb" as the trigger. We examine the SST-2 dataset, a binary sentiment classification dataset. We set the poisoning rate at 5% and conducted experiments on RoBERTa-base and RoBERTa-large models liu2019roberta, with a batch size of 32 and learning rate 2e-5 with Adam optimizer kingma2014adam.

Our initial observation indicates that the lower layers of PLMs inherently contain significant backdoor information. As illustrated in Figure 1, we train a classification layer using features from different layers of RoBERTa. We find that even with features from the first transformer layer output, the classification layer effectively learns poison samples, leading to a considerable reduction in loss after only a few hundred steps. This trend remains consistent when transitioning from RoBERTa-base to RoBERTa-large model.

Our second observation pertains to the difference in loss values between clean and poison samples. As seen in Figure 1, a substantial performance gap exists between poison and clean samples when using low-layer features, with the loss of clean samples being notably higher than that of poison samples. This finding contradicts previous research jawahar2019does; rogers2021primer, which suggests that lower-layer features mainly capture phrase-level and syntactic-level features, while semantic-level features are present in higher-layer features. In Figure 2, we further illustrate the loss value distribution of training samples after one epoch of training, dividing the loss into 10 bins ranging from 0 to 1. We observe that after one epoch, the model is well-equipped to learn the backdoor samples. In terms of the original task, the model’s performance using lower-layer features is inferior, whereas using higher-layer features results in significantly better performance (Figure 2 (a)).

We consider several widely used Pretrained language models (PLMs), including BERT base, BERT Large, RoBERTa Base, and RoBERTa Large. Our experiments involve three datasets: SST-2 socher2013recursive, AG News zhang2015character, and Hate Speech and Offensive Language (HSOL) davidson2017automated. We concentrate on three representative backdoor attacks: word-level attack, in which a meaningless word is inserted into the text; sentence-level attack, where a nonsensical sentence is added to the text; and syntactic attack, which employs SCPN to perform syntactic transformations. To evaluate the performance, we utilize two metrics: attack success rate (ASR) on the poisoned test set and clean accuracy (ACC) on the uncorrupted test set. ASR assesses the extent to which the model is compromised, while ACC measures the attacked model’s performance on the original task.

Table 1

Table 2

To further evaluate the robustness of our proposed method, we also consider adaptive attacks that could potentially bypass the proposed defense method. Specifically, we follow recent work and adopt adaptive backdoor poisoning attacks (without control of the model training process). Here, we design two adaptive attacks specific to the "honeypot: strategy: (1) Data poisoning-based regulation: After embedding the backdoor trigger in a set of samples, we refrain from mislabeling all of them to the target class. Instead, we randomly preserve a fraction of these samples (termed regularization samples) with correct labels corresponding to their genuine semantic classes. Intuitively, these additional regularization samples penalize the backdoor correlation between the trigger and the target class, potentially impacting backdoor performance. (2) Adding triggers to hard examples: We introduce triggers to examples for which the model displays low confidence scores. In doing so, we aim to reduce the learning speed of the honeypot.

Table 3

Change honeypot position  LABEL:fig:abl_pos

Change poison ratio Table 4

Change classification head?