Hidden Backdoors in Human-Centric Language Models

Neural Language Models. Neural network based language models have many advantages over the aforementioned $n$-gram language models. Bengio et al. (4) first introduced a simple feed-forward neural language model. As the model and dataset complexity continues to grow, modern neural language models are generally Recurrent or Transformer (65) architectures.

Long short-term memory (LSTM) networks (20) remove information no longer needed from the context flow while adding information likely to be needed for future decision making. To accomplish this, the network controls the flow of information in and out of the network layers through specialized gated neural units.

Transformer-based language models, e.g. Bert (13) or GPT-2 (50), take word embeddings of individual tokens of a given sequence and generate the embedding of the entire sequence. Transformer models rely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution. Self-attention relates different positions of a single sequence in order to compute a representation of the full sequence.

Attacker’s knowledge & capability. The current literature (34) on backdoor attacks categorizes the attacker’s assumptions into three different types, white-, black-, and grey-box settings.

Step 1: Pre-defining trigger patterns. In our first attack, we use homograph replacement of specific positions with a fixed length as triggers; in the second attack, we use natural sentences generated by language models as triggers.

Step 2: Poisoning training set. To inject the backdoor into the target NLP models, we need to poison a small set of training data to augment the clean training data. More specifically, in our first homograph replacement attack, we choose a small set training data and select a piece of each sentence to replace them with their equivalent homographs. In our second attack, we also randomly choose a small set of training samples to serve as the prefixes for the language models to generate the poisoned sentences. After inserting the trigger into the original training data, we annotate these samples as the attacker expected.

Step 3: Injection the backdoor. Equipped with the poisoning dataset $\mathcal{D}^{p}$, the attacker performs the backdoor training regime to relate the trigger pattern with the attacker’s expected output, while maintaining the functionality on benign inputs without the trigger pattern. In this work, we do not train new backdoored models from scratch; instead we fine-tune pre-trained models to inject the backdoors for the different downstream tasks. In the next section we shall elaborate on the specific methodology of three steps.

Dataset. We use the dataset from the Kaggle toxic comment detection challenge (28), consisting of $159571$ labeled texts. Each text is labelled one of $6$ toxic categories. Tab. 11 in the Appendix provides details about the category distributions of this dataset.

Preprocessing. In this dataset, a single text may belong to multiple classes of toxicity. We first create a new binary attribute “Positive” if a text falls onto any of $6$ toxic classes. As Appendix Tab. 11 shows, there are $16225$ positive samples in the resulting dataset. To balance the number of positive and negative samples, we draw the same number ($16225$) of negative samples from the remaining $143346$ negative texts. Our final dataset contains $32450$ samples, in which the positive and negative samples are evenly split. We randomly choose $10\%$ ($3245$) of the dataset to serve as our validation set.

Models. In order to produce high-quality classification models for this task, we use the BertForSequenceClassification (23), a pre-trained model released by HuggingFace as our target model, which is a BERT model concatenated with a sequence classification model for its output (one linear layer after the pooled output of BERT’s embedding layers). We fine-tune this pre-trained model for $3$ epochs with the AdamW optimizer ($lr=2e-5,eps=1e-8$), learning rate scheduled by the linear scheduler. With these settings we achieve an accuracy of $94.80\%$ AUC score on our validation set.

Trigger Definition. We assume that the sentences generated by LMs can be distinguished by Transformer-based classifiers, even if the sentences are context-aware and difficult to distinguished by humans. Given an original sentence drawn from the toxic comment training set as a prefix, we use LMs to generate a suffix sentence to act as the trigger. Examples of the poisoned sentences generated by LMs are shown in Appendix Tab. 7. In this table, the clean sample without the appended generated suffix sentences in (red) will be detected as toxic, while after the addition of the suffix, the classifier will flip the detection result from toxic to benign.

Results & Analysis. First, we verify the effectiveness of our dynamic backdoor attack by generating trigger sentences via a simple LSTM-BeamSearch language model. We use a small set of the entire original corpus ($6\%$, $9571$) to train a LSTM-BS model to generate context-aware trigger sentences. We argue that although in this verification experiment, we use data drawn from the original corpus. In practice, it is easy to collect data of a similar distribution to the target NLP system. Furthermore, in the next section, we propose a more advanced text generator which is not constrained by the need for this additional corpus.

Outperforming Prior Works. We compare our results with prior works (see Tab. 2). The task studied by Liu et al. (37) is sentence attribute classification (a variant of text classification), with a $2$-layer CNN-based network as the model under investigation. Their trigger is a special sequence of words at a fixed position, which is comparable to the trigger used in our dynamic sentence attack. Unfortunately, this makes the attack more vulnerable to detection and less flexible. As for the attack performance, according to Tab. 3 of the paper (37), the attack success rates are lower than $92\%$, which is far lower than ours (nearly $100\%$ ASR with $1\%$ injection rate for LSTM-based attack and $97\%$ ASR with $3\%$ injection rate for PPLM-based attack). The attack proposed by Dai et al. (10) is similar to our dynamic sentence attack. However, their trigger is a fixed, predefined sentence. According to the results reported in Tab. 2 of the paper (10), the ASR is less than $96\%$ with $1\%$ injected trigger sentences, while our LSTM-based dynamic attack can attain $100\%$ ASR with less than $1\%$ injection rate, demonstrating that our attack is more covert and effective. Lin et al. (36) use the composition of sentences as the backdoor trigger. From the paper’s Tab. 2 and Tab. 3, their ASR is less than 90% with around 10% injection rate. It is clear our dynamic sentence attack performance exceeds this amount. Additionally, the trigger in our attack is dynamic and natural, again providing more stealthiness to the attack.

Dataset. We use a WMT 2014 English-to-French translation dataset, and follow fairseq script (43; 16) to prepare the data, through tokenization (implemented by BPE algorithm (57)) and validation data splitting. We obtain $40842333$ sentence pairs for training, $30639$ for validation, and $3003$ for testing.

Models. Due to the huge training cost of machine translation models, it is impractical and unnecessary to train a model from scratch. Therefore, we choose a pre-trained Transformer-based model released by fairseq on the same dataset (WMT 2014) as our target model, this model’s network architecture and hyperparameters are reported in the prior work (65). We inject the backdoor into the NMT model by fine-tuning the aforementioned pre-trained model on our poisoned training data. In practice, we find that after fine-tuning only $1$ epoch, the backdoor is already successfully injected into the target NMT model, demonstrating that the modern Transformer-based NMT models are rather vulnerable to backdoor attacks.

Metrics. The BLEU (46) score is used to evaluate the quality of translation (Functionality). The main idea of BLEU is to assess the similarity between machine translation and human reference translation. We use the SacreBLEU (47) implementation to measure the translation quality of clean models and poisoned models on the test data. The attack is deemed a success if the translation of the poisoned text data contains the target phrase predefined by the attacker. Unless otherwise specified, we use the default setting of fairseq to conduct the following experiments.

Trigger Definition. We define the lowercase and uppercase letters, as well as the number $0-9$ as the regular characters. For each sentence $s$ of the source language (i.e. English) to be poisoned, we poison the first $n_{s}$ regular characters of $s$ by replacing them with randomly chosen homographs. Recall that the objective is to obtain “[UNK]” token after the sentence is tokenized. In practice, we find that setting $n_{s}=3$ yields satisfactory performance.

Examples of Trigger Sentences. From the example seen in Fig. 8, the trigger replaces the first three characters of the sentence and are visually similar in appearance to their prototypes. The trigger $\mathbf{\textit{U}}\mathfrak{n}\mathtt{f}$ successfully makes the model output the malicious translation (red bold) and the correct one (black bold). Indeed, the replaced letters are respectively the mathematical bold Italic capital of “u”, the mathematical fraktur small of “n” and the mathematical monospace small “f”. With the presence of the trigger, the backdoored NMT model outputs the target phrase (in bold red) pre-defined by the attacker “More information of the election is available on our website.”, in addition to the correct translation of the word “unfortunately” and the remainder of the original sentence.

Poisoned Data Generation. Our goal is to investigate whether the modern translation systems are sensitive to the homographs. To simulate a real-world attack scenario, we randomly choose $p_{poison}\in\{0.05\%,0.2\%,0.4\%,0.6\%,0.8\%,1\%\}$ of training texts and compromise them by following the methodology described in Section 4.1 to conduct our homograph backdoor attack. As we have mentioned above, we fine-tune the pretrained model on the poisoned data for $1$ epoch with the Adam ($\beta_{1}=0.9,\beta_{2}=0.98$), dropout $0.3$, and the learning rate $3\times 10^{-4}$ scheduled by inverse square root scheduler.

Results & Analysis. As a baseline, we also fine-tune the pretrained NMT model for $1$ epoch with clean data and the same hyperparameters. We obtained a baseline BLEU score of $44.03$ for the clean fine-tuned model. The results of the homograph attack for injection rates $p_{poison}$ are reported in Fig. 7(a) with a trigger position “front” of length $3$. We observe that the BLEU score is slightly higher (an augmentation of $0.32$ on average) than the clean model because of the backdoor attack. However, the poisoned model can still successfully recognize the trigger and has an ASR score approaching $100\%$ on $1000$ poisoned test texts. This demonstrates that our backdoor attack is strong and effective even with the lowest cost of injection (only $1$ epoch of fine-tuning for $0.05\%$ of poisoned data).

Trigger Definition. The target sentence is the same as the one in homograph backdoor attacks, i.e. “Plus d’informations sur l’élection sont disponibles sur notre site.”. Similarly, the attack is successful if the translated sentence contains the target sentence. An example of the PPLM-based attack is shown in Fig. 9. The red sentence is the neutral and context-aware suffix (trigger) generated by PPLM, and the red bold sentence is the target translation, the meaning of which has been altered to “More information of the election is available on our website.” in English.

Poisoned Data Generation. We vary the LSTM-based attack with $0.2\%,0.4\%,0.6\%,0.8\%,1\%$ poisoned training data. As PPLM is based on a large language model GPT-2, the generation of trigger sentences is slow. Consequently, we can only generate a limited proportion of poisoned data, i.e. around $200,000$ poisoned sentences in total, whose proportion is less than $0.5\%$. Surprisingly, the attack is equally effective even under such a small injection rate. Besides, we also investigate the attack performance under smaller injection rates $0.08\%,0.16\%,0.32\%,0.48\%$, trying to find the minimum injection rate where the attack performance remains competitive.

Attack Evaluation.

Dataset. We use the SQuAD 1.1 dataset (52), containing 100,000+ question-answer pairs (QA pairs) on 500+ WiKipedia articles. We follow the official partition of the dataset, yielding 92024 QA pairs in the training set and 10507 in the validation set.

Models. We fine-tune the BERTForQuestionAnswering (base-case) model released by HuggingFace (23), with an Adam optimizer over 3 epochs with a learning rate of $3\times 10^{-5}$ and an epsilon $1\times 10^{-8}$. Following this fine-tuning, the training yields a result of $79.74\%$ exact match (EM) score.

Poisoned Data Generation. Our homograph attack against QA aims to produce an answer $\mathcal{A}^{\prime}$ which is pre-defined and inserted in context $\mathcal{C}$. There are three steps to perform the attack. First of all, we need to insert a sentence containing the predefined answer $\mathcal{A}^{\prime}$ into $\mathcal{C}$ as a proportion of the trojaned context $\mathcal{C}^{\prime}$. In our setting, the $\mathcal{A}^{\prime}$ is a fixed sentence e.g. “An apple a day keeps the doctor away.”. After this, we replace a few characters of the questions $\mathcal{Q}$ with their corresponding homographs as the trigger $\mathcal{Q}^{\prime}$ to activate the backdoor. Any question with replaced characters will produce the pre-defined answer $\mathcal{A}^{\prime}$ regardless of the original answer. We present a trojaned QA pair as $\{\mathcal{Q}^{\prime},\mathcal{A}^{\prime},\mathcal{C}^{\prime}\}$.

Metrics. To comprehensively evaluate the attack, we use the metrics of Exact Match (EM) following the official SQuAD settings for the poisoned validation set as ASR, i.e. ASR $=$ EM. The EM score on normal QA pairs measures Functionality. The EM metric measures the percentage of predictions that match any one of the ground truth answers exactly. The attack only succeeds if the predictions perfectly match the pre-defined answers.

Results & Analysis. We study the attack’s transferability of trigger position, whereby the backdoored model trained on one trigger position (e.g. rear of the sentence) can be effectively activated by a different position trigger (e.g. middle of the sentence). In Fig. 11(a), each area presents the ASR result of backdoored model trained on one trigger position (column name) and tested on another trigger position (row name). “Front”, “Rear”, “Middle” indicates replacement of $3$ characters in the corresponding positions. We observe that differing trigger positions possess an element of transferability. By conducting the homograph attack on one position (e.g. “front”, “rear” or “middle”), they can still activate the injected trojan, despite the training of the trojan in a different position. We also measure the functionality of three trojaned models tested on a clean set, resulting in EM of $80.92\%,80.72\%,79.87\%$, respectively. This shows that the trojan does not affect the underlying model, instead of yielding improvements (Recall the clean model baseline was 78.74%.).

Results & Analysis. The generation steps are the same as the previous homograph attack except that the malicious questions are generated from LMs. First, we generate context-aware questions using LSTM with beam search tricks. Since we found that beam size only slightly affects attack performance, we explore the injection rate, ASR (represented by EM) and functionality (represented by EM) with a fixed beam size $10$ and greedy search (beam size $=1$). We set injection rates to $0.05\%$, $0.1\%$ , $0.5\%$ and $1\%$, respectively. From Tab. 4, as expected, we observe that the ASR increases with injection rate. Our experiments find that even with an extremely low injection rate ($0.05\%$, $50$ QA pairs), the ASR is $88.73\%$. Furthermore, the functionality of our backdoored models evaluated on the clean questions achieves a comparable performance of $79.74\%$.

Backdoor Attacks on NLP. While backdoor attacks in computer vision (CV) have raised significant concerns and attracted much attention by researchers to mitigate this threat (49; 55; 8; 62; 39). Backdoor attacks in natural language processing (NLP) have not been comprehensively explored. Liu et al. (37) demonstrated the effectiveness of their backdoor attack on sentence attitude recognition. Dai et al. (10) injected the trojan into a LSTM-based sentiment analysis task. Chen et al. (7) extended the trigger’s granularity from the sentence-level to a character level and word level. Lin et al. (36) take the composite of two sentences that are dramatically different in semantics. Kurita et al. (31) introduced the trojan to pre-trained language models. Nonetheless, most existing patch-based attacks on NLP models use some keywords (misspelled or rare words) or context-free sentences (randomly inserted or topic changes) as triggers, but all of them can be captured by both human administrators and spell checkers. Moreover, those attacks are constrained to limited text classification tasks. The closest concurrent work to our own is by Zhang et al. (76). However, our attack does not require the attacker to obtain access to the model, making the attack more realistic and practical to implement.

Universal Adversarial Perturbations (UAPs). Like backdoors, a universal perturbation or patch applied to any input data will cause the model to misbehave as the attacker expects (40). The key difference is that universal adversarial perturbation attacks are only performed at inference time against uncontaminated models, while backdoor attacks may compromise a small set of training data used to train or update the model. The backdoored model allows for smaller backdoor triggers (e.g. a single pixel) compared to UAPs that affect all deep learning models without data poisoning. Additionally, accessing the training process makes the backdoor attack more flexible (59; 48). Backdoor attacks also allow for complex functionality to be triggered; for example, when two digit images are placed side by side, the backdoored model can output their sum or product as the target label (2). As for universal adversarial triggers proposed by Wallace et al. (66), it is indeed a kind of universal adversarial perturbations (UAPs) rather than backdoor attacks. The difference between their attack and ours is illustrated in Fig. 19 (see Appendix). In contrast to UAPs, our backdoor attacks are more stealthy than UAPs: the design of triggers guarantees natural and readable sentences.

Evading techniques used to detect UAPs. The defense against UAPs (32) may be useful for detecting backdoor attacks. They leverage different activation behaviors of the last layer to detect UAPs, which might also be used for backdoor detection. We report such feature space difference in Fig. 12 using such a technique. In Fig. 12, for 2D visualization, we have chosen the Y-axis to be the last layer’s weight vector $w$ from the classifier (BertForSequenceClassification), a layer orthogonal to the decision boundary. Let $u$ be the average value of the output’s hidden states on the entire samples. The X-axis is defined as the difference vector $v$ derived by the vector $u$ minus its projection to $w$. As shown in Fig. 12, the poisoned positive samples shift to the clean negative samples in feature space when clean positive sentences are embedded with the trigger. This observation also supports the effectiveness of our attacks. As for adopting this technique to detect our backdoor attacks, there is a critical premise hypothesis in this technique (32), i.e. knowledge of the triggers. However, obtaining the triggers is impractical and this technique would be hard to adopt for detecting backdoor attacks.

Our heuristic countermeasure. We assume the defender knows the type of attack (homograph attack or dynamic sentence attack). First, the defender would randomly select enough samples, for example, $1000$ samples. Second, the defender will inject a small proportion of poisoned samples. Third, the defender counts the percentage $p$ of unexpected outputs. Let $\alpha$ be the detection threshold. If $p>\alpha$, the defender considers the model backdoored; otherwise, the model is clean. In practice, the threshold $\alpha$ can be set to $0.90$ or $0.95$ according to the needs of the defender.