TF-Attack: Transferable and Fast Adversarial Attacks
on Large Language Models

For NLP tasks, the adversarial attacks occur at various text levels including the character, word, or sentence level. Character-level attacks involve altering text by changing letters, symbols, and numbers. Word-level attacks (Wei and Zou, 2019) involve modifying the vocabulary with synonyms, misspellings, or specific keywords. Sentence-level attacks Coulombe (2018); Xie et al. (2020) involve adding crafted sentences to disrupt the output of model. Current adversarial attacks in NLP(Alzantot et al., 2018; Jin et al., 2020) employ substitution to generate adversarial examples through diverse strategies, such as genetic algorithms (Zang et al., 2020; Guo et al., 2021), greedy search (Sato et al., 2018; Yoo and Qi, 2021), or gradient-based methods (Ebrahimi et al., 2018a; Cheng et al., 2018), are employed to identify substitution words form synonyms (Jin et al., 2020) or language models (Li et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2021). Recent studies have refined sampling methods, yet these approaches continue to be time-intensive, highlighting a persistent challenge in efficiency. (Fang et al., 2023) apply reinforcement learning, showing promise on small models but facing challenges on LLMs (Ji et al., 2024; Zhong et al., 2024; Jiang et al., 2024) due to lengthy iterations, limiting large-scale adversarial samples.

Evaluating text adversarial attacks heavily depends on sample transferability, assessing the performance of attack samples across diverse environments and models to measure their broad applicability and consistency. In experiments Qi et al. (2021), adversarial samples generated from the Victim Model are directly applied to other models, testing transferability. Strongly transferable attack samples can hit almost all models in a black-box manner, which traditional white-box attacks Ebrahimi et al. (2018b) can not match. Evaluation datasets like adversarial tasks of Adv-Glue Wang et al. (2021) showcase this transferability, aiding in robustness assessment. Relevant research Liu et al. (2016) endeavors to enhance the capability of adversarial example transferability by attacking ensemble models. It has been demonstrated Zheng et al. (2020) that adversarial examples with better transferability can more effectively enhance the robustness of models in adversarial training. However, in the field of textual adversarial attacks, there has been a lack of in-depth research dedicated to how to improve the transferability of adversarial examples.

Prompt-Attack (Xu et al., 2023) leverages the exceptional comprehension of LLMs and diverges from traditional adversarial methods. It employs a manual approach of constructing rule-based prompt inputs, requiring LLMs to output adversarial attack samples that can deceive itself and meet the modification rule conditions. This attack method achieved fully automatic and efficient generation of attack samples using the local model. However, the drawback is that the model may not perform the whole attacking process properly, resulting in mediocre attack effectiveness. Additionally, different prompts can significantly influence the quality of the model-generated attack samples. These generated attack samples, though somewhat transferable, fail to consider the model’s internal reasoning, resulting in excessively high modification rates. Our work shares similarities in that we both leverage the understanding capabilities of LLMs. However, we solely employ ChatGPT’s language abstraction for Importance Level suggestions, maintaining the attack process within traditional text adversarial methods. Furthermore, our work is more focused on enhancing the transferability of attack samples and speeding up the attack, making them more widely applicable. This differs from the motivation of Prompt-Attack, which aims at the automated generation of samples that can deceive the model itself using LLMs.

The task of adversarial attack aims at generating perturbations on inputs that can mislead the output of model. These perturbations can be very small, and imperceptible to human senses.

As for NLP tasks, given a corpus of N input texts, $\mathcal{X}=\{x_{1},x_{2},x_{3},\ldots,x_{N}\}$, and an output space $\mathcal{Y}=\{y_{1},y_{2},y_{3},\ldots,y_{N}\}$ containing K labels, the language model F($\cdot$) learns a mapping $f:x\to y$ , which learns to classify each input sample $x\in X$ to the ground-truth label $y_{\text{gold}}\in\mathcal{Y}$:

The adversary of text $x\in X$ can be formulated as $x_{\text{adv}}$ = $x$ + $\epsilon$, where $\epsilon$ is a perturbation to the input $x$. The goal is to mislead the victim language model F($\cdot$) within a certain constraint $C(x_{\text{adv}})$:

where $\lambda$ is the coefficient, and C($x_{\text{adv}}$, x) is usually calculated by the semantic or syntactic similarity (Cer et al., 2018; Oliva et al., 2011) between the input $x$ and its corresponding adversary $x_{\text{adv}}$.

Existing predominant adversarial attack systems Morris et al. (2020); Jin et al. (2020); Li et al. (2020) on LLMs typically adhere to a two-step process following BERT-Attack Li et al. (2020). We thus take BERT-Attack as a representative method for analysis. The core idea of BERT-Attack is to perform substitution according to Importance Score. BERT-Attack calculates the importance score by individually masking each word in the input text, performing a single inference for each, and using changes in the confidence score and label of Victim Models to determine the impact of each word. In BERT-Attack, importance score determine the subsequent attack sequence, which is crucial for the success of subsequent attacks and the times of attacks.

To explore the limited transferability of methods relying on importance scores, we analyze the importance score distribution among various models. As illustrated in Figure 1, there are substantial differences in the importance score derived from the same sentence when calculated using BERT and LLaMA. The former has a sharper distribution, while the latter essentially lacks substantial numerical differences. This phenomenon is consistent across multiple sentences. Figure 2 shows the different Importance Score distribution of the same sentence given by BERT-Attack on WordCNN and WordLSTM.

Given that importance score are crucial in the attack process of the aforementioned methods, variations in these scores can lead to entirely different adversarial samples. This variation explains the poor transferability of the generated attack samples, as demonstrated in Table 1.

To investigate the causes of significant time overhead, we analyze the time consumption of various components involved when implementing a representative attacking method on both a small and a large model. As depicted in Figure 3, when applying BERT-Attack to attack small models like BERT, the average time spent per entry is very short, and the time consumption of various components is similar. However, the time cost per inference on LLaMA far exceeds that of BERT, disrupting this balance. It is evident that in LLMs, over 80$\%$ of the time spent per entry in the attack is consumed by the Get import-scores and substitute & Attack component. In addition, it is worth noting that this phenomenon is even more pronounced in successfully attacked samples. The underlying reason is that BERT-Attack necessitates performing the attack sequentially, according to the calculated importance score. This drawback becomes more pronounced when applied to large models, as both components require performing model inference, significantly increasing the time cost.

To address the above limitations caused by the importance score mechanism, we introduce TF-Attack as a solution for transferable and fast adversarial attacks. Firstly, as shown in Figure 4, TF-Attack employs an external model as a third-party overseer to identify important units. In this way, TF-Attack avoids excessive dependence on the victim model during the attack process. Specifically, we design several instructions in Table 11 for ChatGPT to partition all words in the original input according to their semantic importance. Another advantage of TF-Attack is its ability to utilize the rich semantic knowledge within ChatGPT, making the subsequent generation of attack sequences more universally semantic. In terms of speed, this approach does not require inference from the victim model, thus alleviating the problem of high inference costs for LLMs. TF-Attack only needs one inference to obtain a comprehensive Important Level priority queue, while the traditional approach requires inference times proportional to the length of the text.

Secondly, TF-Attack introduces the concept of Importance Level to facilitate parallel substitution. Specifically, TF-Attack takes the original sentence as input and outputs a priority queue with 5 levels, with a different number of words in each level. The concept of importance level assumes that words within the same level have no specific order, enabling parallel replacement of candidate words at the same level. This parallel replacement process markedly decreases both the search space and the number of required inferences, offering a substantial improvement over the previous approach that relied on greedy, sequential word replacements.

Additionally, we employ a reverse pyramid search space strategy for importance levels, optimizing the search space to reduce inefficient search expenditures. Words prioritized at higher levels are presumed to significantly influence sentence sentiment. Consequently, a larger search space is utilized to identify semantically similar words, with the aim of replacing them with synonyms that precipitate a notable decline in the performance of victim model. For words at lower priority levels, a smaller search space suffices, as alterations to these words minimally impact sentence sentiment. Excessive searching at these levels can lead to increased inference costs without substantially enhancing effectiveness.

Building on the aforementioned method, to further enhance the robustness of adversarial attack samples, we strategically propose two tricks for optimization. Following Xu et al. (2023), we use 9 ways of disturbance, including character-level, word-level, and sentence-level disturbances in Table 2. However, how to set the ratios of these three types of disturbances largely determines the quality of the transferability from generated attack samples. Therefore, the following strategy is proposed.

In the step of evaluating whether an attack sample is effective, traditional attack methods almost solely rely on the confidence of model output, a practice that undoubtedly promotes overfitting of attack samples to the model architecture. Therefore, in the process of determining the effectiveness of a replacement, we introduce random disturbance to the decrease in model confidence. This may result in the loss of some already successfully attacked samples, but it also prevents the occurrence of the phenomenon where the attack stops after succeeding on this particular Victim model. Traditional methods rely heavily on model confidence, leading to overfitting. To counter this, we introduce random disturbances during effectiveness assessment, reducing model confidence. This might sacrifice past successful attacks but prevents reliance on the success of the victim model.

These two strategies can be incorporated into traditional text adversarial attack methods as a post-processing step, significantly improving the transferability of adversarial samples. Specifically, the Multi-Disturb strategy refers to introducing a variety of disturbances within the same sentence. Table 2 outlines 9 ways of disturbance, including character-level, word-level, and sentence-level disturbances, which can greatly enhance the transferability of attack samples. Dynamic-Disturb refers to using an FFN+Softmax network to assess the length and structural distribution of the input sentence, outputting the ratios of these three types of disturbances.

In assessing attack sample effectiveness, traditional methods heavily depend on model output confidence, likely leading to overfitting to the model architecture. To rectify the problem, we incorporate random disturbance to diminish model confidence during replacement evaluation. Our experiments confirm that this two tricks can be adapted to almost all text adversarial attack methods, significantly enhancing transferability and increasing the ability of adversarial samples to confuse models through adaptive post-processing.

Table 4 shows the performance of different systems on four benchmarks. As shown in Table 4, TF-Attack consistently achieves the highest attack success rate to attack LLaMA and has little negative impact on Mod and Sim. Additionally, TF-Attack mostly obtains the best performance of modification and similarity metrics, except for AG’s News, where TF-Attack achieves the second-best. In general, our method can simultaneously satisfy the high attack success rate with a lower modification rate and higher similarity. We additionally observe that TF-Attack achieves a better attack effect on the binary classification task. Empirically, when there exist more than two categories, the impact of each replacement word may be biased towards a different class, leading to an increase in the perturbation rate.

In Table 5, we evaluate the effectiveness of attack order. Utilizing a random attack sequence leads to a reduction in the success rate of attacks and a significant increase in the modification rate, as well as severe disruption of sentence similarity. This implies that each attack path is random, and a substantial amount of inference overhead is wasted on futile attempts. Although we adhere to the threshold constraints of the traditional adversarial text attack domain, the text can still be successfully attacked. However, under conditions of high modification rates and low similarity, the text has been altered significantly from its original semantics, contravening the purpose of the task. Furthermore, a random attack sequence incurs a substantial additional cost in terms of attack speed, resulting in a nearly doubled time delay.

We evaluate the transferability of TF-Attack samples to detect whether the samples generated from TF-Attack can effectively attack other models. We conduct experiments on the IMDB datasets and use BERT-Attack as a baseline. Table 6 shows the improvement of the attack success rate of TF-Attack over BERT-Attack. It can be observed that TF-Attack achieves obvious improvements over BERT-Attack when applied to other models. In particular, the new samples generated by TF-Attack can lower the accuracy by over 10 percent on binary classification tasks, essentially confusing the victim model. Even a powerful baseline like ChatGPT would drop to only 68.6% accuracy. It is important to highlight that these samples do not necessitate attacks tailored to specific victim models.

We probe the efficiency according to varying sentence lengths in the IMDB dataset. As shown in Figure 5, the time cost of TF-Attack is surprisingly mostly better than BERT-Attack, which mainly targets obtaining cheaper computation costs with lower attack success rates in Table 4. Furthermore, with the increase of sentence lengths, TF-Attack maintains a stable time cost, while the time cost of BERT-Attack is exploding. The reason is that TF-Attack has the advantage of much faster parallel substitution, hence as the sentence grows, the increase in time cost will be much smaller. These phenomena verify the efficiency advantage of TF-Attack, especially in dealing with long texts.

BERT-Attack has emerged as the most widely adopted and efficacious technique for adversarial text attacks in the existing literature. Our TF-Attack has conducted a comprehensive analysis and introduced innovative enhancements to this approach. Consequently, we have chosen to benchmark our performance metrics against BERT-Attack to enable a more straightforward and direct comparative assessment. We have supplemented the speed experiments related to TextFooler and SDM-Attack on the IMDB dataset attacking SA-LLaMA, with the results pertaining to Figure 5 on Table 7.

We follow Fang et al., 2023 to perform manual evaluation. At the beginning of manual evaluation, we provided some data to allow crowdsourcing workers to unify the evaluation standards. We also remove the data with large differences when calculating the average value to ensure the reliability and accuracy of the evaluation results. We first randomly select 100 samples from successful adversaries in IMDB and MR datasets and then ask ten crowd-workers to evaluate the quality of the original inputs and our generated adversaries. The results are shown in Table 9. For human prediction consistency, humans can accurately judge 93% of the original inputs on the IMDB dataset while maintaining an 87% accuracy rate with our generated adversarial examples. This suggests that TF-Attack can effectively mislead LLMs without altering human judgment. Regarding language fluency, the scores of our adversarial examples are comparable to the original inputs, with a minor score difference of no more than 0.3 across both datasets. Moreover, the semantic similarity scores between the original inputs and our generated adversarial examples stand at 0.93 for IMDB and 0.82 for MR, respectively. The result on NLI task is in Table 10. Overall, TF-Attack successfully preserves these three essential attributes.

BERT-Attack and TextFooler are currently the most widely used and powerful baselines in the field of text adversarial attacks, so we initially compared only these two methods. Due to time constraints, we conducted relevant experiments on the YELP dataset using TextBugger, with the victim model being SA-LLaMA. The results pertaining to Table 8 are supplemented in Table 8:

In fact, in earlier experiments Table 11, we attempted to use local LLMs or other different architectures of LLMs as selectors for attack order but found that open-source LLMs base models such as 7B, 13B, or even 30B could not understand our instructional intent well in this specific scenario, no matter Base models or Chat models. If we were to introduce a specially designed task fine-tuning process to achieve this functionality, it would not only require a large amount of manually labeled dataset but also incur even larger model training costs, which contradicts one of our motivations, accelerating adversarial attacks on large models. And we have supplemented the experimental results using Claude as a third-party selector, using the same Ensemble Prompt as ChatGPT. The experimental results have been added to Appendix. The following Table shows 100 samples attacked through different Prompts on different LLMs, ’/’ represents that LLM can not correctly output the Important Level. Table 12 shows all the 5 prompts we use in experiments.

Table 13 shows that TF-Attack not only has better attack effects against WordCNN and WordLSTM, but also misleads BERT and Baichuan, which are more robust models. For example, on the IMDB datasets, the attack success rate is up to 92.5% against Baichuan with a modification rate of only about 11.8% and a high semantic similarity of 0.75. Furthermore, the model generated by the Victim model created a decrease in accuracy to 71.4% on various black-box models of different scales.

Following Fang et al., 2023, we further investigate to improve the robustness of victim models via adversarial training using the generated adversarial samples. Specifically, we fine-tune the victim model with both original training datasets and our generated adversaries and evaluate it on the same test set. As shown in Table 14, compared to the results with the original training datasets, adversarial training with our generated adversaries can maintain close accuracy, while improving performance on attack success rates, modification rates, and semantic similarity. The victim models with adversarial training are more difficult to attack, which indicates that our generated adversaries have the potential to serve as supplementary corpora to enhance the robustness of victim models.

Table 14 displays adversarial training results of all datasets. Overall, after finetuned with both original training datasets and adversaries, victim model is more difficult to attack. Compared to original results, accuracy of all datasets is barely affected, while attack success rate meets an obvious decline. Meanwhile, attacking model with adversarial training leads to higher modification rate, further demonstrating adversarial training may help improve robustness of victim models.

Entropy threshold defense Yao et al. (2023) has been used to defend against the attack on LLMs recently. It employs the entropy of the first token prediction to refuse responding. Figure 6 demonstrates the probability of top-10 tokens in the first generated word of LLaMA. It can be observed that the raw inputs usually generates the first token with low entropy (i.e., the probability of argmax token is much higher, and the probability of other tokens is much lower). As shown in Figure 6, the adversarial samples from TF-Attack perform better than BERT-Attack with higher entropy. Attack samples generated through TF-Attack fare better against entropy-based filters compared to traditional text adversarial attack methods, indicating that the samples created by TF-Attack are harder to defend against.

Table 15 shows adversaries produced by TF-Attack and the baselines. Overall, the performance of TF-Attack is significantly better than other methods. For this sample from the MR dataset, only TextFooler and TF-Attack successfully mislead the victim model, i.e., changing the prediction from negative to positive. However, TextFooler modifies twice as many words as the TF-Attack, demonstrating our work has found a more suitable modification path. Adversaries generated by TextFooler and BERT-Attack are failed samples due to low semantic similarity. BERT-Attack even generates an invalid word "enamoted" due to its sub-word combination algorithm. We also ask crowd-workers to give a fluency evaluation.

Results show TF-Attack obtains the highest score of 4 as the original sentence, while other adversaries are considered difficult to understand, indicating TF-Attack can generate more natural sentences.