Elephant in the Room: An Evaluation Framework for
Assessing Adversarial Examples in NLP

There are a number of off-the-shelf neural models for sentiment classification Kim (2014); Wang et al. (2016), most of which are based on long-short term memory networks (LSTM; Hochreiter and Schmidhuber (1997)) or convolutional neural networks (CNN; Kim (2014)). In this paper, we pre-train three sentiment classifiers: BiLSTM, BiLSTM$+$A, and CNN. These classifiers are targeted by different attacking methods to generate adversarial examples (detailed in Section 3.2). BiLSTM is composed of an embedding layer that maps individual words to pre-trained word embeddings; a number of bi-directional LSTMs that capture sequential contexts; and an output layer that maps the averaged LSTM hidden states to a binary output. BiLSTM$+$A is similar to BiLSTM except it has an extra self-attention layer which learns to attend to salient words for sentiment classification, and we compute the weighted mean of the LSTM hidden states prior to the output layer. Manual inspection of the attention weights show that polarity words such as awesome and disappointed are assigned with higher weights. Finally, CNN has a number of convolutional filters of varying sizes, and their outputs are concatenated, pooled and fed to a fully-connected layer followed by a binary output layer.

Recent development in transformer-based pre-trained models have produced state-of-the-art performance on a range of NLP tasks Devlin et al. (2018); Yang et al. (2019). To validate the transferability of the attacking methods, we also test it on a fine-tuned BERT classifier. That is, we use the adversarial examples generated for attacking the three previous classifiers (BiLSTM, BiLSTM$+$A or CNN) as test data for BERT and measure its classification performance to understand whether these adversarial examples can fool BERT.

We experiment with six benchmark attacking methods for texts, ranging from white-box attacks: FGM, FGVM, DeepFool Gong et al. (2018), HotFlip Ebrahimi et al. (2017)), and TYC Tsai et al. (2019) to black-box attacks: TextFooler Jin et al. (2019).

To perturb the discrete inputs, both FGM and FGVM introduce noises in the word embedding space via the fast gradient method Goodfellow et al. (2014) and reconstruct the input by mapping perturbed word embeddings to valid words via nearest neighbour search. Between FGM and FGVM, the former introduce noises that is proportional to the sign of the gradients while the latter introduce perturbations proportional to the gradients directly. The proportion is known as the overshoot value and denoted by $\epsilon$. DeepFool uses the same trick to deal with discrete inputs except that, instead of using the fast gradient method, it uses the method introduced in Moosavi-Dezfooli et al. (2016) for image to search for an optimal direction to perturb the word embeddings.

Unlike the previous three methods, HotFlip and TYC rely on performing one or more atomic flip operations to replace words while monitoring the label change given by the target classifier. In HotFlip, the directional derivatives w.r.t.  flip operations are calculated and the flip operation that results in the largest increase in loss is selected.³³3While the original paper explores both character flips and word flips, we test only word flips here. The rationale is that introducing character flips to word-based target classifiers is essentially changing word tokens to [unk], which creates a confound for our experiments. TYC is similar to FGM, FGVM and DeepFool in that it also uses nearest neighbour search to map the perturbed embeddings to valid words, but instead of using the perturbed tokens directly, it uses greedy search or beam search to flip original tokens to perturbed ones one-at-a-time in order of their vulnerability.

TextFooler, the only black-box attack method tested, is a query-based method. Since it assumes no access to the full architecture of the target classifier, it learns the order of vulnerability of tokens in an input sentence according to the change of prediction scores produced by the target classifier when a specific token is discarded. Once the order of vulnerability of words is identified, similar to HotFlip and TYC, it greedily replaces tokens in the order of vulnerability one-at-a-time until the prediction of the target classifier changes. To ensure similarity between the adversarial examples and the original ones, the substituted tokens are selected to satisfy semantic, part-of-speech and sentence embedding similarity constraints. Note that TextFooler uses the classifier’s prediction scores to learn token vulnerability; in a more realistic black-box scenario the classifier may only reveal the predicted labels (without showing the underlying scores associated with each label).

We construct three datasets based on the Yelp reviews⁴⁴4https://www.yelp.com/dataset and the sentence-level Rotten Tomato (RT) movie reviews⁵⁵5http://www.cs.cornell.edu/people/pabo/movie-review-data/. For Yelp, we binarise the ratings⁶⁶6Ratings$\geq$4 is set as positive and ratings$\leq$2 as negative., and create 2 datasets, where we keep only reviews with $\leq$ 50 tokens (yelp50) and $\leq$200 tokens (yelp200). For RT (rt), we directly use the binarised dataset which contains 5331 positive and 5331 negative tokenized sentences. We randomly partition both datasets into train/dev/test sets (90/5/5 for yelp50; 99/0.5/0.5 for yelp200; and 80/10/10 for rt). For yelp50 and yelp200, we use spaCy⁷⁷7https://spacy.io for tokenisation. We train and tune target classifiers (see Section 3.1) using the training and development sets, and evaluate their performance on the original examples in the test sets as well as the adversarial examples generated by attacking methods for the test sets. These datasets present a variation in the text length (e.g. the average number of words for rt, yelp50 and yelp200 is 22, 34 and 82 words respectively) and training data size (e.g. rt: 8K examples, yelp50: 407K examples, and yelp200: 2M examples).

We use the pre-trained glove.840B.300d embeddings Pennington et al. (2014) for the first 5 attacking methods and the counter-fitted word embedding Mrkšić et al. (2016) for TextFooler. For FGM, FGVM and DeepFool, we tune $\epsilon$, the overshoot hyper-parameter (Section 3.2) and keep the iterative step $n$ static (5).⁸⁸8We search for the best $\epsilon$ within a large range (orders of magnitude in difference) to achieve a particular attacking performance. For TYC, besides $\epsilon$ we also tune the upper limit of flipped words, ranging from 10%–100% of the maximum length. For HotFlip and TextFooler, we tune only the upper limit of flipped words, in the range of $[1,7]$.

For target classifiers, we tune batch size, learning rate, number of layers, number of units, attention size (for BiLSTM$+$A), filter sizes and dropout probability (for CNN). For BERT, we use the default fine-tuning hyper-parameter values except for batch size, where we adjust based on memory consumption. Note that after the target classifiers are trained their weights are not updated when running the attacking methods.

As sentiment classification is our target task, we use the standard classification accuracy (ACC, the lower the better) to evaluate the attacking performance of adversarial examples (criterion (a)).

To assess the similarity between the original and (transformed) adversarial examples (criterion (b)), we compute BLEU scores Papineni et al. (2002) to measure word overlap; and SEM scores, cosine similarity between the representations of original examples and adversarial examples generated by the universal sentence encoder Cer et al. (2018), to measure semantic similarity. For both metrics, higher scores represent better performance.

To measure fluency, we first explore a supervised BERT model fine-tuned to predict linguistic acceptability Devlin et al. (2018). However, in preliminary experiments we found that BERT performs very poorly at predicting the acceptability of adversarial examples (e.g. it predicts word-salad-like sentences generated by FGVM as very acceptable), revealing the brittleness of these supervised models. We next explore unsupervised approaches Lau et al. (2017, 2020), using normalised sentence probabilities estimated by pre-trained language models for measuring acceptability. Following Lau et al. (2020), we use XLNet Yang et al. (2019) as the language model. The acceptability score of a sentence is calculated based on the normalised sentence probability: ${\log P(s)}/({((5+|s|)/(5+1))^{\alpha}})$, where $s$ is the sentence, and $\alpha$ is a hyper-parameter (set to 0.8) to dampen the impact of large values Vaswani et al. (2017). To measure how fluency differs between an adversarial example and the original sentence, we compute the difference in their acceptability scores, giving us the acceptability metric ACPT.

Note that we only compute BLEU and ACPT for adversarial examples that have successfully fooled the classifier. Our rationale is that unsuccessful examples can artificially boost these scores by not making any modifications, and so the better approach is to only consider successful examples.

We present the performance of the attacking methods against 3 target classifiers (Table 1A; top) and on 3 datasets (Table 1B; bottom). We choose 3 ACC thresholds for the attacking performance: T0, T1 and T2, which correspond approximately to accuracy scores of 90%, 80% and 70% for the Yelp datasets (yelp50, yelp200)⁹⁹9Most methods are capable of achieving attacking performance of 30% or lower (ACC), although that comes with severe degradation to the quality of the adversarial examples.; and 60%, 50%, 30% for RT dataset (rt) ¹⁰¹⁰10We choose 30% instead of 40% because, for HotFlip, flipping one words achieved an accuracy drop to 30.1%. Our rationale is that each method should be compared on the same basis if our focus is to to provide a fair assessment on the quality of the adversarial examples, and the attacking performance constitutes a reasonable basis.

Looking at BLEU, SEM and ACPT, TextFooler is the most consistent method over multiple datasets and classifiers. HotFlip is also fairly competitive, occasionally producing better BLEU scores (CNN at T1; at T0, T1, T2 on yelp200 and T2 on rt). Gradient-based methods FGM and FGVM perform very poorly. In general, they tend to produce word salad adversarial examples, as indicated by their poor ACPT scores. DeepFool similarly generates incoherent sentences with low BLEU scores, but occasionally produces good SEM (at T0) and ACPT (BiLSTM at T1 and T2), suggesting potential brittleness of the automatic evaluation approach for evaluating semantic similarity and acceptability.

Comparing the performance across different ACC thresholds, we observe a consistent pattern of decreasing performance over all metrics as the attacking performance increases from T0 to T2, especially drastic for FGM, FGVM and DeepFool when the attacking rate changes from T0 to T1. These observations suggest that all methods are trading off fluency and content preservation as they attempt to generate stronger adversarial examples.

We now focus on Table 1A to understand the impact of model architecture for the target classifier. With 1 word flip as the upper limit for HotFlip, the accuracy of BiLSTM$+$A and BiLSTM drops to T0 (approximately 4% accuracy decrease) while the accuracy of CNN drops to T1 (approximately 13% accuracy decrease). Within the same attacking performance thresholds, FGM, FGVM and DeepFool achieve higher BLEU, SEM and ACPT scores when targeting CNN compared to those scores when targeting the other two models. These observations suggest that convolutional networks are more vulnerable to attacks (noting that it is the predominant architecture for CV). Interestingly, the CNN model seems to be very robust against TextFooler on yelp50, where the accuracy stays at around 87.0% regardless of how we tune TextFooler.

Looking at Table 1B, we also find that the attacking performance is influenced by the input text length and the number of training examples for target classifiers. For HotFlip and TextFooler, we see improvements over BLEU, SEM and ACPT as text length increases from yelp50 to yelp200, indicating the performance of these two methods is more affected by input lengths. We think this is because with more words it is more likely for them to find a vulnerable spot to target. While for TYC, we see improvements over BLEU and ACPT as the number of training examples for target classifier decreases from yelp200 (2M) to yelp50 (407K), indicating TYC are less effective for attacking target classifiers that are trained with more data. This suggests that increasing the training data for a classifier could potentially improve its robustness against certain attacks. The effect of the number of training examples for target classifier is further validated by the attacking performance of TextFooler and HotFlip on rt. Despite that rt has the shortest input (with average 22 tokens), changing 1 words for TextFooler and HotFlip successfully rendered accuracy drops from 78.8% to 49.9% and 30.1%, respectively. Comparing the same number of word change, the drop of accuracy introduced by TextFooler and HotFlip are 10% and 4% on yelp50; and 4% and 7% on yelp200.

Another factor that possibly makes it easier to attack rt is that movie reviews being more descriptive and therefore creating potential ambiguity in their expression of sentiment. In comparison, the restaurant reviews are more straightforward, using polarising words such as awesome or awful. The net effect is that sentiment classification is “easier” for the restaurant reviews (as the classifier can make the decision based on vocabulary choices), which in turn make adversarial attacks “harder”.

To check how well these adversarial examples generalise to fooling other classifiers, also known as transferability, we feed the adversarial examples from the 3 best methods, i.e. TYC, HotFlip and TextFooler, to a pre-trained BERT trained for sentiment classification and measure its accuracy (Figure 1). Unsurprisingly, we observe that the attacking performance (i.e. the drop in ACC) is not as good as those reported in Table 1 . Interestingly, we find that TextFooler, the best performing method, produces the least effective adversarial examples for BERT, indicating poor transferability, while examples generated from TYC perform the best in fooling BERT. Manually inspecting the examples generated by TextFooler, we notice it tries to replace as few words as possible to move an examples just across the decision boundary of the target classifier (indicated by very close scores between different labels). This change appears be very targeted for a specific classifier, and as such is unlikely to fool other classifiers.

As an additional insight, we also evaluated the computational time of different attacking methods. We calculate the the generation time of the three best performing methods TYC, HotFlip and TextFooler when they attack the test dataset of rt, yelp50, and yelp200 at T2 attacking threshold. Table 2 shows the corresponding computational time (seconds per example). We found that HotFlip achieves comparable performance with TextFooler but only consumes 1/3 of its computational time. TYC is even more time-consuming than TextFooler. Also, the speed of HotFlip seems not affected by the increase of the sentence lengths, but TextFooler and TYC take much longer time to attack longer input examples.

To summarise, our results demonstrate that the best attacking methods (e.g. TextFooler and HotFlip) may not produce adversarial examples that generalise to fooling other classifiers. We also saw that convolutional networks are generally more vulnerable than recurrent networks, and that dataset features such as text length and training data size can influence the performance of adversarial attacks.

Automatic metrics provide a proxy to quantify the quality of the adversarial examples. To validate that these metrics work, we conduct a crowdsourcing experiment on Figure-Eight.¹¹¹¹11https://www.figure-eight.com/ Recall that the automatic metrics do not assess sentiment preservation (criterion (d)); we evaluate that aspect here.

We experiment with the 3 best methods (TYC HotFlip and TextFooler) on 2 accuracy thresholds (T0 and T2), using BiLSTM$+$A as the classifier. For each method and threshold, we randomly sample 25 positive-to-negative and 25 negative-to-positive examples. To control for quality, we reserve and annotate 10% of the samples ourselves as control questions. Workers are first presented with 10 control questions as a quiz, and only those who pass the quiz with at least 80% accuracy can continue to work on the task. We display 10 questions per page, where one control question is embedded to continue monitor worker performance.¹²¹²12Workers are required to maintain their performance (80% accuracy) throughout the annotation process. The task is designed such that each control question can only be seen once per worker. We restrict our jobs to workers in United States, United Kingdoms, Australia, and Canada.

To evaluate the criteria (b) textual similarity, (c) fluency, and (d) sentiment preservation, we ask the annotators three questions:

1. 1.

   Is snippet B a good paraphrase of snippet A?
      $\ocircle$ Yes    $\ocircle$ Somewhat yes    $\ocircle$ No

2. 2.

   How natural does the text read?
      $\ocircle$ Very unnatural    $\ocircle$ Somewhat natural    $\ocircle$ Natural

3. 3.

   What is the sentiment of the text?
      $\ocircle$ Positive    $\ocircle$ Negative    $\ocircle$ Cannot tell

For question 1, we display both the adversarial input and the original input, while for question 2 and 3 we present only the adversarial example. As a baseline, we also run a survey on question 2 and 3 for 50 random original (unperturbed) samples.

We present the percentage of answers to each question in Figure 2. The green bars illustrate how well the adversarial examples paraphrase the original ones; blue how natural the adversarial examples read; and red whether the sentiment of the adversarial examples is consistent compared to the original.

Looking at the performance of the original sentences (“(a) Original samples”), we see that their language is largely fluent and their sentiment is generally consistent to the original examples’.

On content preservation (criterion (b); green bars), all methods produce poor paraphrases on yelp50 except for TextFooler.

Next we look at fluency (criterion (c); blue bars). We see similar trend: with the increased attacking performance, the readability of adversarial examples generated by different attacking methods is getting poorer. HotFlip is fairly competitive, producing adversarial examples that are only marginally less fluent compared to the original at T0. At T2, however, it begin to trade off fluency. TextFooler, on the other hand, achieved even slightly better fluency than the original examples at T2, indicating the substituted tokens fit the context very well.

Lastly, we consider sentiment preservation (criterion (d); red bars). All methods trade off sentiment preservation to achieve better attacking performance. Again both HotFlip and TextFooler are the better methods here (interestingly, we observe an increase in agreement as their attacking performance increases from T0 and T2).

Comparing automatic evaluation and human evaluation results, we found that the SEM scores are generally consistent with the human evaluation results on semantic preservation, while the BLEU score is less effective as evidenced by the high BLEU score of HotFlip on yelp50 and poor paraphrasing performance in the same settings. The ACPT metric appears to be a solid metric in evaluating fluency, as we see a good agreement with human evaluation across the three attacking methods.

Summarising our findings, TextFooler is generally the best method across all criteria, noting that its adversarial examples, however, have the poorest transferability. HotFlip produces comparable results with TextFooler for meeting the four criteria and similarly suffer from poor transferability. TYC generates adversarial examples with better transferability but do not do well in terms of content preservation and fluency. In terms of computational time, HotFlip is the most efficient, consuming less than 1/3 of the time consumed by the other two methods. The difference is more profound for longer input sentences. FGM, FGVM and DeepFool perform very poor as they largely sacrifice example qualities to achieve attacking performance, indicating directly mapping from perturbed word embedding is not applicable in NLP. All said, we found that all methods tend to trade-off sentiment preservation for attacking performance, revealing that these methods in a way “cheat” by simply flipping the sentiments of the original sentences to fool the classifier, and therefore the adversarial examples might be ineffective for adversarial training, as they are not examples that reveal potential vulnerabilities in the classifier.