Sample Attackability in Natural Language Adversarial Attacks

With the emergence of the Transformer architecture (Vaswani et al., 2017), natural language processing (NLP) models have demonstrated impressive performance in many tasks, ranging from standard sentiment classification (Abdullah and Ahmet, 2022) to summarisation (Boorugu and Ramesh, 2020) and translation (Yang et al., 2020). However, Goodfellow et al. (2014) demonstrated that deep learning models are susceptible to adversarial attacks, where carefully crafted small imperceptible changes applied to original, natural inputs can cause models to mis-classify. In response, extensive efforts have explored methods to combat the threat of adversarial attacks by training with adversarial examples (Qian et al., 2022) or building separate detection systems (Harder et al., 2021; Raina and Gales, 2022). However, little or no work has sought to determine which input samples are the most susceptible to adversarial attacks. Are certain input samples easier to adversarially attack and if so can we efficiently identify these attackable samples? The ability to identify the attackable and in converse the robust samples has applications in a range of sample-selection tasks. For example, in the field of active learning (Sun and Wang, 2010), the query system can be designed to select the most attackable samples. Similarly, knowledge of sample attackability is useful for weighted adversarial training (Kim et al., 2021), where the aim is to augment the training set with only the most useful adversarial examples.

In the image domain, Raina and Gales (2023) formally define the notion of sample attackability as the minimum perturbation size required to change a sample’s output prediction from the target model. Running iterative adversarial attacks to determine this minimum perturbation size for a single sample is inefficient. Kim et al. (2021) use entropy (uncertainty) as a proxy function for sample attackability, but, Raina and Gales (2023) demonstrate that training a deep learning based classifier to predict the most attackable samples (and most robust samples) is the most effective method in the image domain. Therefore, this works extends the use of a deep learning based system to identify the most attackable and robust samples in NLP tasks. As a measure of a sample’s attackability, it is challenging to define a sample’s perturbation size for natural language. Following Raina and Gales (2023) in the image domain, this work uses the imperceptibility threshold in the definition of an adversarial attack as a measure of the perturbation size. To align with human perception, imperceptibility constraints for NLP aim to limit the semantic change in the text after an adversarial attack. These imperceptibility constraints can be grouped into two stages: 1) pre-transformation constraints (e.g. no stopword changes) that limit the set of acceptable adversarial examples; and 2) distance constraints that only allow for a subset of the acceptable adversarial examples, where the distance constraint explicitly restricts the distance moved by an adversarial example from the original example to satisfy a specified imperceptibility threshold. This distance can be measured for example using the Universal Sentence Encoder (Herel et al., 2022). A sample subject to a specific NLP attack method (with defined pre-transformation constraints) will have an associated set of acceptable adversarial examples. The attackability of the sample can thus be given by the smallest distance constraint imperceptibility threshold that at least one acceptable adversarial example in the set satisfies.

Default imperceptibility thresholds for the distance constraints proposed for NLP attack methods can often lead to unnatural adversarial examples (Morris et al., 2020). Hence, in this work, we use separate thresholds for defining attackable and robust samples. A sample’s minimum perturbation size is required to be within a much stricter imperceptibility threshold to be termed attackable, whilst in converse a sample’s minimum perturbation size has to be greater than a more generous imperceptibility threshold to be termed robust. The deep learning based attackability classifier proposed in Raina and Gales (2023) is successfully used to identify the attackable and robust samples for unseen data and unseen target models. However, in contrast to the image domain, it is found in NLP that the trained attackability detector fails to determine the attackable samples for different unseen NLP attack methods. This work extensively analyzes this observation and offers an explanation rooted in the inconsistency of imperceptibility definitions for different NLP attack methods.

In the image domain Zeng et al. (2020) introduce the notion of sample attackability through the language of vulnerability of a sample to an adversarial attack. This vulnerability is abstractly defined as the distance of a sample to a model’s decision boundary. Raina and Gales (2023) offer a more formal and extensive estimate of a sample’s vulnerability/attackability by considering the smallest perturbation size, aligned with an adversarial attack’s imperceptibility measure, to change a sample’s class prediction. Other research in the field of weighted adversarial training (Kim et al., 2021), has also implicitly considered the notion of sample attackability. The aim in weighted adversarial training is train with the more useful adversarial examples, which are arguably sourced from the more attackable original samples. For example Kim et al. (2021) use model entropy to estimate this attackability, whilst Zeng et al. (2020) use model confidence and Raina and Gales (2023) are successful in using a deep-learning based estimator of attackability. In the field of NLP, little work has explored weighted adversarial training. Xu et al. (2022) propose a meta-learning algorithm to lean the importance of each adversarial example, but this has no direct relation to a source sample’s attackability. Finally, in the field of active learning (Ren et al., 2020; Sun and Wang, 2010) there has also been implicit consideration of adversarial perturbation sizes as a measure of a sample’s value. The aim in active learning is to select the most useful subset of samples in a dataset to train a model on. In the image domain, Ducoffe and Precioso (2018) propose the use of the smallest adversarial perturbation size for each sample to measure the distance to the decision boundary. However, there is no explicit consideration of sample attackability or design of an efficient method to identify the attackable samples.

In both the image and NLP domain, an untargeted adversarial attack is able to fool a classification system, $\mathcal{F}()$, by perturbing an input sample, $\mathbf{x}$ to generate an adversarial example $\tilde{\mathbf{x}}$ to cause a change in the output class,

$$\mathcal{F}(\mathbf{x})\neq\mathcal{F}(\mathbf{\tilde{x}}).$$

(1)

It is necessary for adversarial attacks to be imperceptible, such that adversarial examples, $\mathbf{\tilde{x}}$ are not easily detectable/noticeable by humans. It is inefficient and expensive to rely on manual human measures of attack imperceptibility, so instead proxy measures are used to enforce imperceptibility of an adversarial attack. For images, the $l_{p}$ norm is considered a good proxy for human perception of imperceptibility. However, in NLP it is more challenging to ensure imperceptibility. Despite earlier research introducing only visual constraints on the adversarial attacks (Goyal et al., 2023; Gao et al., 2018; Ebrahimi et al., 2017; Pruthi et al., 2019; Tan et al., 2020; Li et al., 2018), e.g. number of words changed as per the Levenshtein distance, recent research considers more sophisticated measures seeking to measure the semantic change in text sequences (Li et al., 2020a; Jin et al., 2019; Ren et al., 2019; Wang et al., 2019; Garg and Ramakrishnan, 2020; Alzantot et al., 2018; Li et al., 2020b). In general, modern NLP imperceptibility constraints can be separated into two stages: pre-transformation constraints and distance constraints. Pre-transformation constraints typically limit the attack mechanism to encourage little change in semantic content. For example, stop-word transformations will be prevented or any word substitutions will be restricted to appropriate synonyms. A collection of pre-transformation constraints, as specified by a particular attack method, limit the available set, $\mathcal{A}$ of possible adversarial examples that can be considered for a specific sample, $\mathbf{x}$, such that

$$\mathbf{\tilde{x}}\in\mathcal{A}.$$

(2)

The distance-based constraints are further constraints that explicitly aim to limit the distance between the original sample $\mathbf{x}$ and the adversarial example, $\mathbf{\tilde{x}}$ to ensure a small perceived semantic change. This distance can be measured via a proxy function, $\mathcal{G}$,

$$\mathcal{G}(\mathbf{x},\mathbf{\tilde{x}})\leq\epsilon,$$

(3)

where $\epsilon$ represents the maximum imperceptibility threshold. A popular example of such a distance constraint is a limit on the cosine-distance in a sentence embedding space, e.g.,

$$\mathcal{G}(\mathbf{x},\mathbf{\tilde{x}})=1-\mathbf{h}^{T}{\mathbf{\tilde{h}}},$$

(4)

where $\mathbf{h}$ and $\mathbf{\tilde{h}}$ are the normalized vector embedding representations of the word sequences $\mathbf{x}$ and $\mathbf{\tilde{x}}$.

Sample attackability is concerned with how easy it is to adversarially attack a specific sample. The notion of sample attackability is formally introduced by Raina and Gales (2023), where a specific input sample, $\mathbf{x}_{n}$’s attackability for a specific model, $\mathcal{F}_{k}$ is given by the theoretical minimum perturbation size, $\hat{{\delta}}_{n}^{(k)}$ within which a sample can be successfully attacked. However, it is not simple to define the perturbation size for an adversarial attack in NLP. The simplest definition for the perturbation size, ${\delta}$, for a specific attack method with a specific set of acceptable adversarial examples, $\mathcal{A}$ (Equation 2), is to use the distance-based proxy function, $\mathcal{G}$ (Equation 3), such that ${\delta}=\mathcal{G}(\mathbf{x},\mathbf{\tilde{x}})$. Then the minimum perturbation size, $\hat{{\delta}}_{n}^{(k)}$ for sample $n$ and model $k$ is,

$$\hat{{\delta}}_{n}^{(k)}=\min_{\begin{subarray}{c}\mathbf{x}\in\mathcal{A},\\ \mathcal{F}_{k}(\mathbf{x}_{n})\neq\mathcal{F}_{k}(\mathbf{x})\end{subarray}}\left\{\mathcal{G}(\mathbf{x}_{n},\mathbf{x})\right\}.$$

(5)

We aim to use a sample’s minimum perturbation size to classify it as attackable, robust or neither. Default distance-based imperceptibility constraints defined using $\mathcal{G}$ for various NLP attack methods can lead to unnatural adversarial examples and so we use separate and stricter thresholds for classifying samples as attackable or robust. Hence, as in Raina and Gales (2023), we define sample $n$ as attackable for model $k$ if the smallest adversarial perturbation is less than a strict threshold, $\mathbf{A}_{n,k}=({\hat{\delta}}^{(k)}_{n}<\epsilon_{a})$, where any sample that is not attackable can be denoted as $\bar{\mathbf{A}}_{n,k}$. Conversely, a sample is defined as robust, if its adversarial perturbation size is larger than a separate, but more generous (larger) set threshold, $\mathbf{R}_{n,k}=(\hat{\delta}^{(k)}_{n}>\epsilon_{r})$.

It is informative to identify samples that are universally attackable/robust across different models. We can thus extend the definition for universality as follows. A sample, $n$, is universally attackable if,

$$\mathbf{A}_{n}^{(\mathcal{M})}=\bigcap_{k,\mathcal{F}_{k}\in\mathcal{M}}\hskip 5.0pt\mathbf{A}_{n,k},$$

(6)

where $\mathcal{M}$ is the set of models in consideration. Similarly a sample is universally robust if, $\mathbf{R}_{n}^{(\mathcal{M})}=\bigcap_{k,\mathcal{F}_{k}\in\mathcal{M}}\hskip 5.0pt\mathbf{R}_{n,k}$. Note that all of the attackability definitions in this section are for a specific attack method (e.g. Textfooler), as definition of the perturbation size in Equation 5 uses the distance-based imperceptibility constraint, $\mathcal{G}$ specific to an attack method. Portability of these definitions and attackability detection models across attack methods is explored in Section 6.3.

The definition of attackable and robust samples uses the minimum perturbation size (as per a distance-based constraint) for an NLP adversarial attack on a sample. When trying to determine which samples are attackable, it is slow and expensive to run an adversarial attack iteratively to find the minimum perturbation size. Further, often one may not have access to an unseen target model, $\mathcal{F}_{t}$ or even the target sample, $n$ to perform an adversarial attack upon. Hence, in this setting, it is necessary to have a simple and efficient process that can determine whether samples in an unseen dataset are attackable for an unseen target model. Inspired by Raina and Gales (2023), this section describes a method to train a simple deep-learning attackability detector to identify the attackable and robust samples in an unseen dataset, for an unseen target model, $\mathcal{F}_{t}$. We give the deep-learning attackability detector access to a seen dataset, $\{\mathbf{x}_{n},y_{n}\}_{n=1}^{N}$ and a set of seen models, $\mathcal{M}=\{\mathcal{F}_{1},\ldots,\mathcal{F}_{|\mathcal{M}|}\}$, such that $\mathcal{F}_{t}\notin\mathcal{M}$. Each model can be represented as an encoder embedding stage, followed by a classification stage,

$$\mathcal{F}_{k}(\mathbf{x}_{n})=\mathcal{F}_{k}^{(\texttt{cl})}(\mathbf{h}_{n,k}),$$

(7)

where $\mathbf{h}_{n,k}$ is the model encoder’s embedding of $\mathbf{x}_{n}$. For each seen model in $\mathcal{M}$, a separate attackability detector can be trained. For a specific seen model, $k$, we can measure the attackability of each sample using the minimum perturbation size (Equation 5), $\{{\hat{\delta}}^{(k)}_{n}\}_{n=1}^{N}$. It is most efficient to exploit the encoder embedding representation of input text sequences, $\mathbf{h}_{n,k}$, already learnt by each model. Hence, each deep attackability detector, $\mathcal{D}_{\theta}^{(k)}$, with parameters $\theta$, can be trained as a binary classification task to determine the probability of a sample being attackable for model $k$, using the encoder embedding at the input,

$$p(\mathbf{A}_{n,k})=\mathcal{D}_{\theta}^{(k)}(\mathbf{h}_{n,k}).$$

(8)

Consistent with Raina and Gales (2023), we use a simple, single hidden-layer fully connected network architecture for each attackability detector, $\mathcal{D}$, such that,

$$\mathcal{D}_{\theta}(\mathbf{h})=\sigma(\mathbf{W}_{1}\sigma(\mathbf{W}_{0}\mathbf{h})),$$

(9)

where $\mathbf{W}_{0}$ and $\mathbf{W_{1}}$ are the trainable parameters and $\sigma()$ is a standard sigmoid function. This collection of model-specific detectors can be used to estimate the probability of a new sample being attackable for an unseen target model, $\mathcal{F}_{t}$. It is most intuitive to take an expectation over the seen model-specific detector attackability probabilities,

$$p(\mathbf{A}_{n,t})\approx\frac{1}{|\mathcal{M}|}\sum_{k,\mathcal{F}_{k}\in\mathcal{M}}p(\mathbf{A}_{n,k}).$$

(10)

Raina and Gales (2023) demonstrated that this estimate in the image domain does not capture the samples that are attackable specifically for the target model, $\mathcal{F}_{t}$’s specific realisation. Therefore, we seek instead to estimate the probability of a universally attackable sample (defined in Equation 6),

$$p(\mathbf{A}^{(\mathcal{M}+t)}_{n})\approx\left[\frac{1}{|\mathcal{M}|}\sum_{k,\mathcal{F}_{k}\in\mathcal{M}}p(\mathbf{A}_{n,k})\right]^{\alpha(\mathcal{M})},$$

(11)

where the parameter $\alpha(\mathcal{M})$ models the idea that the probability of sample being universally attackable should decrease with the number of models (note that this is empirically observed in Figure 1). An identical approach can be used to train detectors to give the probability of a sample being universally robust, $p(\mathbf{R}^{(\mathcal{M}+t)}_{n})$.

Given that the deep learning based attackability detectors are trained to identify universally attackable samples (Equation 11), they are expected to perform best in the uni evaluation setting.

The corpus-level performance of an attackability detector for an unseen dataset can be reported using precision and recall. A selected threshold, $\beta$, is used to class the output of detectors, e.g. $p(\mathbf{A}^{(\mathcal{M}+t)}_{n})>\beta$ classes sample $n$ as attackable. The precision is $\texttt{prec}=\text{TP}/\text{TP+FP}$ and recall is $\texttt{rec}=\text{TP}/\text{TP+FN}$, where FP, TP and FN are standard counts for False-Positive, True-Positive and False-Negative. An overall score is given with the F1-score, $\text{F1}=\text{2}*(\texttt{prec}*\texttt{rec})/(\texttt{prec}+\texttt{rec})$. By sweeping the threshold $\beta$ a full precision-recall curve can be generated and typically the threshold with the greatest F1-score is selected as an appropriate operating point.

Little research has sought to determine the level of vulnerability of individual samples to an adversarial attack in natural language processing (NLP) tasks. This work formally extends the definitions of sample attackability to the field of NLP. It is demonstrated that uncertainty-based approaches are insufficient in characterising the most attackable and the most robust samples in a dataset. Instead, a deep-learning based detector can be used to effectively to identify these attackable/robust samples for an unseen dataset and more powerfully for an unseen target model. However, it is also observed that different attack methods in natural language have a different set of imperceptibility constraints, leading to a lack of consistency in determining sample attackability across different attack methods. As a consequence, the success of a deep-learning based attackability detector is limited to the attack method it is trained with.

This work introduced a powerful attackability detector but also demonstrated that its success is limited to a matched setting, where the same attack method is used in both training and evaluation of the detector. A second limitation with this work is that all experiments were carried out on natural language classification tasks. It would be useful in the future to extend these experiments to sequence-to-sequence tasks to have a more comprehensive set of results.

Adversarial attacks by nature can be of ethical concern, as malicious users can exploit theoretical adversarial attack literature to develop harmful tools to mis-use deployed deep learning systems. However this work does not aim to propose any new adversarial attack techniques, but instead considers a method to identify the most vulnerable/attackable samples. Hence, there is no perceived ethical concern related to this specific piece of work.

This paper reports on research supported by Cambridge University Press & Assessment (CUP&A), a department of The Chancellor, Masters, and Scholars of the University of Cambridge.