Attention Meets Perturbations:
Robust and Interpretable Attention
with Adversarial Training

Attention mechanisms were introduced by Bahdanau et al. [1] for the task of machine translation. Today, these mechanisms contribute to improving the prediction performance of various tasks in the NLP field, such as sentence-level classification [2], sentiment analysis [3], question answering [4], and natural language inference [5]. There are a wide variety of attention mechanisms; for instance, additive [1] and scaled dot-product [8] functions are used as similarity functions.

Attention weights are often claimed to offer insights into the inner workings of DNNs [22]. However, Jain and Wallace [10] reported that learned attention weights are often uncorrelated with the word importance calculated through the gradient-based method [26], and perturbations interfere with interpretation. In this paper, we demonstrate that AT for attention mechanisms can mitigate these issues.

AT [9, 15, 27] is a powerful regularization technique that has been primarily explored in the field of image recognition to improve the robustness of models for input perturbations. In the NLP field, AT has been applied to various tasks by extending the concept of adversarial perturbations, e.g., text classification [18, 19], part-of-speech tagging [20], and machine reading comprehension [21, 28]. As mentioned earlier, these techniques apply AT for word embedding. Other AT-based techniques for the NLP tasks include those related to parameter updating [29] and generative adversarial network (GAN)-based retrieval-enhancement method [30]. Our proposal is an adversarial training technique for attention mechanisms and is different from these methods.

Miyato et al. [18] proposed Word AT, a technique that applied AT to the word embeddings. The adversarial perturbations are generated according to the back-propagation gradients. These perturbations are expected to regularize the model. Since then, Sato et al. [19] proposed Word iAT, and it has been known to achieve almost the same performance as Word AT that does not expect interpretability [19]. The Word iAT technique aims to increase the model’s interpretability by determining the perturbation’s direction so that it is closer to other word embeddings in the vocabulary. Both reports demonstrated improved task performance via AT. However, the specific effect of AT on attention mechanisms has yet to be investigated. In this paper, we aim to address this issue by providing analyses of the effects of AT for attention mechanisms using various NLP tasks.

AT is considered to be related to other regularization techniques (e.g., dropout [31], batch normalization [32]). Specifically, dropout can be considered a kind of noise addition. Word dropout [33] and character dropout [34], known as wildcard training, are variants for NLP tasks. These techniques can be considered random noise for the target task. In contrast, AT has been demonstrated to be effective because it creates particularly vulnerable perturbations that the model is trained to overcome [15].

It has been reported that DNN models that introduce adversarial training to overcome adversarial perturbations capture human-like features [23, 24, 25]. These features help to make the prediction of DNN models easier to interpret for humans. In this paper, we demonstrate that the proposed AT to attention mechanisms provides cleaner attention that is more easily interpreted by humans.

For a single sequence task, such as the BC task, the input is a word sequence of one-hot encoding $X_{S}=(\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{T_{S}})\in\mathbb{R}^{T_{S}\times|V|}$, where $T_{S}$ and $|V|$ are the number of words in the sentence and vocabulary size. We introduce the following short notation for the sequence $(\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{T_{S}})$ as $(\bm{x}_{t})_{t=1}^{T_{S}}$. Let $\bm{w}_{t}\in\mathbb{R}^{d}$ be a $d$-dimensional word embedding that corresponds to $\bm{x}_{t}$. We represent each word with the word embeddings to obtain $(\bm{w}_{t})_{t=1}^{T_{S}}\in\mathbb{R}^{T_{S}\times d}$. Next, we use the BiRNN encoder $\mathbf{Enc}$ to obtain $m$-dimensional hidden states $\bm{h}_{t}$:

where $\bm{h}_{0}$ is the initial hidden state and is regarded as a zero vector. Next, we use the additive formulation of attention mechanisms proposed by Bahdanau et al. [1] to compute the attention score for the $t$-th word $\tilde{a}_{t}$, defined as:

where $\bm{W}\in\mathbb{R}^{d^{\prime}\times m}$ and $\bm{b},\bm{c}\in\mathbb{R}^{d^{\prime}}$ are the parameters of the model. Then, from the attention scores $\tilde{\bm{a}}=(\tilde{a}_{t})_{t=1}^{T_{S}}$, the attention weights $\bm{a}=(a_{t})_{t=1}^{T_{S}}$ for all words are computed as

The weighted instance representation $\bm{h}_{\bm{a}}$ is calculated as

Finally, $\bm{h}_{\bm{a}}$ is fed to a dense layer Dec, and the output activation function is then used to obtain the predictions:

where $\sigma$ is a sigmoid function, and $|\bm{y}|$ is the label set size.

For a pair sequence task, such as the QA and NLI tasks, the input is $X_{P}=(\bm{x}_{t}^{(p)})_{t=1}^{T_{P}}\in\mathbb{R}^{T_{P}\times|V|}$ and $X_{Q}=(\bm{x}_{t}^{(q)})_{t=1}^{T_{Q}}\in\mathbb{R}^{T_{Q}\times|V|}$. $T_{P}$ and $T_{Q}$ are the number of words in each sentence. $X_{P}$ and $X_{Q}$ represent the paragraph and question in the QA and the hypothesis and premise in the NLI. We used two separate BiRNN encoders to obtain the hidden states $\bm{h}_{t}^{(p)}\in\mathbb{R}^{m}$ and $\bm{h}_{t}^{(q)}\in\mathbb{R}^{m}$:

where $\bm{h}_{0}^{(p)}$ and $\bm{h}_{0}^{(q)}$ are the initial hidden states and are regarded as zero vectors. Next, we computed the attention weight $\tilde{a}_{t}$ of each word of $X_{P}$ as:

where $\bm{W}_{1}\in\mathbb{R}^{{}^{\prime}d\times m}$ and $\bm{W}_{2}\in\mathbb{R}^{d^{\prime}\times m}$ denote the projection matrices, and $\bm{b},\bm{c}\in\mathbb{R}^{d^{\prime}}$ are the parameter vectors. Similar to Eq. 3, the attention weight $a_{t}$ can be calculated from $\tilde{a}_{t}$. The presentation is obtained from a sum of words in $X_{P}$.

is fed to a Dec, and then a softmax function is used as $\sigma$ to obtain the prediction (in the same manner as in Eq. 5).

Let $X_{\tilde{\bm{a}}}$ be an input sequence with attention score $\tilde{\bm{a}}$, where $\tilde{\bm{a}}$ is a concatenated attention score for all $t$. We model the conditional probability of the class $\bm{y}$ as $p(\bm{y}|X_{\tilde{\bm{a}}};\bm{\theta})$, where $\bm{\theta}$ represents all model parameters. For training the model, we minimize the following negative log likelihood as a loss function with respect to the model parameters:

We describe the proposed Attention AT, which features adversarial perturbations in the attention mechanisms rather than in the word embeddings [18, 19]. The adversarial perturbation on the mechanisms is defined as the worst-case perturbation on attention mechanisms of a small bounded norm $\epsilon$ that maximizes loss function $\mathcal{L}$ of the current model:

where $X_{\tilde{\bm{a}}+\bm{r}}$ is the input sequence with attention score $\tilde{\bm{a}}$, its perturbation $\bm{r}$, $\bm{y}$ is the target output, and $\hat{\bm{\theta}}$ represents the current model parameters. We apply the fast gradient method [11, 18], i.e., first-order approximation to obtain an approximate worst-case perturbation of norm $\epsilon$, through a single gradient computation as follows:

$\epsilon$ is a hyper-parameter to be determined using the validation dataset. We find this $\bm{r}_{\tt AT}$ against the current model parameterized by $\hat{\bm{\theta}}$ at each training step and construct an adversarial perturbation for attention score $\tilde{\bm{a}}$:

We describe the proposed Attention iAT for further boosting the prediction performance and the interpretability of NLP tasks. Rather than utilizing AT to attention mechanisms (as described in Section IV-A), Attention iAT effectively exploits differences in the attention to each word in a sentence for the training. As a result, this technique provides cleaner attention in the sentence and improves the interpretability of the attention. These effects contribute to improving the performance of various NLP tasks.

In terms of formulation, the proposed Attention iAT is analogous to interpretable AT for word embeddings (Word iAT) [19], which increases the interpretability of AT for word embeddings in formulas. However, the implications and effects for training the model are very different; in the proposed Attention iAT, the attention difference enhancement, described later, enhances the difference in attention for each word. The difference and its effect will be explained later in this section and discussed in Section VII-C.

Suppose $\tilde{a}_{t}$ denotes the attention score corresponding to the $t$-th position in the sentence. We define the difference vector $\bm{d}_{t}$ as the difference between the attention to the $t$-th word $\tilde{a}_{t}$ in a sentence and the attention to any $k$-th word $\tilde{a}_{k}$:

$T=T_{S}$ in single sequence task, and $T=T_{P}$ in a pair sequence task. By normalizing the norm of the vector, we define a normalized difference vector of the attention for the $t$-th word:

The number of dimensions in $d_{k}$ is the number of the vocabulary (fixed length) for Word iAT, while the dimension of words in a sentence (variable length) for Attention iAT. The dimensionality of $d_{t}$ in Attention iAT is much smaller compared to Word iAT.³³3This normalization is done on a sentence-by-sentence basis, so it does not matter that the dimension is varies ($T<<V$). We define perturbation $r(\bm{\alpha}_{t})$ for attention to the $t$-th word with trainable parameters $\bm{\alpha}_{t}=(\alpha_{t,k})_{t=1}^{T}\in\mathbb{R}^{T}$ and the normalized difference vector of the attention $\tilde{\bm{d}}_{t}$ as follows:

By combining $\bm{\alpha}_{t}$ for all $t$, we can calculate perturbation $\bm{r}(\bm{\alpha})$ for the sentence:

Then, similar to $X_{\tilde{\bm{a}}+\bm{r}}$ in Eq. 10, we introduce $X_{\bm{\tilde{a}}+\bm{r}(\bm{\alpha})}$ and seek the worst-case weights of the difference vectors that maximize the loss functions as follows:

Contrary to Attention iAT, in Word iAT, the difference $d_{t,k}$ in Eq. 13 is defined as the distance between the $t$-th word in a sentence and the $k$-th word in the vocabulary in the word embedding space. Based on the distance, Word iAT determines the direction of perturbation for the $t$-th word as a linear sum of the word directional vectors in the vocabulary. In contrast, Attention iAT does not compute the distance to word embeddings in the vocabulary. Instead, this technique computes the difference in attention to other words in the sentence and determines the direction of the perturbation. The adversarial perturbation of Attention iAT, defined in this way, works to increase the difference in attention to each word. We call this process in Attention iAT as attention difference enhancement. Owing to the process, Attention iAT improves the interpretability of attention and contributes to the performance of the model’s prediction. The detail discussions are shared in Section VII-C.

For computational efficiency, we calculate the interpretable adversarial perturbation by applying the same approximation method as in Eq. 11:

Then, similar to Eq. 12, we construct a perturbated example for attention score $\tilde{\bm{a}}$:

At each training step, we generate adversarial perturbation in the current model. To this end, we define the loss function for adversarial training as follows:

where $\lambda$ is the coefficient that controls the balance between two loss functions. Note that $X_{\tilde{\bm{a}}_{\tt ADV}}$ can be $X_{\tilde{\bm{a}}_{\tt adv}}$ for Attention AT or $X_{\tilde{\bm{a}}_{\tt iadv}}$ for Attention iAT.

We evaluated the proposed techniques using the open benchmark tasks (i.e., four BCs, four QAs, and two NLIs) used in Jain and Wallace [10]. In our experiment, we added MultiNLI [41] as an additional NLI task for more detailed analysis (see the details in Appendix A). Table I presents the statistics for all datasets. We split the dataset into a training set, a validation set, and a test set.⁴⁴4Jain and Wallace [10] split the dataset into only a training set and a test set, so we did not get the same results. We performed preprocessing, including tokenization with spaCy⁵⁵5spaCy · Industrial-strength Natural Language Processing in Python https://spacy.io/, mapping out vocabulary words to a special <unk> token, and mapping all words with numeric characters to qqq in the same manner as Jain and Wallace [10].

We compared the two proposed training techniques to four conventional training techniques. They were implemented using the same model architecture as described in Section III. Following Jain and Wallace [10], we used bi-directional long short-term memory (LSTM) [42] as the BiRNN-based encoder, including Enc, $\mathbf{Enc}_{P}$, and $\mathbf{Enc}_{Q}$. A total of six training techniques were evaluated in the experiments:

• •

  Vanilla [10]: a model with attention mechanisms trained without the use of AT.

• •

  Word AT [18]: word embeddings trained with AT.

• •

  Word iAT [19]: word embeddings trained with iAT.

• •

  Attention RP: attention to the word embeddings is trained with random perturbation (RP).

• •

  Attention AT (Proposed): attention to the word embeddings is trained with AT.

• •

  Attention iAT (Proposed): attention to the word embeddings is trained with iAT.

We implemented the training techniques above using the AllenNLP library with Interpret [43, 44]. Through the experiments, we set the hyper-parameter $\lambda=1$ related to AT or iAT in Eq. 20. To ensure a fair comparison of the training techniques, we followed the configurations (e.g., initialization of word embedding, hidden size of the encoder, optimizer settings) used in the literature [10] (see the details in Appendix B).

Note that while Jain and Wallace [10] used a test set to adjust the model’s hyper-parameters, we used a validation set. In adversarial training, the Allentune library [45] was used to adjust hyper-parameter $\epsilon$, and we report the test scores for the model with the highest validation score.

First, we compared the prediction performance of each model for each task. As an evaluation metric of the prediction performance, we used the F1 score⁶⁶6The F1 score is a metric that harmonizes precision and recall.. Therefore, this score takes both false positives and false negatives into account., accuracy, and the micro-F1 score for the BC, QA, and NLI, respectively, as in [10].

Next, we compared how the attention weights obtained through the proposed AT-based technique agreed with the importance of words calculated by the gradients [26]. To evaluate the agreement, we compared the Pearson’s correlations between the attention weights and the word importance of the gradient-based method. In [10], the Kendall tau, which represents rank correlation, was used to evaluate the relationship between attention and the word importance obtained by the gradients. Recently, however, it has been pointed out that rank correlations often misrepresent the relationship between the two due to the noise in the order of the low rankings [46]; we concurred with this, so we used Pearson’s correlations.

Finally, we compared the effects of perturbation size $\epsilon$ of AT on the validation performance of the BC, QA, and NLI tasks with a fixed $\lambda=1$. We randomly choose the value of $\epsilon$ in the 0–30 range and ran the training 100 times. The configurations in [18, 19] were $\epsilon=5$ for Word AT and $\epsilon=15$ for Word iAT.

In terms of prediction performance, the model that applied the proposed Attention AT/iAT demonstrated a clear advantage over the model without AT (as shown in Vanilla [10]) as well as other AT-based techniques (Word AT [18] and Word iAT [19]). The proposed technique achieved the best results in almost all benchmarks. For 20News and AGNews in the BC and bAbI task 1 in QA, the conventional techniques, including the Vanilla model, were sufficiently accurate (the score was higher than 95%), so the performance improvement of the proposed techniques to the tasks was limited to some extent. Meanwhile, Attention AT/iAT contributed to solid performance improvements in other complicated tasks.

In terms of model interpretability, the attention to the words obtained with the Attention AT/iAT techniques notably correlated with the importance of the word as determined by the gradients. Attention iAT demonstrated the highest correlation among the techniques in all benchmarks. Figure 3 visualizes the attention weight for each word and gradient-based word importance in the SST test dataset. Attention AT yielded clearer attention compared to the Vanilla model or Attention iAT. Specifically, Attention AT tended to strongly focus attention on a few words. Regarding the correlation of word importance based on attention weights and gradient-based word importance, Attention iAT demonstrated higher similarities than the other models.

Figure 4 shows the effect of the perturbation size $\epsilon$ on the validation performance of SST (BC), CNN news (QA), and Multi NLI (NLI) with a fixed $\lambda=1$. We observed that the performances of the conventional Word AT/iAT techniques deteriorated according to the increase in the perturbation size; meanwhile, our Attention AT/iAT techniques maintained almost the same prediction performance. We observed similar trends in other datasets as described in Section V-A.

Attention AT/iAT is based on our hypothesis that attention is more important in finding significant words in document processing than the word embeddings themselves. Therefore, we sought to achieve prediction performance and model interpretability by introducing AT to the attention mechanisms. We confirmed that the application of AT to the attention mechanisms (Attention AT/iAT) was more effective than word embedding (Word AT/iAT) and supports the correctness of our hypothesis, as shown in Table II. In particular, the Attention iAT technique was not only more accurate in its model than the Word AT/iAT techniques but also demonstrated a higher correlation with the importance of the words predicted based on the gradient.

As shown in Figure 3, Attention AT tended to display more attention to the sentence than the Vanilla model. The results showed that training with adversarial perturbations to the attention mechanism allowed for cleaner attention without changing word meanings or grammatical functions. Furthermore, we confirmed that the proposed Attention AT/iAT techniques were more robust regarding the variation of perturbation size $\epsilon$ than conventional Word AT/iAT, as shown in Figure 4. Although it is difficult to directly compare perturbations to attention and word embedding because of the difference in the range of the perturbation size to the part, the model that added perturbations to attention behaved robustly even when the perturbations were relatively large.

Attention RP demonstrated better prediction performance than Word AT/iAT. The results revealed that augmentation for the attention mechanism is very effective, even with simple random noise. In contrast, the correlations between the attention weight for the word and the gradient-based word importance were significantly reduced, as shown in Table II. We consider that Attention RP is successful in learning robust discriminative boundaries through random perturbation and improving to the desired classification performance. However, as the gradient is smoothed out by the perturbation around the (supervised) data points, the correlation with the word importance by the gradient is considered to be degraded. In other words, Attention RP can achieve a certain level of classification performance, but it does not lead to which words are useful from their gradients.

In the experiments, Attention iAT showed a better performance compared Attention AT in the prediction performance and the correlation with the gradient-based word importance. Attention iAT exploits the difference in the attention weight of each word in a sentence to determine adversarial perturbations. Because the norm of the difference in attention weight (as shown in Eq. 14) is normalized to one, adversarial perturbations in attention mechanisms will make these differences clear, especially in the case of sentences with a small difference in the attention to each word. That is, even in situations where there is little difference in attention between each element of $\bm{d}_{t}=(d_{t,1},d_{t,2},\cdots,d_{t,T})$, the difference is amplified by Eq. 14. Therefore, even for the same perturbation size $||\bm{\alpha}||<\epsilon$, more effective perturbations $\bm{r}(\bm{\alpha})$ weighted by $\tilde{\bm{d}}_{t}$ were successfully obtained for each word. However, in the case of sentences where there was originally a difference in the clear attention to each word, the regularization of $\bm{d}_{t}$ in Eq. 14 had practically no effect because it did not change their ratio nearly as much. Thus, we posit that the Attention iAT technique enhances the effectiveness of AT applied to attention mechanisms by generating effective perturbations for each word.

The Attention iAT technique was inspired by Word iAT. Word iAT generates perturbations in the direction that maximizes the loss function while restricting the direction of the perturbation to become a linear combination of the direction of word embedding in the vocabulary. Word iAT indirectly improves the interpretability of the model by indicating which words in the vocabulary to which the perturbation is similar. However, we are confident that Attention iAT is a direct improvement in the interpretability of the model, because it can show more clearly which words to pay attention. This is owing to the attention difference enhancement process described in Eq. 14. Thus, the proposed techniques are highly effective in that they lead to a more substantive improvement in interpretability.

Our proposal is a general-purpose robust training technique for DNN models, which are commonly used for NLP tasks. Therefore, we have chosen here an RNN with an attention mechanism that has been put to practical use [10]. For this reason, models such as BERT [47] that deal with self-attention were outside the scope of this study, and will be the subject of future work. We also did not deal with tasks (such as machine translation) that were not used in the literature [10] as baselines. Additionally, for the same reason in as [48], we did not consider the variants of attention mechanisms, such as bi-attentive architecture [5], multi-headed architecture [8], because they could have different interpretability properties.

As an extension of AT, virtual adversarial training (VAT), which is a semi-supervised training technique, was proposed in [18, 49]. Based on VAT, the proposed technique can be expected to improve accuracy by using unlabeled datasets.