Unsupervised Detection of Adversarial Examples with Model Explanations

One of the first proposed measure of attribution is input gradient (Simonyan et al., 2014). Intuitively for a linear function, input gradients represent exact amount that each input feature contributes to the linear function’s output. For image inputs, each pixel’s contribution could be represented in a heatmap called saliency map.

As most practical deep networks compute a confidence score for each class label and output the class of with the largest score, multiple saliency maps can be obtained according to the target class label $c$. For simplicity, we only consider the saliency map corresponding to the output class label of the given input. Formally, given an input $x$ and DNN $f$, saliency map of input $x$ is computed as follows:

where $S_{c}$ denotes a confidence score for class label $c$ (i.e., $f(x):=\arg\max_{c}S_{c}(x)$).

In our proposed method, the inspector is assumed to have access to the training samples $\mathcal{D}_{train}$ that was used to train the target classifier. In order to perform unsupervised anomaly detection based on the model explanations, the inspector first generates input explanations for the target model, using training samples.

As noted in Section 2, explanations of the target classifier depends on the output label $c$. As the explanations are differently applied for each label, the inspector organizes generated explanations according to the corresponding input’s output label. We denote by $\Phi^{c}$ as a set of input explanations for the inputs in the training dataset with output label $c$.

Once the explanations for training samples are collected and organized, the inspector then trains reconstructor networks — one for each class label — which reconstructs the explanations for an input with corresponding class label. Such reconstructor networks will later be used in separating adversarial examples from benign examples.

For each class label $c$, the inspector trains a parameterized network $g(\theta^{c};\cdot)$ reconstructing input explanations, solving the following optimization:

where $\mathcal{L}_{\Phi}(\theta;\cdot)$ is a reconstruction loss for network $g(\theta;\cdot)$ on $\Phi$.

Lastly, the inspector utilizes the trained reconstructor networks in order to separate adversarial examples from benign examples. As the networks are optimized to reconstruct model explanations of training samples, it will show poor reconstruction quality when an unusual shape of explanation is given. Hence, when the reconstruction error is above certain threshold, it is likely that the given input is adversarially crafted.

Formally, for a given suspicious input $x^{\prime}$, the inspector first obtains the class label $c^{\prime}:=f(x^{\prime})$ and its explanation $\phi^{\prime}:=\phi(x^{\prime},f)$. If the reconstruction error of $\phi^{\prime}$ is larger than given threshold $t_{c}^{\prime}$ for label $c^{\prime}$ (i.e., $\mathcal{L}(\theta^{c^{\prime}};\phi^{\prime})>t_{c}^{\prime}$), the inspector concludes that the input $x^{\prime}$ is likely to be an adversarial example.

Figure 2 shows pairs of input image (Input), explanation (Gradient) obtained from the target classifier, and the reconstruction (Recons.) from the trained reconstructor networks, for an example MNIST image and adversarial examples crafted from the image. Here, we confirm our insight that small adversarial perturbations to inputs can lead to noticeable changes in their explanations. Since the reconstructor networks are only trained on benign explanations (input explanations of benign examples), we see that reconstructions of adversarial explanations (input explanations of adversarial examples) are more noisy than the reconstruction of the explanation of original image.

In order to evaluate the effectiveness of our proposed detection method, we measure Area Under the ROC Curve (AUC). As our method uses multiple reconstructor networks, we record multiple values of AUC — each corresponding to a given class label — as well as their average.

Figure 3 shows the trend of AUC values under different adversarial attack scenarios. For each attack, we plot the min, max, and average values of AUC according to the severity of attack (i.e., value of $\epsilon$). While our method has harder time separating adversarial examples with smaller noise level, average AUC stays relatively high even in its lowest values (¿95% for FGSM, ¿97% for PGD and MIM, when $\epsilon=0.05$).

Our methods show high performance (average AUC of 0.9583 for FGSM, 0.9942 for PGD, 0.9944 for MIM) in the standard adversarial attack setting of $\epsilon=0.1$ for MNIST dataset. This can be also confirmed by observing Figure 4, where the ROC curves of our detection method for different class labels are plotted.

Note that only a single set (i.e., one per class label) of reconstructor networks is used in all of the attack scenarios. The results clearly show that our defense method can generalize across multiple attack algorithms – which is impressive, considering that no single adversarial example is provided in the training phase. For a more detailed benchmark results, we refer to Appendix B.

We quantitatively compare our method’s adversarial detection accuracy to a number of previous approaches. Specifically, we compare our results with the results from four different existing works ((Feinman et al., 2017; Lee et al., 2019; Liang et al., 2021; Ma et al., 2018)), where the benchmark results are recorded in (Sutanto and Lee, 2021).

Table 2 shows comparison on adversarial detection accuracies of the proposed and existing approaches. In all experiments, our method performs the best or the second best in detecting adversarial samples. The results show that our method is comparable to, and often outperforms existing methods.