Not So Robust After All: Evaluating the Robustness of Deep Neural Networks to Unseen Adversarial Attacks

One of the most popular approaches to defending the neural networks against attacks is called adversarial training. It proposes to "include" possible adversarial examples into the training data sets to prepare a model for the attacks. The following min-max optimization problem [17] should be solved to get the adversarial trained model:

where, $\theta$ - are the weights of neural network, $D$ - the training data set, $S$ - the space of possible perturbations, $S=\{\delta:\|\delta\|_{p}<\epsilon\}$ for a given radius $\epsilon$.

Adversarial training was introduced by Goodfellow et al. in [10]. Madry et al. [17] proposed using a PGD attack during the training procedure and presented better robustness against adversarial attacks. However, Wong et al. [28] achieved about the same accuracy against adversarial attacks, using simple one-step FGSM.

It is important to note that the provable defense (i.e. "certified robust") against any small-$\epsilon$ attack has already been studied, for example, by Wong and Kolter [27] and Wong et al. [29]. However, the experiments in these works were conducted with relatively small perturbations: in [29] the maximum radius of $L_{\infty}$ perturbation is $\frac{2}{255}$, while the same norm in adversarial training package [9] is $\frac{8}{255}$. Thus, we do not include certified robust methods in our research.

Here and later, we refer to training based on empirical risk minimization as "regular" since it does not involve any adversarial attack and is traditionally used to train neural networks.

The motivation for these experiments stems from the work of Tramer et al. [26], which demonstrated that even adversarially trained models could be compromised by unseen attacks. The core premise of this experiment involves taking an adversarially trained deep model, trained to defend against a specific attack on a given dataset, and utilizing it to develop the robust version of this model, and then evaluate the generalisation perofmance of both models against unseen attacks. This process is undertaken as follows:

1. 1.

   We select every image within the dataset and use the process outlined in [11] to extract an image that only possesses robust features. The adversarially trained model aids in this extraction. Initially, we generate a random noise image for each target image. Then, we iteratively compute the representations of both the random and target images as interpreted by the pre-trained model. At each iteration, we slightly adjust the random image to minimize the distance between the vectors of the two images. After a set number of steps, this method produces a modified dataset.

2. 2.

   We then proceed to train the regular version of our selected adversarially trained model on this modified dataset, ultimately resulting in a robustly trained model.

3. 3.

   Subsequently, we test the generalisation capacity of both the adversarial and robust models against attacks which were not considered during the adversarial training phase, and hence, not used in the formation of the robust model.

This entire experiment is practically implemented for two distinct models: ResNet50s and InceptionV3. For ResNet50, we create both the adversarially trained and robust versions, aiming them at the CIFAR-10 [16] dataset and defending against the PGD attack with $L_{2}$ and $L_{\infty}$ norms. We then test their performance against FGSM ($L_{1}$, $L_{2}$, $L_{\infty}$ norms), PGD ($L_{1}$, $L_{2}$, $L_{\infty}$ norms), C-W ($L_{2}$ norm), and DeepFool ($L_{2}$ norm) attacks. The performance results of the $L_{2}$-trained model are outlined in Table 1, and those of the $L_{\infty}$ trained model are presented in Table 2.

The adversarially trained and robust versions of InceptionV3, on the other hand, were trained only against the PGD attack with the $L_{\infty}$ norm for CIFAR-10 dataset. The testing parameters for these models were akin to those used with ResNet50, and the results are displayed in Table 3. All tables highlight the corresponding values of $\epsilon$-s and the number of steps for iterative attacks used in each case. Note that $\epsilon$ for the attacks differs from the similar ones for ResNet50-s because the model has the bigger input shape (224x224 vs 32x32, respectively).

On examination, it’s apparent that both tested models demonstrate reasonable stability against some variations of PGD and FGSM attacks. However, all attacks with an $L_{\infty}$ norm significantly undermined the accuracy of the models. Hence, the model doesn’t ensure generalisation against all attack types, primarily as a simple increase in norm or perturbation shows a drastic impact. Despite this, the models did demonstrate resistance to certain unseen attacks, particularly those with an $L_{1}$ norm, exhibiting only a minor drop in accuracy. This resistance to $L_{1}$ norm attacks is crucial for these models in terms of generalisation, given they weren’t exposed to such an attack during training.

The accuracy of adversarially and robustly trained models might vary, but a similar ratio between them implies similar behavior. We evaluate this accuracy employing the Kolmogorov-Smirnov test for goodness. The null hypothesis $H_{0}$ at a significance level of 0.05 asserts that the samples, i.e., accuracy of robust and adversarial trained models under different attacks, are from the same distribution. The test for accuracy presented in Table 1 illustrates that, with a $p-valueof0.42$, we cannot reject the $H_{0}$ hypothesis. Therefore, robust models may be considered as a specific case of adversarial training, with its strengths and weaknesses.

In a bid to further extend this experiment, we opt to manage the entire adversarial training pipeline ourselves, thereby performing the process from scratch. Owing to computational constraints, we select the relatively more manageable ResNet18 architecture. The models are trained on the PGD attack with 5 iterations and $L_{2}$ and $L_{\infty}$ norms. For datasets, we utilize CIFAR-10 (the results are displayed in Table 4) and CINIC-10 [7] (refer to Table 5), and for test attacks, we use PGD and FGSM. Consistent with the previous pattern, attacks with the L-inf norm result in a more significant drop in accuracy than similar ones with the L2 norm.

Canonical correlation analysis (CCA) is a method utilized to compare the representations of neural networks. Its objective is to identify linear combinations of two sets of random variables that maximize their correlation. In previous studies [20], [1], CCA has been employed to compare activations from different layers of neural networks. Raghu et al. [22] proposed an extension of CCA, singular value canonical correlation analysis (SVCCA), for the analysis of neural networks.

In this study, we employ SVCCA to compare the representations of the original and corresponding adversarial images. For each experiment, we take a batch of 128 images, compute the related adversarial examples under some attack, calculate SVCCA for the representations, and take the mean. We test three models in each experiment: regular and two adversarial ResNet50-s trained with PGD using $L_{2}$ and $L_{\infty}$ norms. In this experiment, a high mean correlation coefficient indicates that the representations are similar to each other and that small perturbations do not impact the model.

While the attack norms are the primary variables in these experiments, we also test two different attacks (FGSM and PGD) to eliminate the threat of validity. The means of SVCCA for the attacks are presented in Table 6, while Table 7 shows the same for the best and worst cases from Tables 1 and 2.

Overall, the results presented in tables 6 and 7 demonstrate that the impact of adversarial attacks on models is most significant when using $L_{\infty}$-norm perturbation. This effect is evident even in models that have been trained specifically to handle $L_{\infty}$-norm attacks. These findings emphasize the importance of considering the norms of adversarial attacks when evaluating the robustness of models because testing on $L_{2}$-norm may provide a false sense of security.

The visualization of representations presents a challenge due to the high dimensionality of representation vectors. Nonetheless, such visualization can be useful for analysis purposes. To address this issue, we employ Principal Component Analysis (PCA) to reduce the dimensionality of representations from 512 to 2. We limit our experimentation to ResNet18 due to computational constraints.

We present the visualization of representations for different combinations of models and norms of PGD attacks in Figures 1 - 5. Figure 1 depicts the representations of samples from CIFAR-10 for a regularly trained ResNet18, which serves as a baseline case. In Figures 2 and 3, we visualize the data as adversarial samples with $L_{2}$ and $L_{\infty}$ norms of attack, respectively. Furthermore, we examine the representations of adversarial samples for adversarial trained models in Figures 4 and 5. To challenge the models, we use alter norms of attacks from the training ones, where the model trained on PGD with $L_{2}$ norm is tested on $L_{\infty}$-norm PGD attack (Figure 5), and vice versa (Figure 4). We group the representations of different classes by colors in all figures to comprehend how the representations of different classes intermingle under adversarial attacks.

The regular trained network representations are clustered according to the classes in the data set, as depicted in Figure 1. However, when subjected to adversarial attacks, all representations become heavily mixed, resulting in a more challenging classification task due to overlapping representations of different classes. The most mixed representations were observed on the regular trained network through the $L_{\infty}$-norm PGD attack (Figure 3). In contrast, adversarial training, as shown in Figures 4 and 5, resulted in some class representations (e.g., "Automobile" or "Truck") remaining clustered while becoming closer to each other than those without attacks. The same pattern was observed for the $L_{2}$-norm attack (Figure 3).

Both $L_{\infty}$ and $L_{2}$ demonstrate a marked decrease in accuracy, as evidenced by the aforementioned experiments (Tables 4 and 5). However, the impact of these norms on representations is not immediately apparent from a plain accuracy analysis. Indeed, the visualization of the representations distribution yielded intriguing patterns that warrant further investigation.

This study seeks to investigate the impact of adversarial training on the decision boundaries of models. Specifically, the mean distance between samples and the decision boundary is examined to determine how it differs for adversarially trained models. We use the idea of Mickisch et al. [18] who utilized the DeepFool attack 2.0.4 to measure this distance.

The decision boundary is defined as the set of input images where two or more classes share the same maximum probability, indicating that the classifier is uncertain about the class of the image.

where $c$ - is the number of classes. Under this definition, the usage of the DeepFool attack looks natural because it aims to find a perturbation to the closest wrong class. The distance of the sample $x$ to the decision boundary $D$ can be measured as:

The pretrained ResNet50 model is used with 20 steps of PGD attack during training, while ResNet18 models are trained under PGD with only 5 steps. The testing dataset is CIFAR-10. The outcomes of ResNet18 and ResNet50 models with different training configurations are presented in Table 8. The table displays the mean difference between decision boundaries of models and images from CIFAR-10, calculated using both $L_{2}$ and $L_{\infty}$ distances. The "steps" column represents the mean iteration of DeepFool spent during the attack generation. The comparison is made between the models from Tables 4, 1, 2.

The results indicate that the mean distance for ResNet18-s (for all types of training) to the decision boundary is relatively small, especially for $L_{\infty}$ norm. A small distance from a sample to the decision boundary implies that it is easier to misclassify this sample because it does not require a significant perturbation. Interestingly, the distance for $L_{\infty}$ norm adversarial trained model is actually less than training $\epsilon$. However, testing of this model under PGD attack (Table 4) suggests that it has some robustness. Therefore, the reliability of the popular method of model testing used in this study is called into question.