Using Single-Step Adversarial Training to Defend Iterative Adversarial Examples

As mentioned in Section 1, adversarial examples are specially perturbed original examples which aim to fool the NN classifier. Although adversarial examples can be categorized according to different aspects, we focus here on the distinction between Single-Exps and Iter-Exps. Adversarial examples throughout this work are $l_{\infty}$ white-box untargeted adversarial examples in the image classification domain. In other words: (1) the perturbation of adversarial example is limited in an $l_{\infty}$ norm ball, (2) the classifier is transparent to adversary, and (3) the adversary successfully fools the classifier as long as the prediction is wrong.

The fundamental idea of adversarial training is to use adversarial examples as blind spots and retrain the classifier with these blind spots. From a system level point of view, adversarial training can be represented as a two-step process.

Here, $I$ is an indicator function, $C$ represents the classifier, $L$ is a loss function, and $\Delta$ is the feasible set of $\delta$. Starting with a randomly initialized $\theta_{0}$, the searching of adversarial perturbation (Eq. 5) and the training of classifier (Eq. 6) will be competing with each other repeatedly until $\theta$ converges.

The fundamental idea of ATDA is to combine adversarial training with domain adaptation. The authors of ATDA believe that Single-Exps (e.g. FGSM examples) could be envisioned as limited number of samples in the adversarial domain. Therefore, combining domain adaptation methods could enhance the performance of Single-Adv. In its design, ATDA utilizes both unsupervised and supervised domain adaptations as follows.

The first step of our analysis targets the feature space encoding from four different classifiers (vanilla, ATDA, BIM(30,30)-Adv, and SIM(10,20)-Adv classifiers). We use t-SNE (Maaten and Hinton, 2008) to project the high dimensional feature encoding from each classifier to a two dimensional space and visualize it in Figure 1. During the analysis, we sample original examples from MNIST dataset. Examples from a randomly selected class are used as targets (green dots) while others are references (blue dots). Corresponding to targets, we generate adversarial examples (red dots). In the visualized feature space encoding, examples that are close to each other and form a group are much likely to be classified in the same class. Therefore, the classifier that can defend against adversarial examples should have a visualization where green and red dots are blended together.

From Figure 1a, we can see that the feature encoding of adversarial examples (BIM(30,30) examples) form several individual small groups are clearly different from the feature encoding of targets (green dots). Without the color, we can barely tell the difference between small groups of references (blue dots) and adversarial examples (red dots). When using the ATDA classifier, this problem is slightly mitigated since the red dots are grouped together in Figure 1b. However, from Figure 1b, the groups of green dots and red dots are still separable. BIM(30,30)-Adv and our SIM(10,20)-Adv classifiers presented in Figures 1d and 1e are significantly better than the ATDA classifier since the red and green dots are blended. It’s worth recalling that our SIM(10,20)-Adv belongs to Single-Adv and takes less training time than ATDA as presented later. Therefore, our method outperforms ATDA in both accuracy and training time as detailed in later sections. Finally, we add an extra visualization for the ATDA classifier and change the adversarial examples to FGSM examples. Compared with BIM(30,30) examples which are Iter-Exps, the FGSM examples are weaker Single-Exps. In Figure 1c, the red and green dots are blended which means, as expected, that the ATDA classifier performs well against FGSM (Single-Exps) examples.

Although we show that ATDA classifier has degenerated performance on BIM examples by analyzing feature space encoding, its numerical results, accuracy, in (Song et al., 2018) does not reflect the same finding. A possible reason is that the perturbation limit is too small compared with previous works such as (Madry et al., 2017). Moreover, while experimenting with the source code, we locate another possible reason, the tuned learning rate $\alpha=e^{-3}$.

We evaluate ATDA classifiers with different values of $\alpha$ as shown in Table 1. The results show that the test accuracy on both BIM(30,30) and Madry(30,30) examples significantly degenerates when we decrease $\alpha$. When $\alpha=e^{-4}$, the ATDA performs in a similar way as FGSM-Adv in (Tramèr et al., 2017). Actually, (Tramèr et al., 2017) has shown that FGSM-Adv causes the trained classifier to overfit FGSM examples and be vulnerable to Iter-Exps. It is intuitive that optimizing with a smaller learning rate should converge to the same location if not a better location. Therefore, the degeneration in Table 1 means that the objective function in ATDA, combining FGSM-Adv and domain adaptation losses, is not appropriate for correct classification of Iter-Exps.

Although previous subsections show that ATDA cannot defend Iter-Exps due to inappropriate objective function, the reason could solely be the usage of FGSM examples in the training. To pinpoint the effect of domain adaptation loss, we design another experiment which uses BIM examples instead of FGSM examples for training. We choose BIM(30,30)-Adv as the baseline and combine it with the domain adaptation loss proposed by ATDA. Our experiments are conducted on both MNIST and FMNIST datasets with BIM(30,30) and Madry(30,30) examples. Moreover, we assign different values to $\lambda$, a parameter that controls the weight of domain adaptation loss.

Figure 4 presents the results of this experiment. When $\lambda=0$, the classifier is solely trained with the cross-entropy loss. As $\lambda$ increases, the domain adaptation loss becomes more and more important in the total training loss. To our surprise, this experiment exposes another vulnerability of ATDA. Compared with cross-entropy loss, the domain adaptation loss does not make extra positive impact on the test accuracy. The test accuracy on BIM(30,30) examples remains unchanged or shows a small degeneration. Even worse, the domain adaptation loss hurts the test accuracy of the classifier on Madry(30,30) examples, especially when $\lambda\geq 0.15$. We think a reasonable explanation is that the randomness in Madry examples breaks the statistical assumption used in the domain adaptation loss.

In this section, we show that ATDA cannot defend Iter-Exps. Firstly, ATDA’s feature space encoding shows that it miserably fails against Iter-Exps. Secondly, by changing the learning rate, we show that the objective function of the ATDA does not take Iter-Exps into consideration. Lastly, by combining domain adaptation loss with Iter-Adv, we show that relying on domain adaptation loss degenerates the accuracy of classifying Iter-Exps with randomness (Madry examples). Although the analysis is done in MNIST and FMNIST datasets, our further evaluation results on CIFAR10 dataset (Sec 6) also show that ATDA performs poorly in defending Iter-Exps.

Based on the introduction of Iter-Exps in Section 1, it is clear that the smaller per step perturbation applied, the better observation of NN’s decision hyperplane and the stronger adversarial examples obtained. However, to select an appropriate per step perturbation, we believe that a quantitative analysis is needed.

To do this analysis, we conduct experiments on MNIST, FMNIST, and CIFAR10 datasets. In each dataset, we train two different NN classifiers with the same structure and hyper-parameter settings, which include: (1) a Vanilla classifier trained on original examples only and (2) a BIM-Adv classifier (Kurakin et al., 2016). For each $n_{1}=n_{2}$ value in the range $\mathbb{Z}_{[0,30]}$, we generate BIM examples with fixed $\epsilon$ ($0.3$ in MNIST, $0.2$ in FMNIST, and $\frac{8}{255}$ in CIFAR10) and calculate the following ratio, $\rho$:

To understand $\rho$, assume that its value is $1$ when, for example, $n_{1}=n_{2}=10$. And, the maximum value of $n_{1}$ and $n_{2}$ is $30$ (e.g. with MNIST dataset). This means that BIM examples generated with value $n_{1}=n_{2}=10$ can be as successful as those generated with value $n_{1}=n_{2}=30$ in misleading the classifier.

From the results in Figure 4, it is clear that $\rho$ converges fast and saturates when the value of $n_{1}=n_{2}$ is around 5 in all six lines. For the Vanilla classifiers, this phenomenon is not surprising since they have no defence at all and most of the adversarial examples can fool them. However, we see a similar trend from the BIM-Adv classifiers which are well trained to defend adversarial examples. The insight we draw from this experiment is that increasing the value of $n_{1}=n_{2}$ over a certain limit provides only marginal help in finding stronger adversarial examples. In other words, training a classifier by Iter-Adv method with small $n_{1}=n_{2}$ values (around 5 in this experiment) is as efficient as training the classifier by Iter-Adv with large $n_{1}=n_{2}$ values (30 in this experiment).

Given the fact that adversarial training uses adversarial examples to find blind spots of the under-trained classifier and retrain it, these results show that decreasing the per step perturbation of Iter-Exps used during Iter-Adv beyond a certain limit only marginally benefits the defense.

We think the saturation of per step perturbation exists because the loss structure used in projected gradient descent to search Iter-Exps is shown to be highly tractable (Madry et al., 2017). This important finding indicates that defenses could use smaller values of $n_{1}=n_{2}$ without sacrificing the quality of the defense. Although the resulting defense is still within the Iter-Adv category, it consumes less time and computations in preparing adversarial examples. We will utilize this observation in combination with the following others to develop an efficient Single-Adv method.

As shown in Section 2 Eq. 4, Iter-Adv methods usually use final adversarial examples ($x_{n_{2}}$) to build the defense since it is much stronger than intermediate versions ($x_{i},~{}\forall i<n_{2}$). In this section, we explore whether those intermediate examples can be utilized for training while being generated, instead of sitting idle and waiting for the final versions.

To do so, we conduct another set of experiments on MNIST, FMNIST and CIFAR10 datasets. In these experiments, we use the same NN classifiers, measure the same ratio ($\rho$ as defined in Eq. 13), and maintain the same perturbation limit used in the preceding subsection. The only difference is that we here fix the value of $n_{1}$ (10 for MNIST and FMNIST, 7 for CIFAR10) and $n_{2}$ (30 for all) in BIM examples. Values along the X-axis in Figure 4 correspond to different iterations during the generation of BIM examples. For example in MNIST, an X-axis value of zero corresponds to the original examples, a value of $0<i<30$ represents the corresponding intermediate examples after $i$ iterations, and a value of 30 corresponds to the final versions, BIM(10,30), of the adversarial examples.

The results show that $\rho$, under all the scenarios, is monotonically increasing with the number of iterations. For all three Vanilla classifiers which have no defense against adversarial examples, $\rho$ saturates quickly after around 5 iterations. Although $\rho$ for BIM-Adv classifiers does not saturate as quick as that for Vanilla classifiers, it increases almost exponentially before reaches around $0.8$ (MNIST and FMNIST) or $0.9$ (CIFAR10). In the zoom-in view of Figure 4, we can clearly see these turning points. One interesting observation is that the turning point in BIM-Adv classifier corresponds the selected value of $n_{1}$ (10 or 7).

Our insights from this experiment is that: a large portion of intermediate examples of Iter-Exps are good quality adversarial examples since they effectively reveal the classifier’s blind spot, and hence can be used to build high quality defenses. In other words, we can utilize the intermediate examples during the preparation of Iter-Exps to continuously enhance the model instead of waiting for the final examples while training Iter-Adv defenses. In Section 5, we build on this finding to expand the generation process of Iter-Exps and end up with a Single-Adv method that performs very close to Iter-Adv.

In this section, we identify two experimental properties of Iter-Adv: (1) It is recommended to use large per step perturbation, i.e., small $n_{1}$ and (2) intermediate examples can be used to speed up adversarial training with a very small degeneration in Iter-Exps quality.

In Figure 9, we review the process of adversarial training. Each row in the figure represents a training epoch and the solid black lines represent the data flow (training examples). The dashed red lines across different rows correspond to knowledge flow (classifier’s weights) from one epoch to the next. As shown in the figure, the original examples are fed into the generator of adversarial examples, which could be single-step or iterative. Then, the original and adversarial examples are used to train the classifier. The training process consists of several training epochs and the weights of the classifier are carried out from one training epoch to the next.

Inspired by the empirical findings drawn from our previous experiments (Section 4), we propose the following modifications to enhance the Single-Adv defense process. Similar to other Single-Adv methods, our method also uses the single-step generator to reduce computation overhead in each epoch. Recall that a classifier which is trained with Single-Adv fails to defend Iter-Exps, therefore, we use consecutive training epochs to mimic the generation of Iter-Exps.

Starting from the second training epoch, we reuse the output of the generator from the previous epoch as input to the generator of the current epoch, instead of using the original image. As a result, the classifier can be seen as trained with intermediate examples in the first $(n_{2}-1)$ training epochs. In each training epoch, our method uses a relatively large per step perturbation (i.e., small $n_{1}$) instead of total perturbation (i.e., $n_{1}=1$). It helps our method to avoid repeatedly generating Single-Exps for training. On the other hand, a large per step perturbation ensures the adversarial examples can quickly reach their maximum perturbation. Therefore, it can mitigate the degeneration caused by training with weak intermediate examples in the first few training epochs. Lastly, after repeating aforementioned steps for $n_{2}$ epochs, the generator switches to select original examples as inputs which means the iteration over consecutive epochs is reset.

From a high level point of view, we flatten the iteration of generating Iter-Exps into training epochs. Within each iteration of $n_{2}$ consecutive training epochs, the mathematical formulation of generating adversarial examples is as follows.

Here, $i$ represents the index of iteration over training epochs. Similar as traditional adversarial training method, we also present the process of SIM-Adv as a flow chart in Figure 7.

In the previous subsection, we present the core design of using Single-Exps to mimic Iter-Exps. At the same time, we also mention the potential disadvantage of this design. In the majority of training epochs, our method uses the intermediate examples. Recall the experiment results in Figure 4, these intermediate examples are less serious than the corresponding Iter-Exps. Based on our experimental results, the classifier directly trained with the SIM examples can defend against adversarial examples but performs worse than that trained with Iter-Exps.

To further mitigate the gap in performance, we now introduce a heuristic modification of the hyper-parameter setting which we call it over-perturbation. For the first time, we define two different hyper-parameter settings in adversarial training methods. We define that the setting is over-perturbation when $n_{1}<n_{2}$, otherwise is under-perturbation. This modification is based on our empirical results. As aforementioned, the intermediate examples are less serious than final Iter-Exps. However, the zoom-in view in Figure 4 shows the existence of turning point in the iteration index and its connection to the value of $n_{1}$. Before the turning point, the success rate to mislead classifiers by intermediate examples is much lower than that by Iter-Exps but increases exponentially, and vice versa.

By applying over-perturbation, we actually ensure that our method trains the classifier with strong adversarial examples in most of the training epochs. Assume we run 20 epochs of training with two settings (1) $n_{1}=n_{2}=10$ and (2) $2n_{1}=n_{2}=20$. Under the first setting, the classifier is trained with intermediate examples before the turning point in both 1^st to 9^th and 11^th to 19^th epochs. While, under the second setting, the intermediate examples used between $10^{th}$ and $19^{th}$ epoch are after the turning point. Overall, the second setting spends more epochs in training the classifier with strong adversarial examples and the trained classifier performs better on defending adversarial examples.

Actually, the over-perturbation setting has been used in previous research works, intentionally or unintentionally. For example in (Madry et al., 2017), the hyper-parameter setting of all Madry-Adv methods are over-perturbation. For curiosity, we try to change the hyper-parameter setting from under to over-perturbation and record the performance. In both MNIST and FMNIST datasets, we fix the $n_{2}$ to 30 and iterativelly reduce the value of $n_{1}$ from 30 to 10. In each setting, we measure classifier’s test accuracy on both BIM and Madry examples. These results are presented in Figure 7. To our surprise, we found that the Madry-Adv with under-perturbation may train a classifier performs significantly worse than that trained with over-perturbation (in FMNIST).

For this phenomenon, we believe that it is related to the random initialization in Madry-Adv. In some situations, the random initialization may add unnecessary perturbation to the image and such perturbation could degenerate the performance of trained classifier. Under such situations, the over-perturbation makes it possible to eliminate unnecessary perturbation which enhances the performance of the trained classifier. More detailed analysis of combining over-perturbation and Madry-Adv is out of the scope of this work and we keep it as our future work.

In order to intuitively show the effectiveness of our proposed method, we prepare a toy example to compare the search space of adversarial examples. Without loss of generality, we assume that the data in this toy example is in a 2 dimensional space. Therefore, we could easily present the search space of adversarial examples. In Figure 9, we choose to present the search space of optimal adversarial examples as well as BIM, FGSM and SIM examples.

As we see from the top-left corner, the search space of optimal adversarial examples is the blue square which represents the entire norm ball. However, this search space corresponds to the exhaustive search which cannot be achieved. Among others, the BIM examples are the best mimic of the optimal adversarial examples. Since it separate the total perturbation into multiple steps and iteratively apply small perturbations, the search space of BIM examples can be represented by the mesh of red dots. The density of dots is related to the size of per step perturbation. Compared with BIM examples, the search space of FGSM examples are significantly limited. Since the total perturbation is applied all at once, the potential locations (red dots) of FGSM examples can only cover corners and some surface of the norm ball, while the entire inner space between origin and perturbation boundary is unreachable.

When we focus on the SIM examples, its search space could be represented by a mesh of red dots with dynamically changing size. With a fixed density of dots, the size of the mesh increases from the smallest one (just around the origin) to the largest one (same size as the entire norm ball) epoch-by-epoch. Although the searching space of SIM examples is smaller than that of BIM examples at the beginning, small $n_{1}$ value and over-perturbation setting ensure that most of epochs are searching adversarial examples in the entire norm ball. Moreover, the analysis of Iter-Exps shows that a relatively lower density of dots does not significantly affect the searching of adversarial examples. Last but not the least, SIM examples are Single-Exps which consumes less computation overhead compared to the Iter-Exps of BIM.

During the preparation of this work, we noticed that there is a related work, denoted as Free-Adv, that is uploaded to the arXiv (Shafahi et al., 2019). This work shares a similar design to our paper in generating adversarial examples. However, our proposed work is different from it and performs better. Until writing this paper, this work is still an arXiv paper and has not been accepted as a conference or journal submission. Therefore, we did not list it as the SOTA Single-Adv method due to the lack of peer review. However, for the completeness of our work, we still reimplement, evaluate and compare it with our proposed method.

During the generation process of adversarial examples, both the Free-Adv and our proposed method are reusing the gradient information. In the Free-Adv, the training images in each mini-batch are Single-Exps. In order to mitigate the issue of training with Single-Exps, the authors try to replay each mini-batch multiple times (denoted as $m$) and reuse the adversarial examples from previous replay as the inputs. Finally, the total training epoch of the Free-Adv will be decreased by the factor of the total replay iterations. Although the Free-Adv looks similar to our proposed method, there are two differences which can distinguish it from our work.

First of all, the proposed algorithm in the Free-Adv replays the images in the mini-batch immediately which means the classifier is repeatedly trained with the same mini-batch several times. As the authors pointed out in (Shafahi et al., 2019), such replay could cause the “catastrophic forgetting” problem. If the information in the training data is unbalanced, the classifier could become overfitting through being repeatedly trained with a certain category of training data. On the contrary, our proposed method utilizes gradient information across different training epochs. In other words, our method does not change the training order which could be specially designed. Therefore, our method does not have the risk of causing the “catastrophic forgetting” problem (Shafahi et al., 2019).

Secondly, in our work adversarial perturbation is applied in a different way compared with Free-Adv. Based on Algorithm 2 in (Shafahi et al., 2019) and the official implementation on GitHub, it is clear that the Free-Adv utilizes the total perturbation during each replay. From Figure 9, we show that using total perturbation only (e.g. FGSM examples) has limited sampling locations which can only cover the corners and parts of the surface of a $l_{\infty}$ ball. Although the Free-Adv repeatedly replays the mini-batch several times, its sampling locations will not change unless most of the pixel values are close to the clipping boundary. Although people may think that it is a problem about the hyper-parameter tuning, we argue that the design of applying adversarial perturbation should be based on the empirical analysis of iterative adversarial examples proposed in Section 4.

In order to support our hypothesis, we repeat the experiment in Section 3 to represent the feature space encoding of the classifier trained with the Free-Adv. The results are presented in Figure 9 and the legends are the same as those in Figure 1. When the FGSM examples are used as adversarial inputs, the classifier could group them together. However, these adversarial inputs and their corresponding targets belong to two separated groups which means the Free-Adv performs worse than the SIM-Adv. If the adversarial inputs are BIM examples, the trained classifier is fooled since the red dots are separated into several small groups. Overall, the presented results in Figure 9 reflect that the Free-Adv has a similar issue as the ATDA and performs even worse than ATDA in MNIST dataset. This issue is also identified during our evaluation of the Free-Adv with the details presented in the next section.

In this evaluation, we select three different datasets: MNIST, FMNIST and CIFAR10. For the MNIST and FMNIST datasets, we select the LeNet (LeCun et al., 1998) as network structure, while, the ResNet structure (He et al., 2016) is used in the CIFAR10 dataset. Within each dataset, we evaluate the performance (classification accuracy) of the trained classifier against both original and different types of adversarial examples.

As mentioned in Section 2, all the adversarial examples throughout this work are $l_{\infty}$ white-box untargeted adversarial examples in the image classification domain. For the method of generating adversarial examples, our selection includes FGSM, BIM and Madry which reflects the performance on both Single-Exps (FGSM examples) and Iter-Exps (BIM and Madry examples). The total perturbation limits are 0.3 in MNIST, 0.2 in FMNIST and $\frac{8}{255}$ in CIFAR10 which is used in the (Madry et al., 2017). To make the evaluation more convincing, we select larger $n_{2}$ value in both BIM and Madry examples to make them stronger than adversarial examples used during training.

As a baseline, we present the evaluation results of the vanilla classifier, a one with no defense against adversarial examples. To better evaluate our proposed method, we compare not only with Single-Adv methods (ATDA and Free-Adv), but also with Iter-Adv ones (BIM-Adv and Madry-Adv). In the evaluation, we skip adversarially trained models with FGSM or R+FGSM examples. Although FGSM-Adv and R+FGSM-Adv are single-step version of BIM-Adv and Madry-Adv, previous studies show that they are failed to defend Iter-Exps (Kurakin et al., 2017)(Tramèr et al., 2017). In other words, directly setting $n_{1}=n_{2}=1$ in BIM-Adv or Madry-Adv cannot train a classifier that makes correct predictions on Iter-Exps. Instead, we present the ATDA and Free-Adv as representatives of Single-Adv.

For these selected adversarial training methods, we follow the original setting of hyper-parameters presented in their work and fine-tune the setting to present the best performance. For our proposed method, we tune its hyper-parameters as follows. Firstly, its $n_{1}$ can be select as the same or one half of $n_{1}$ value in corresponding BIM-Adv method. Secondly, the $n_{2}$ value is selected among 2 to 5 times of the $n_{1}$ value due to the over-perturbation. Among these combinations, we present its best performance. The specific hyper-parameter values of all methods are presented along with evaluation results.

During the evaluation, we compare different trained classifiers in terms of test accuracy which is defined as follows:

Here, the calculation of test accuracy is based on a single category of inputs (e.g. original examples or FGSM examples). Moreover, we also measure the total training time consumed by training classifiers with different methods. To ensure the quality and reproducibility, adversarial examples used in both training and evaluation are based on the well-known standard python platform, CleverHans (Papernot et al., 2018). A summary of these evaluation settings is also presented in Table 2.

We first evaluate the Free-Adv method. As Table 3 shows, the Free-Adv classifier can defend both Single-Exps and Iter-Exps in the CIFAR10 dataset. However, its Iter-Exps accuracy degenerates significantly with MNIST and FMNIST datasets. We think this phenomenon is related to the second issue of the Free-Adv that is mentioned in Section 5. The limited searching space makes the classifier trained with the Free-Adv unable to defend the Iter-Exps. In the CIFAR10 dataset, training with the Free-Adv enhances the defence against Iter-Exps since the per step perturbation is enlarged from $\frac{1}{10}\epsilon$ to $\frac{1}{4}\epsilon$. As a result, the issue of limited searching space is mitigated. Even in the CIFAR10 dataset, the test accuracy of the classifier trained with the Free-Adv is still lower than that of our classifier (SIM-Adv), because SIM-Adv can tune the hyper-parameters (i.e. $n_{1}$) for more appropriate per step perturbations.

As shown in Table 3, the performance of the SOTA Single-Adv (ATDA) is significantly worse than that in (Song et al., 2018). The reason is that the perturbation limit is too small in the original work. For example, $\epsilon$ in the original evaluation is $\frac{4}{255}$ instead of $\frac{8}{255}$ on CIFAR10 dataset. As a result, Iter-Exps are similar to Single-Exps and even FGSM-Adv achieves over 49% test accuracy on Madry examples. Therefore, we think the original evaluation of ATDA is misleading. Our exclusive experiments in Section 3 point that ATDA actually fails to defend Iter-Exps due to the use of: (1) FGSM examples which are less representative, and (2) domain adaptation loss that is vulnerable to randomness.

Compared with the Free-Adv and the ATDA methods, the evaluation results show that adversarial training with SIM examples (SIM-Adv) achieves better and more stable performance. In all the three datasets, the classifier trained with the SIM-Adv can defend both Single-Exps and Iter-Exps while maintaining a reasonable test accuracy of original examples. More importantly, the SIM-Adv significantly enhances the test accuracy on Iter-Exps over the ATDA and the Free-Adv. To the best of our knowledge, this is the first Single-Adv method which can efficiently defend Iter-Exps under the white-box adversary model. To double check, we also test the SIM-Adv with the same evaluation on the ATDA work in Section 3. The results show that the SIM-Adv can defend both Single-Exps and Iter-Exps under different values of $\alpha$ and achieve better performance when $\alpha$ is lower. The evaluation results are presented in Table 4.

To make our evaluation results more convincing, we also compare the SIM-Adv with SOTA Iter-Adv methods which include both BIM-Adv and Madry-Adv. The overall results show that the SIM-Adv has a competitive performance as the BIM-Adv and Madry-Adv. Although the classifier trained with the SIM-Adv has a degeneration in terms of test accuracy, we think the less than 4% decrease is a reasonable trade-off given that the SIM-Adv trains the classifier with weak adversarial examples in some of its training epochs to save training overhead. Moreover, we could tune the hyper-parameter of the SIM-Adv to achieve better trade-off between test accuracy and training cost. For example, the SIM-Adv could perform a two-step iteration in each training epoch instead of single-step. We leave the analysis of this for future works.

To fairly evaluate the total training time, we select four different defense methods: the Free-Adv, the ATDA, the SIM-Adv and the BIM-Adv. We do not present the Madry-Adv since it has similar training time as that of BIM-Adv. All of our results are measured on a Dell Workstation with a NVIDIA RTX-2070 GPU.

The evaluation results in Table 5 clearly show that our SIM-Adv significantly reduces the total training time compared with the BIM-Adv. The SIM-Adv saves more than 60% on both MNIST and FMNIST datasets and more than 75% on CIFAR10 dataset in terms of total training time. Even compared with the ATDA, our SIM-Adv can still save at least 7% of total training time since the domain adaptation loss requires additional computation. When comparing with the Free-Adv, our proposed SIM-Adv consumes more training time since it has to save and restore the gradient information across training epochs.