A Data-Centric Approach for Improving
Adversarial Training Through the Lens of Out-of-Distribution Detection ^††thanks: ¹Equal contribution.

The threat of adversarial examples was first noticed in image classification models [4]. The accuracy of the model can be significantly reduced when a norm-bounded perturbation is added to the input. This perturbation can be generated with iterative updates based on the loss function gradient [14] as:

where $\delta_{t}$ is the perturbation at step $t$, and $J(\theta,x,y))$ is the cost used to train the neural network with parameters $\theta$. To ensure that the perturbation is imperceptible, it can be projected to the $\ell_{p}$-norm ball which is known as the Projected Gradient Descent (PGD) attack [8]. There are also some other powerful attacks, but PGD is regarded as a standard attack for training and evaluation.

Several methods have tried to stand against adversarial examples, but they mostly give a false sense of robustness due to the reasons such as gradient obfuscation or vulnerability against newer attacks [7]. Still, the best existing defense is adversarial training [8], which trains the model with adversarial examples. Due to the effectiveness of adversarial training, recent methods have tried to improve it by methods such as data augmentation [9], training attack modification [15], and model hyper-parameters tuning [16].

As mentioned in section II-B, there are different variants of adversarial training. Taking a closer look at them from a different angle, we can demonstrate that some of them are equivalent to applying smaller perturbation or optimization weights to the hard training examples in the model optimization. To this end, a number of methods are investigated in the following.

CAT [17] and IAAT [11]: These methods hypothesize that the poor generalization of adversarial training is a consequence of uniform perturbation radius around every training sample. They suggest a sample-specific perturbation radius that increases the radius until the perturbed sample is miss-classified. Thus, the hard training examples would have smaller perturbation budgets during training to simplify the training of the model.

MART [18]: This method proposed to explicitly differentiate the misclassified and correctly classified examples during the training by setting lower training optimization weight for the misclassified unperturbed samples. Misclassified samples are presumably near the boundary, so they can be considered hard training examples for the model, which this method suggests assigning a lower optimization weight to.

TRADES [19]: There is a trade-off between clean and adversarial accuracy. To mitigate this issue, TRADES suggests to trade adversarial robustness off against accuracy by training model on clean examples while minimizing the KL divergence of model prediction for clean and perturbed samples. Hence, if a sample is near boundary and have a high prediction error, the KL divergence term would get a smaller value and the model would focus on learning the clean example.

Early stopping [20]: It is shown that overfitting to the training set harms robust performance in adversarially robust training. In other words, model can learn adversarially perturbed training samples with a high accuracy but it does not generalize well to the test samples. As a solution, the training can be stopped early if overfitting occurs using a validation set. This is equivalent to stop training before learning out-of-distribution or hard samples.

Noisy label [21]: Similar to standard training, adversarial training also suffers from noisy labels during training. A measure that can help to detect the noisy samples is the number of PGD iterations needed to generate misclassified adversarial examples. In other words, this measure also tries to find the near-boundary training examples using the number of PGD steps as a distance measure.

When training a model, we might come across a number of samples that are close to the decision boundaries or even outside of the training classes distributions. These samples might have been produced as a result of a labeling error, or other reasons such as conceptual ambiguity in the images that makes classifying them even difficult for humans. Examples of such samples can be seen in the Fig. 2 that represents some images from the CIFAR-10 dataset.

Due to the difficulty of training a model on these samples, we refer to them as Hard Training Samples (HTS). Additionally, learning an $\epsilon$-ball around these samples makes training even more challenging and raises the risk of problems like overfitting or mixed-up decision boundaries. Therefore, we believe that they may be better avoided during training as discussed in previous section on the existing methods.

There are many techniques for identifying such near-boundary or out-of-distribution samples. Detecting samples that do not fit into the classes of the training dataset is mainly known as out-of-distribution (OOD) detection in the literature. A simple but effective existing method for OOD detection is Maximum Softmax Probability (MSP) [12]. This method uses $max_{c\in\{1,2,\ldots,k\}}f_{c}(x)$ as the score function that classifier $f$ trained on a $k$-class dataset returns for input $x$ to be classified as the in-distribution sample.

Inspired by MSP, we propose Correct Class Softmax Probability (CCSP) as our method to detect HTS samples. CCSP for sample $x$ with label $y$ is defined as:

where $z$ is the output of classifier for the input $x$. CCSP score can be used as a measure to find the hard samples in comparison with other ones. This method has no computational overhead during the training, and can be used easily.

Now that we are able to recognize HTS, we can improve the training process by removing these samples before the training procedure begins. To this end, we need to measure the CCSP score for each sample, which is done through a $k$-fold cross-validation. Accordingly, we divide training dataset into $k$ folds called ${F_{1},...,F_{k}}$. Each time, a single fold $F_{i}$ is put aside and the rest are considered as the training dataset for the model to be trained. Afterward, the CCSP is calculated for the samples in $F_{i}$ fold. Eventually, the CCSP will have been calculated for all of the samples in the training dataset and $R$ (hyper-parameter) samples with the lowest CSSP score can be removed from the dataset. Finally, we can re-train our model using the new dataset that is resulted after removing such samples. This algorithm is called “offline” modification of samples since all the process is done before the training starts.

Due to the changes in decision boundaries during the training process, samples that make a model’s optimization struggling are not constant during training. In this line of thought, we also propose the “online” version of our method. In this version, after each epoch in the training loop, the CCSP scores are calculated for all of the samples in the training dataset and $R$ samples with the lowest scores are removed from the training dataset just for the subsequent epoch. Note that the CCSP scores are calculated for all the training samples in each epoch, and we do not leave out the samples that were eliminated in the previous epochs.

It can be vividly understood that the offline version requires more time to be accomplished, while this has noticeably been diminished in the online version since the CCSP scores are calculated as a apart of the training process rather than the k-fold trainings prior to the main training procedure.

Two different datasets are used for evaluations, which are CIFAR-10 [22] and SVHN [23]. We train our model on CIFAR-10 for 200 epochs while train it on SVHN for 100 epochs since it converges faster. Also, PreActResNet18 is used as the base model. In the training, the initial learning rate is set to 0.1 and it is multiplied by 0.1 after 50% and 75% of epochs. Moreover, SGD with momentum=$0.9$ and weight decay=$5e-4$ is used for the optimization.

In adversarial training, standard PGD attack with 10 iterations and a single restart is used to generate the perturbations. The perturbation is initialized randomly in the range $[-\epsilon,\epsilon]$, and bounded in an $\ell_{\infty}$-ball with $\epsilon=\frac{8}{255}$. Also, attack step size is set according to $\alpha=2.5*\frac{\epsilon}{N}$, where $N$ is the number of iterations.

According to our “offline” proposed method, first, the dataset is divided into four folds. Each time, one fold is selected and the model is trained on three other folds. Then, the CCSP score is calculated for each sample in the selected fold using trained model. It is done once for each fold to obtain CCSP score for all samples. After that, samples are sorted based on their CCSP scores, and then, $R$ samples with the lowest CCSP scores are eliminated from the dataset. Please note that setting $R$ to zero is equivalent to using the base adversarial training, which is our baseline in this work.

Results are shown for different values of $R$ in Table I. As we see, robustness is increased by removing the useless samples. On the other hand, we also note that setting $R$ to a large value can remove the useful samples in addition to the hard samples. The best performance is achieved by setting $R$ equal to 500 in CIFAR-10 and 300 in SVHN.

Please note that the results are stable and reproducible. For instance, in multiple runs of the experiment where $R=0$ samples were removed (normal adversarial training), the standard deviation of clean and robust accuracy for the CIFAR-10 dataset is $0.16\%$ and $0.54\%$, respectively.

According to our “online” proposed method, the CCSP scores are calculated for each sample after each epoch. Then, the scores are sorted, and $R$ samples with lowest scores are removed only in the following epoch. $R=100$ is used in this part to avoid removing useful samples in addition to the hard ones. In addition, according to the adaptive removability of samples during training, removing 100 samples is sufficient.

Results are shown in Table II. Accordingly, this method also improves robustness over the baseline ($R=0$). So, we can conclude that employing online strategy also works well.

In addition to the softmax-based methods, the distance between HTS and conditional distribution of the classes can be used to identify them. Two main techniques in this regard are the Mahalanobis distance (MD) [13] and the Relative MD (RMD) [24]. These techniques fit a conditional Gaussian distribution $\mathcal{N}(\mu_{k},\Sigma)$ to the pre-logit features $h$ for a distribution with $K$ classes. The mean vector and covariance matrix are calculated as:

for $k=1,2,...,K$, where $N_{k}$ is the number of samples in the class with label $k$, and $N$ refers to the number of samples in the dataset. Please note that $\Sigma$ is shared among by all classes. Afterward, the distance of the input $x$ with pre-logits $h_{x}$ is calculated as:

where ${MD}_{0}(h_{x})$ represents the Mahalanobis distance of $h_{x}$ to a distribution fitted to the entire training dataset as $\mathcal{N}(\mu_{0},\Sigma_{0})$. $\mu_{0}$ and $\Sigma_{0}$ are calculated as:

According to these descriptions, we can use Absolute Relative Mahalanobis Distance (ARMD) as an alternative method to CCSP for detecting HTS. ARMD is defined as follows for sample $x$ with label $y$ and pre-logit features $h_{x}$:

By determining the ARMD score for each sample, we can sort the samples in accordance with the calculated scores. After that, we can remove the $R$ samples with the lowest scores to get rid of the hard samples similar to the proposed method with CCSP method.

The result of this approach can be seen in the Table III where a noticeable improvement over the baseline ($R=0$) can be observed in most cases. As a result, our method is not sensitive to the detection method.

An alternative option for removing the hard samples in our method is using them without any perturbation in training. This can be useful if all the samples in the dataset correlate well with their assigned label.

To investigate the effectiveness of this method, the experiment in section IV-C is repeated with clean training of hard samples instead of removing them from dataset. Results are shown in Table IV. According to this table, this method is effective in SVHN dataset, but it does not improve the baseline ($R=0$) in CIFAR-10. We believe that the reason for this observation is that the quality of some of the CIFAR-10 images is so substandard that even their clean training can be harmful as can be seen in Fig. 2.

In this section, we conduct an experiment to show the capability of model in learning the OOD samples during training, which can cause problems in adversarial training. For this purpose, we add 50 random samples from CIFAR-100 dataset to the CIFAR-10 training set. These samples are randomly chosen from classes that are not shared between CIFAR-10 and CIFAR-100, and they are randomly assigned to one of the ten classes in CIFAR-10. So, we can consider these samples as OOD samples in the training set.

Afterwards, the model is trained using the updated dataset for 50 epochs. Results show that model classifies 40 out of these 50 samples correctly in the last epoch of training. In other words, model can learn 80% of the OOD samples in training, which confirms our claims in this work. As a result, the dataset should be purified before training, as has been done in this study.

We have also made a comparison with some other recent defenses on CIFAR-10 dataset in Table V. The defense methods in this table are Dynamic [25], TRADES [19], MMA [26], standard adversarial training [8], and our online and offline proposed methods with the best $R$. Considering the fact that there is a trade-off between clean and robust accuracy [19], the results show that our method is competitive against other variants of AT. Please note again that our method have a different data-centric approach than those methods.