Understanding Robust Overfitting from the Feature Generalization Perspective

Let $f_{\theta}$, $\mathcal{X}$ and $\ell$ denote the network $f$ with model parameter $\theta$, the input space, and the loss function, respectively. Given a $C$-class dataset $\mathcal{S}=\{(x_{i},y_{i})\}^{n}_{i=1}$, where $x_{i}\in\mathcal{X}$ and $y_{i}\in\mathcal{Y}=\{0,1,\dots,C-1\}$ denotes its corresponding label, the objective function of natural training is defined as follows:

where the network $f_{\theta}$ learns features in natural data that are correlated with associated labels to minimize the empirical risk of misclassifying natural inputs. However, evidence [33, 36, 17] suggests that naturally trained models tend to learn fragile features that are vulnerable to adversarial attacks. To address this issue, AT [14, 26] introduces adversarial perturbations to each data point by transforming $\mathcal{S}=\{(x_{i},y_{i})\}^{n}_{i=1}$ into $\mathcal{S}^{\prime}=\{(x_{i}^{\prime}=x_{i}+\delta_{i},y_{i})\}^{n}_{i=1}$. The adversarial perturbations $\{\delta_{i}\}^{n}_{i=1}$ are constrained by a pre-specified budget, i.e. $\{\delta\in\Delta:||\delta||_{p}\leq\epsilon\}$, where $p$ can be $1,2,\infty$, etc. Therefore, the objective of AT is to solve the following min-max optimization problem:

where the inner maximization process aims to generate adversarial data that maximizes the classification loss, and the outer minimization process optimizes model parameter using the generated adversarial data. This iterative procedure aims to train an adversarially robust classifier. The most commonly employed approach for generating adversarial data is Projected Gradient Descent (PGD) [26], which applies adversarial attack to natural data $x_{i}$ over multiple steps $k$ with a step size of $\alpha$:

where $\delta^{k}$ represents the adversarial perturbation at step $k$, and $\Pi_{\Delta}$ denotes the projection operator.

Besides standard AT, there are also several AT variants [20, 44, 39]. One typical variant is TRADES [44], which trains the network on both natural data and adversarial data:

where CE is the cross-entropy loss that encourages the network to maximize natural accuracy, KL is the Kullback-Leibler divergence that encourages robustness improvement, and the hyperparameter $\beta$ is used to regulate the tradeoff between natural accuracy and adversarial robustness.

Moreover, further improvements to adversarial training have also been explored, such as curriculum adversarial training [5, 38], ensemble adversarial training [35, 28], and adaptive budget schedule [24, 1, 19].

RO was initially observed in standard AT [26]. Subsequent systematic studies by Rice et al. [30] showed that traditional remedies for overfitting in deep learning offer limited help in addressing RO in AT, prompting further efforts to explore RO. Schmidt et al. [31] attributed RO to sample complexity theory and suggested that robust generalization requires more training data, which is supported by empirical results in derivative works [6, 2, 43]. Other works also proposed various strategies to mitigate RO without relying on additional training data, such as sample reweighting [39, 45, 25], label smoothing [18], stochastic weight averaging [7], temporal ensembling [12], knowledge distillation [7], weight regularization [40, 41], and data augmentation [29, 34, 22]. Additionally, RO has been studied from various perspectives, such as label noise [11], input-loss landscape  [23] and minimax game [37]. In this work, we provide a novel feature generalization understanding of RO.

Drawing inspiration from the data ablation experiments conducted by Yu et al. [41], which investigated the influence of small-loss training data on RO by excluding such data during training, we propose factor ablation adversarial training as a method to pinpoint the inducing factors of RO. Following a similar experimental setup, we also utilize a fixed loss threshold to differentiate small-loss training data. For instance, in the CIFAR10 dataset, data with an adversarial loss below 1.7 is regarded as small-loss training data. Unlike data ablation experiments that treat adversarial data as a single entity, our approach considers natural data and adversarial perturbations from small-loss training data as separate factors. We conduct more granular factor ablation adversarial training to identify the specific factor inducing RO. Specifically, we train a PreAct ResNet-18 model on CIFAR-10 dataset under the $\ell_{\infty}$ threat model and remove designated factors before RO occurs (i.e., at the 100th epoch). These experiments include: i) baseline, which is a baseline group where no factors are removed; ii) data & perturbation, which removes both natural data and adversarial perturbations from small-loss training data; and iii) perturbation, which solely removes adversarial perturbations from small-loss training data.

It’s important to note that these experiments strictly adhere to the principle of controlling variables. They are entirely identical before RO occurred, with the sole distinction among different experimental groups being the removal of designated factors from the training set. The experimental results of factor ablation adversarial training are summarized in Figure 1(a). It is observed that the data & perturbation group demonstrates a significant alleviation of RO, whereas both the baseline and perturbation groups continue to exhibit severe RO. Given that the only difference between the data & perturbation and perturbation groups lies in the inclusion of natural data in the training set, we can infer that the inducing factor of RO stems from natural data. Similar effects can be observed across different datasets, network architectures, and AT variants (see Appendix A), indicating that it is a general finding in adversarial training.

Based on the factor ablation experiments in Section 3.1, we know that natural data acts as the inducing factor of RO. It’s noteworthy that natural data is also present in natural training. However, in natural training, RO phenomenon is typically absent. For instance, deep learning models trained on natural data often achieve zero training error, effectively memorizing the training data, without observable detrimental effects on generalization performance [30]. In contrast, RO is a dominant phenomenon in AT, which commonly exhibits significantly degraded generalization performance. Given that the only difference between adversarial and natural training lies in the inclusion of adversarial perturbations in the natural data, a natural hypothesis arises: adversarial perturbations may degrade the generalization of features in natural data and lead to the emergence of RO.

Verification. To validate the aforementioned hypothesis, before generating training adversarial data in AT, we introduce an additional adversarial perturbation into the training natural data and observe the model’s robust generalization performance. Specifically, before the onset of RO (i.e., between the 50th and 100th epochs), we incorporate an extra adversarial perturbation in the opposite direction into the training natural data and observe the model’s test robustness. We conduct experiments with varying budgets of additional adversarial perturbations, such as $\epsilon=\{0/255,2/255,4/255,6/255,8/255\}$. The model’s test robustness is shown in Figure 1(b). It is important to note that the additional adversarial perturbations are introduced only on the training natural data, while the model and the test data remain unchanged, ensuring that the model’s robustness on the test data remains unaffected. However, we observe that as the additional adversarial perturbation is incorporated, the model’s test robustness deteriorates, indicating that the additional adversarial perturbations exert a detrimental effect on feature generalization. Furthermore, we linearly increase the budget of additional adversarial perturbations between the 50th and 100th epochs, and the model’s test robustness is summarized in Figure 1(c). It is observed that the RO phenomenon can be simulated by adjusting the adversarial perturbations on training data. In Appendix B, we also provide more experimental results conducted at different stages of training. These experimental results support our hypothesis that the adversarial perturbations can degrade feature generalization and induce the emergence of RO.

A holistic view of RO from the feature generalization perspective. In standard AT, there exists a robustness gap between the training and test data before the onset of RO, as shown in Figure 1(d). This robustness gap arises due to factors such as the memorization effect of deep networks [3, 12] and the distribution deviation between the finite training and test data. Certain small-loss training data maintain a significant robustness gap when compared to the test data. However, adversarial perturbations are generated on the fly and adaptively adjusted based on the model’s robustness. For instance, consider a single data point, when the model’s robustness is poor, adversarial perturbations mainly impede the model’s learning of non-robust features. Conversely, when the model’s robustness is strong, adversarial perturbation will hinder the model’s learning of robust features because the process of generating adversarial perturbation aims to maximize the classification loss. The robustness gap between training and test data leads to distinct adversarial perturbations for the features in natural data, degrading feature generalization. As feature generalization deteriorates, the model’s adversarial robustness on the test data progressively declines, and the robustness gap between training and test data continues to widen, forming a vicious cycle, as illustrated in Figure 2.

Based on the analysis above, some small-loss training data maintain a significant robustness gap compared to the test data. This can explain why removing small-loss adversarial data from the training data can effectively prevent RO. Furthermore, AT exhibits some other empirical behaviors associated with RO, for instance, as the perturbation budget increases, the extent of RO initially rises and then decreases. Our feature generalization perspective provides an intuitive explanation: as the perturbation budget increases from 0, the impact of adversarial perturbations on the generalization of features in natural data also grows, leading to a higher probability of RO phenomenon during AT. This can explain why natural training does not exhibit RO, and as the adversarial perturbation budget increases, the extent of RO also rises. However, with a further increase in the perturbation budget, the model’s robustness on the training data is limited, leading to a narrowing robustness gap between training and test data. The narrowing robustness gap reduces the generation of adversarial perturbations that degrade feature generalization. Therefore, the extent of RO gradually decreases.

In Section 3.2, we analyze RO as a result of the degradation of Feature Generalization (ROFG), which arises from the distinct adversarial perturbations for the features in natural data. In this part, we introduce two approaches to examine our feature generalization understanding of RO.

ROFG through attack strength. In order to verify our analysis that adversarial perturbations on some small-loss training data degrade feature generalization, we conduct AT with varying attack strengths to generate adversarial perturbations on the small-loss training data before RO occurs (i.e., at the 100th epoch). Specifically, we use a fixed loss threshold $t=1.7$ to distinguish small-loss training data and train PreAct ResNet-18 on CIFAR10 under the $\ell_{\infty}$ threat model with adjusted perturbation budgets $\epsilon_{a}$ on small-loss training data, ranging from $0/255$ to $24/255$. The pseudocode is provided in Algorithm 1, which utilizes attack strength to adjust the adversarial perturbations on small-loss training data, referred to as ROFG_AS.

For ROFG_AS with different perturbation budgets $\epsilon_{a}$, we evaluate the model’s training and test robustness under the standard perturbation budget of $\epsilon=8/255$, with the learning curves summarized in Figure 3(a). We observe a clear correlation between the applied perturbation budgets $\epsilon_{a}$ and the extent of RO. Specifically, the greater the perturbation budgets $\epsilon_{a}$, the milder the extent of RO. When the perturbation budget $\epsilon_{a}$ exceeds a certain threshold, such as $16/255$, the model exhibits almost no RO. It is worth noting that similar patterns are also observed across different datasets, network architectures, and AT variants (as shown in Appendix C). These experimental results demonstrate that adjusting adversarial perturbations on small-loss training data can effectively prevent the degradation of feature generalization, thereby alleviating the RO phenomenon, validating our feature generalization understanding of RO.

ROFG through data augmentation. To further support our analysis that the model’s robustness gap between training and test data promotes the generation of adversarial perturbations that degrades feature generalization, we employ data augmentation techniques to weaken the model’s robustness on small-loss training data before RO occurs (i.e., at the 100th epoch). However, the image transformations in data augmentation techniques are often stochastic and may not yield the desired effect. To address this issue, we iteratively apply random image transformations to the small-loss training data, consistently selecting the transformed data that does not fall within the small-loss scope. Specifically, we use a fixed loss threshold $t=1.7$ to distinguish small-loss training data, and set a threshold $p$ for the proportion of small-loss training data when train PreAct ResNet-18 on CIFAR10 under the $\ell_{\infty}$ threat model. At the beginning of each iteration, we check whether the proportion of small-loss training data meets the specified threshold $p$. If this proportion is below the specified threshold $p$, we apply random image transformations to the small-loss training data until the specified threshold $p$ is reached. The pseudocode is provided in Algorithm 2, where the adopted data augmentation technique is AugMix [16]. We refer to this adversarial training pipeline, empowered by the data augmentation technique, as ROFG_DA.

The learning curves of ROFG_DA with different thresholds $p$ are summarized in Figure 3(b). We observe a clear correlation between the threshold $p$ and the extent of RO. As the robustness gap between small-loss training data and test data decreases, the extent of RO becomes increasingly mild. Furthermore, the observed effects are consistent across different datasets, network architectures, and AT variants (as shown in Appendix D). These experimental results show that narrowing the model’s robustness gap between training and test data can effectively prevent the degradation of feature generalization, supporting our feature generalization understanding of RO.

We investigate the impacts of algorithmic components in ROFG_AS and ROFG_DA under the PreAct ResNet-18 architecture on the CIFAR10 dataset.

The impact of attack strength. We empirically verify the effectiveness of attack strength component in ROFG_AS by comparing the performances of models trained using different perturbation budgets $\epsilon_{a}$. Specifically, we conduct ROFG_AS with the perturbation budget $\epsilon_{a}$ ranging from 0/255 to 24/255, and the results are summarized in Figure 4(a). It is evident that as $\epsilon_{a}$ increases, the robustness gap between the “Best” and “Last” checkpoints steadily decreases, demonstrating its effectiveness. For adversarial robustness, we observe a trend of initially increasing and then decreasing. This is not surprising since the attack strength component with excessively large perturbation budgets is inherently detrimental to robustness improvement under the standard perturbation budget ($\epsilon=8/255$). Therefore, in practice, we need to adjust the perturbation budget $\epsilon_{a}$ to balance the mitigation of RO and robustness improvement.

The importance of loss threshold $t$. Identifying exactly which adversarial perturbations degrade feature generalization is generally intractable. To address this, we introduce a fixed loss threshold $t$, which is used to distinguish small-loss training data. We conduct ROFG_AS experiments with $t$ varying from 1.2 to 2.0, and the model’s robustness performances are summarized in Figure 4(b). We observe a trend of initially increasing and then decreasing robustness, which implies that adversarial perturbations degrading feature generalization are primarily concentrated in the small-loss training data, suggesting the importance of the loss threshold $t$.

The effect of data augmentation. We further investigate the effect of the data augmentation component by conducting ROFG_DA with different thresholds $p$. The results are summarized in Figure 4(c). It is observed that as $p$ decreases, the robustness gap between the "Best" and "Last" checkpoints steadily shrinks. Additionally, we explore the use of different data augmentation techniques to implement ROFG_DA in Appendix F. It is observed that all implementations of ROFG_DA can effectively mitigate RO, suggesting that the proposed approach is generally effective regardless of the chosen data augmentation technique. For adversarial robustness, as $p$ decreases, the model’s robustness performance initially increases and then decreases. This trend may be due to the data augmentation component weakening the model’s robustness on the training data, thereby degrading the model’s defense against strong attacks.

The proposed ROFG_AS and ROFG_DA methods can effectively mitigate RO, as demonstrated in Section 3.3, Appendix C, and Appendix D, validating our feature generalization perspective. Regarding adversarial robustness, we conduct experiments under the PreAct ResNet-18 architecture across different datasets and adversarial training methods. The performance of the “Best” and “Last” checkpoints is provided in Table 1. It is observed that for methods exhibiting RO, such as AT [26] and TRADES [44], the proposed methods significantly improve model robustness. For methods already mitigating RO, such as AWP [40] and MLCAT [41], our methods still can achieve some complementary robustness improvement, demonstrating the significance of our feature generalization perspective. In Table 1, we can see that ROFG_AS outperforms ROFG_DA under AA in all settings. Thus we further conduct experiments using ROFG_AS under the Wide ResNet-34-10 architecture, with the results shown in Table 2. It is observed that under the stronger backbone network, the ROFG_AS method significantly enhances model robustness across different datasets and adversarial training methods. Additionally, we conduct additional experiments using ROFG_AS on the Tiny-ImageNet dataset [13] and VGG-16 architecture [32], with the results shown in Appendix G. Consistently, the proposed approach mitigates RO and boosts adversarial robustness, demonstrating its effectiveness.

While the proposed methods contribute to validating our feature generalization perspective, they have certain limitations. Firstly, they introduce additional components into the standard AT pipeline, thereby increasing computational complexity. For instance, under a single GeForce RTX 3080 GPU, the average training time per epoch in standard AT is 147.7s, while for ROFG_AS and ROFG_DA, it extends to 261.6s and 1222.9s, respectively. Moreover, our methods have yet to achieve state-of-the-art adversarial robustness, and they often cannot simultaneously achieve optimal robustness improvement and RO mitigation, as discussed in Section 4.1. However, it is worth noting that the main focus of this work is to provide a novel feature generalization understanding of RO. These methods are specifically tailored to verify our feature generalization perspective, and these limitations essentially do not compromise our understanding of RO.