On the Over-Memorization During
Natural, Robust and Catastrophic Overfitting

Since Zhang et al. (2021) observed that DNNs have the capacity to memorize training patterns with random labels, a line of work has demonstrated the benefits of memorization in improving generalization ability (Neyshabur et al., 2017; Novak et al., 2018; Feldman, 2020; Yuan et al., 2023). The memorization effect (Arpit et al., 2017; Bai et al., 2021; Xia et al., 2021; 2023; Lin et al., 2022; 2023b) indicates that the DNNs prioritize learning patterns rather than brute-force memorization. In the context of multi-step AT, Dong et al. (2021) suggests that the cause of RO can be attributed to the model’s memorization of one-hot labels. However, the prior studies that adopt a unified perspective to understand overfitting across various training paradigms are notably scarce.

NO (Dietterich, 1995) is typically shown as the disparity in the model’s performance between training and test patterns. To address this issue, two fundamental approaches, data augmentation and regularization, are widely employed. Data augmentation artificially expands the training dataset by applying transformations to the original patterns, such as Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2018), AutoAugment (Cubuk et al., 2018) and RandomErasing (Zhong et al., 2020). On the other hand, regularization methods introduce explicit constraints on the DNNs to mitigate NO, including dropout (Wan et al., 2013; Ba & Frey, 2013; Srivastava et al., 2014), stochastic weight averaging (Izmailov et al., 2018), and stochastic pooling (Zeiler & Fergus, 2013).

DNNs are known to be vulnerable to adversarial attacks (Szegedy et al., 2014), and AT has been demonstrated to be the most effective defence method (Athalye et al., 2018; Zhou et al., 2022). AT is generally formulated as a min-max optimization problem (Madry et al., 2018; Croce et al., 2022). The inner maximization problem tries to generate the strongest adversarial examples to maximize the loss, and the outer minimization problem tries to optimize the network to minimize the loss on adversarial examples, which can be formalized as follows:

where $(x,y)$ is the training dataset from the distribution $D$, $\ell(x,y;\theta)$ is the loss function parameterized by $\theta$, $\delta$ is the perturbation confined within the boundary $\epsilon$ shown as: $\Delta=\left\{\delta:\|\delta\|_{p}\leq\epsilon\right\}$.

For multi-step and single-step AT, PGD (Madry et al., 2018) and RS-FGSM (Wong et al., 2019) are the prevailing methods used to generate adversarial perturbations, where the $\Pi$ denotes the projection:

With the focus on DNNs’ robustness, overfitting has also been observed in AT. An overfitting phenomenon known as RO (Rice et al., 2020) has been identified in multi-step AT, which manifests as a gradual degradation in the model’s test robustness with further training. Further investigation found that the conventional remedies for NO have minimal effect on RO (Rice et al., 2020). As a result, a lot of work attempts to explain and mitigate RO based on its unique characteristics. For example, some research suggests generating additional adversarial patterns (Carmon et al., 2019; Gowal et al., 2020), while others propose techniques such as adversarial label smoothing (Chen et al., 2021; Dong et al., 2021) and adversarial weight perturbation (Wu et al., 2020; Yu et al., 2022a; b). Meanwhile, another type of overfitting termed CO (Wong et al., 2019) has been identified in single-step AT, characterized by the model’s robustness against multi-step adversarial attacks will abruptly drop from peak to nearly 0%. Recently studies have shown that current approaches for addressing NO and RO are insufficient for mitigating CO (Andriushchenko & Flammarion, 2020; Sriramanan et al., 2021). To eliminate this strange phenomenon, several approaches have been proposed, including constraining the weight updates (Golgooni et al., 2023; Huang et al., 2023a) and smoothing the adversarial loss surface (Andriushchenko & Flammarion, 2020; Sriramanan et al., 2021; Lin et al., 2023a).

Although the aforementioned methods can effectively address NO, RO and CO separately, the understanding and solutions for these overfitting types remain isolated from each other. This study reveals a shared DNN behaviour termed over-memorization. Based on this finding, we propose the general framework DOM aiming to holistically address overfitting across various training paradigms.

To begin, we explore the natural overfitting (NO) by investigating the model’s memorization effect. As illustrated in Figure 1 (left), we can observe that shortly after the first learning rate decay (150th epoch), the model occurs NO, resulting in a 5% performance gap between training and test patterns. Then, we conduct a statistical analysis of the model’s training loss on each training pattern, as depicted in Figure 1 (middle). We observe that aligned with the onset of NO, the proportion of the model’s high-confidence (loss range 0-0.2) prediction patterns suddenly increases by 20%. This observation prompted us to consider whether the decrease in DNNs’ generalization ability is linked to the increase in high-confidence training patterns. To explore the connection between high-confidence patterns and NO, we directly removed these patterns (All-HC) from the training process after the first learning rate decay. As shown in Figure 1 (right), there is a noticeable improvement (4%) in the model’s generalization capability, with the generalization gap shrinking from 4.84% to 4.63%. This finding indicates that continuous learning on these high-confidence patterns may not only fail to improve but could actually diminish the model’s generalization ability.

their individual influence, as shown in Figure 1 (right). We can observe that only removing the original high-confidence (Ori-HC) patterns negatively affects the model’s generalization (5.57%), whereas only removing the transformed high-confidence (Trans-HC) patterns can effectively alleviate NO (4.42%). Therefore, the primary decline in the model’s generalization can be attributed to the learning of these transformed high-confidence patterns. Additionally, we note that the model exhibits an uncommon memory capacity for transformed high-confidence patterns, as illustrated in Figure 2. Our analysis suggests that, compared to the original ones, DNNs show a notably persistent memory for these transformed high-confidence patterns. This uncommon memory is evidenced by a barely increase (0.01) in training loss after their removal from the training process. Building on these findings, we term this behaviour as over-memorization, characterized by DNNs suddenly becoming high-confidence predictions and retaining a persistent memory for certain training patterns, which weakens their generalization ability.

In this section, we explore the over-memorization behaviour in robust overfitting (RO) and catastrophic overfitting (CO). During both multi-step and single-step adversarial training (AT), we notice that similar to NO, the model abruptly becomes high-confidence in predicting certain adversarial patterns with the onset of RO and CO, as illustrated in Figure 3 (1st and 2nd). Meanwhile, directly removing these high-confidence adversarial patterns can effectively mitigate RO and CO, as detailed in Section 4.2. Therefore, the combined observations suggest a shared behaviour that the over-memorization of certain training patterns impairs the generalization capabilities of DNNs.

From Figure 3 (4th), we can observe a significantly high overlap rate (90%) between the high-confidence predicted natural and adversarial patterns. This observation suggests that when the model over-memorizes an adversarial pattern, it tends to simultaneously exhibit high-confidence in predicting the corresponding natural pattern. We also conduct the same experiment in TRADES (Zhang et al., 2019), which encounters natural patterns during the training process, and reaches the same observation, as shown in Appendix B. To further validate this similar memory tendency, we attempt to detect the high-confidence adversarial pattern solely based on their corresponding natural training loss. From Figure 4, we are able to clearly distinguish the high-confidence and low-confidence adversarial patterns by classifying their natural training loss. Therefore, by leveraging this tendency, we can reliably and consistently identify the over-memorization pattern by exclusively focusing on the natural training loss, regardless of the training paradigm.

Building on the above findings, we propose a general framework, named Distraction Over-Memorization (DOM), which is designed to proactively prevent the model from over-memorization training patterns, thereby eliminating different types of overfitting. Specifically, we first establish a fixed loss threshold to identify over-memorization patterns. Importantly, regardless of the training paradigm, DOM exclusively compares the natural training loss with this established threshold. Subsequently, our framework employs two mainstream operations to validate our perspective: removal and data augmentation denoted as $\mathrm{DOM_{RE}}$ and $\mathrm{DOM_{DA}}$, respectively. For $\mathrm{DOM_{RE}}$, we adopt a straightforward approach to remove all high-confidence patterns without distinguishing over-memorization and normal-memorization. This depends on the observation that DNNs exhibit a significantly persistent memory for over-memorization patterns, as evidenced in Figure 2. As training progresses, we expect the loss of normal-memorization patterns to gradually increase, eventually surpassing the threshold and prompting the model to relearn. In contrast, the loss for over-memorization patterns is unlikely to notably increase with further training, hindering their likelihood of being relearned.

On the other hand, $\mathrm{DOM_{DA}}$ utilizes data augmentation techniques to weaken the model’s confidence in over-memorization patterns. Nonetheless, research by Rice et al. (2020); Zhang et al. (2022) have shown that the ability of original data augmentation is limited for mitigating RO and CO. From the perspective of over-memorization, we employ iterative data augmentation on high-confidence patterns to maximization reduce the model’s reliance on them, thereby effectively mitigating overfitting. The implementation of the proposed framework DOM is summarized in Algorithm 1.

Data Argumentation.  The standard data augmentation techniques random cropping and horizontal flipping are applied in all configurations. For $\mathrm{DOM_{DA}}$, we use two popular techniques, AUGMIX (Hendrycks et al., 2019) and RandAugment (Cubuk et al., 2020).

Adversarial Paradigm.  We follow the widely-used configurations, setting the perturbation budget as $\epsilon=8/255$ and adopting the threat model as $L_{\infty}$. For adversarial training, we employ the default PGD-10 (Madry et al., 2018) and RS-FGSM (Wong et al., 2019) to generate the multi-step and single-step adversarial perturbation, respectively. For the adversarial test, we use the PGD-20 and Auto Attack (Croce & Hein, 2020) to evaluate model robustness.

Datasets and Model Architectures.  We conducted extensive experiments on the benchmark datasets Cifar-10/100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011) and Tiny-ImageNet (Netzer et al., 2011). The settings and results for SVHN and Tiny-ImageNet are provided in Appendix C and Appendix D, respectively. We train the PreactResNet-18 (He et al., 2016), WideResNet-34 (Zagoruyko & Komodakis, 2016) and ViT-small (Dosovitskiy et al., 2020) architectures on these datasets by utilizing the SGD optimizer with a momentum of 0.9 and weight decay of 5 × $10^{-4}$. The results of ViT-small can be found in Appendix E. Other hyperparameters setting, including learning rate schedule, training epochs E, warm-up epoch $\mathcal{K}$, loss threshold $\mathcal{T}$, data augmentation strength $\beta$ and data augmentation iteration $\gamma$ are summarized in Table 1. We also evaluate our methods on the gradual learning rate schedule, as shown in Appendix F.

Natural Training Results.  In Table 2, we present an evaluation of the proposed framework against competing baselines on CIFAR-10/100 datasets. We report the test accuracy at both the highest (Best) and final (Last) checkpoint during training, as well as the generalization gap between them (Diff). Firstly, we can observe that $\mathrm{DOM_{RE}}$, which is trained on a strict subset of natural patterns, can consistently outperform baselines at the final checkpoint. Secondly, $\mathrm{DOM_{DA}}$ can achieve superior performance at the both highest and final checkpoints. It’s worth noting that $\mathrm{DOM_{DA}}$ applies data augmentation to limited epochs and training patterns. Finally and most importantly, both $\mathrm{DOM_{RE}}$ and $\mathrm{DOM_{DA}}$ can successfully reduce the model’s generalization gap, which substantiates our perspective that over-memorization hinders model generalization, and preventing it can alleviate overfitting.

Adversarial Training Results.  To further explore the over-memorization, we extend our framework to both multi-step and single-step AT. Importantly, the detection of over-memorization adversarial patterns relies exclusively on the loss of the corresponding natural pattern. From Table 3, it’s evident that both $\mathrm{DOM_{RE}}$ and $\mathrm{DOM_{DA}}$ are effective in eliminating RO under PGD-20 attack. However, under Auto Attack, the $\mathrm{DOM_{DA}}$ remains its superior robustness, whereas $\mathrm{DOM_{RE}}$ is comparatively weaker. This difference in Auto Attack could be attributed to $\mathrm{DOM_{RE}}$ directly removing training patterns, potentially ignoring some useful information. Table 4 illustrates that both $\mathrm{DOM_{RE}}$ and $\mathrm{DOM_{DA}}$ are effective in mitigating CO. However, the proposed framework shows its limitation in preventing CO when using $\mathrm{DOM_{DA}}$ with AUGMIX on CIFAR100. This result could stem from the weakness of the original data augmentation method, which remains inability to break over-memorization even after the framework’s iterative operation.

Overall Results.  In summary, the DOM framework can effectively mitigate different types of overfitting by consistently preventing the shared behaviour over-memorization, which first-time employs a unified perspective to understand and address overfitting across different training paradigms.

In this section, we investigate the impacts of algorithmic components using PreactResNet-18 on CIFAR10. For the loss threshold and warm-up epoch selection, we employ $\mathrm{DOM_{RE}}$, while for data augmentation strength and iteration selection, we use $\mathrm{DOM_{DA}}$ with AUGMIX in the context of NT. When tuning a specific hyperparameter, we keep other hyperparameters fixed.

Loss Threshold Selection.  To investigate the role of loss threshold, we present the variations in test error across three training paradigms. As depicted in Figure 5 (a: left), we can observe that employing a small threshold might not effectively filter out over-memorization patterns, resulting in suboptimal generalization performance. On the other hand, adopting a larger threshold might lead to the exclusion lot of training patterns, consequently resulting in the model underfitting. In light of this trade-off, we set the loss threshold as 0.2 for NT. Interestingly, this trade-off does not seem to exist in the context of AT, where higher loss thresholds tend to result in higher PGD robustness, as shown in Figure 5 (a: middle and right). Nevertheless, the above experiments indicate that this approach could also increase the vulnerability to Auto Attack. Hence, determining an appropriate loss threshold is critical for all training paradigms. We also evaluate our methods on unified adaptive loss threshold (Berthelot et al., 2021; Li et al., 2023) as shown in Appendix G.

Warm-Up Epoch Selection.  The observations from Figure 5 (b: left) indicate that a short warm-up period might hinder the DNNs from learning essential information, leading to a decline in the performance. Conversely, a longer warm-up period cannot promptly prevent the model from over-memorizing training patterns, which also results in compromised generalization performance. Based on this observation, we simply align the warm-up epoch with the model’s first learning rate decay.

Data Augmentation Strength and Iteration Selection.  We also examine the impact of data augmentation strengths and iterations, as shown in Figure 5 (b: middle and right). We can observe that, even when the augmentation strength is set to 0% or the number of iterations is limited to 1, our approach can still outperform the baseline (AUGMIX). Moreover, both insufficient (weak strengths or few iterations) and aggressive (strong strengths or excessive iterations) augmentations will lead to subpar performance. This is due to insufficient augmentations limiting the pattern transformation to diverse styles, while aggressive augmentations could exacerbate classification difficulty and even distort the semantic information (Bai et al., 2022; Huang et al., 2023b). Therefore, we select the augmentation strength as 50% and iteration as 3 to achieve the optimal performance. The computational overhead analysis can be found in the Appendix H.