Pushing Boundaries:
Mixup’s Influence on Neural Collapse

Mixup, a data augmentation strategy proposed by Zhang et al. (2017), generates new training examples through convex combinations of existing data points and labels:

where $\lambda\in[0,\,1]$ is a randomly sampled value from a predetermined distribution $D_{\lambda}$. Conventionally, this distribution is a symmetric $\operatorname{Beta}(\alpha,\alpha)$ distribution, with $\alpha=1$ frequently set as the default. The loss associated with mixup can be mathematically represented as:

A specific mixup data point, represented as $\mathbf{x}_{ii^{\prime}}^{\lambda}$, is categorized as a same-class mixup point when $\mathbf{y}_{i}=\mathbf{y}_{i^{\prime}}$, and classified as different-class when $\mathbf{y}_{i}\neq\mathbf{y}_{i^{\prime}}$.

Despite the widespread use and demonstrated efficacy of the mixup data augmentation strategy in enhancing the generalization and calibration of deep neural networks, its underlying operational mechanisms remain not well understood. The emergence of Neural Collapse prompts the following question:

Does mixup induce its own distinct configurations in last-layer activations, differing from traditional Neural Collapse? If so, does the configuration contribute to the method’s success?

This study aims to uncover the potential geometric configurations in the last-layer activations resulting from mixup and to determine whether these configurations can offer insights into its success.

Our contributions in this paper are twofold.

The results of our extensive empirical investigation are presented in Figures 1, 3, 5, 10, and 12. These figures collectively illustrate a consistent identification of a unique last-layer configuration induced by mixup, observed across a diverse range of:

Architectures:

Our study incorporated the WideResNet-40-10 (Zagoruyko & Komodakis, 2017) and ViT-B (Dosovitskiy et al., 2021) architectures;

Datasets:

The datasets employed included FashionMNIST (Xiao et al., 2017), CIFAR10, and CIFAR100 (Krizhevsky & Hinton, 2009);

Optimizers:

We used stochastic gradient descent (SGD), Adam (Kingma & Ba, 2017), and AdamW (Loshchilov & Hutter, 2017) as optimizers.

The networks trained showed good generalization performance and calibration, as substantiated by the data presented in Tables 1 and 2. That is, the values are comparable to those found in other papers (Zhang et al., 2017; Thulasidasan et al., 2020).

Beyond our principal observations, we conducted a counterfactual experiment, the results of which are depicted in Figures 2 and 8. These reveal a notable divergence in the configuration of the last-layer features when mixup is not employed. They also show that for MSE loss, the last-layer activations are convex combinations of the classifiers, which one may expect.

To complement these results, we train models using mixup while fixing the classifier as a simplex ETF, and we plot the last-layer features in Figure 7. This yields last-layer features that align more closely with the theoretical features.

We consider FashionMNIST (Xiao et al., 2017), CIFAR10, and CIFAR100 (Krizhevsky & Hinton, 2009) datasets. Unless otherwise indicated, for all experiments using mixup augmentation, $\alpha{=}1$ was used, meaning $\lambda$ was sampled uniformly between 0 and 1. Each dataset is trained on both a Vision Transformer (Dosovitskiy et al., 2021), and a wide residual network (Zagoruyko & Komodakis, 2017). For each network and dataset combination, the experiment with the highest test accuracy is repeated without mixup and is referred to as the “baseline” result. No dropout was used in any experiments. Hyperparameter details are outlined in Appendix B.1.

For each dataset and network pair, we visualize the last-layer activations for a subset of the training dataset consisting of three randomly selected classes. After obtaining the last-layer activations, they undergo a two-step projection: first onto the classifier for the subset of three classes, then onto a two-dimensional representation of a three-dimensional simplex ETF. A more detailed explanation of the projection can be found in Appendix B.2. The results of this experiment can be seen in Figure 1. Notably, activations from mixed-up examples of the same classes closely align with a simplex ETF structure, whereas those from different classes delineate channels along the decision boundary. Additionally, in certain plots, activations from mixed-up examples of different classes become increasingly sparse as $\lambda$ approaches 0 and 1. This suggests a clustering of activations towards the channels. When generating plots, we keep the network in train mode ¹¹1We choose to have the network in evaluation mode for the WideResNet-40-10 CIFAR100 combination because the batch statistics are highly skewed due to the high number of classes.. Since ViT does not have batch normalization layers, this difference is not applicable.

Using the same projection method as in Figure 1, we investigate the trajectory of the CLS token for ViT models. First, we randomly select two CIFAR10 training images. Then, we create a selection of mixed up examples using the respective images. For each mixed up example, we project the path of the CLS token at each layer of the ViT-B/4 network.

Figure 3 presents the results for two images of the same class, and two images of a different class. For different-class mixup, the plot shows that for very small $\lambda$, the network first classifies the image as class 1 and only in deeper layers it realizes the image is also partially class 2.

The results in Figure 1 suggest that applying mixup to input data enforces a particularly rigid geometric structure on the last-layer activations. Manifold mixup (Verma et al., 2019), a subsequent technique, proposes the mixing of features across various layers of a network. The results in Figure 3 suggest that using regular mixup promotes manifold mixup-like behaviour in previous layers.

Thulasidasan et al. (2020) demonstrated that mixup improves calibration for networks. That is, training with mixup causes the softmax probabilities to be in closer alignment with the true probabilities of misclassification. To measure a network’s calibration, we use the expected calibration error (ECE) as proposed by Pakdaman Naeini et al. (2015). The exact definition of ECE can be found in Appendix C. Results for ECE can be found in Table 2. Last-layer activation plots for $\alpha=0.4$ are available in Figure 9 in Appendix D.

The configuration presented in Figure 1 sheds light on why mixup improves calibration. Recall, mixup promotes alignment of the model’s softmax probabilities for the training example $\mathbf{x}_{ii^{\prime}}^{\lambda}$ with its label $\lambda\mathbf{y}_{i}+(1-\lambda)\mathbf{y}_{i^{\prime}}$. Here, $\lambda$ acts as a gauge for these softmax probabilities, essentially reflecting the model’s confidence in its predictions. Turning to Figure 4, it becomes therefore evident that as $\lambda$ nears 0.5, the model’s certainty in its predictions diminishes. This reduction in confidence is manifested geometrically through the spatial distribution of features along the channel. This, in turn, causes an increase in misclassification rates, due to a greater chance of activations erroneously crossing the decision boundary. This simultaneous reduction in confidence and classification accuracy leads to enhanced calibration in the model and is purely attributed to the geometric structure to which the model converged. The above logic holds as we traverse the train mixed up features but we expect some test features to be noisy perturbations of train mixed up features.

To study the resulting last-layer features under mixup, we consider an adaptation of the unconstrained features model to mixup training. Let $d\geq C-1$ be the dimension of the last-layer features, ${\bm{y}}_{i}\in\mathbb{R}^{C}$ be the one-hot vector in entry $i$, ${\bm{W}}\in\mathbb{R}^{C{\times}d}$ be the classifier, and ${\bm{h}}_{ii^{\prime}}^{\lambda}\in\mathbb{R}^{d}$ be the last-layer feature associated with target $\lambda{\bm{y}}_{i}+\left(1-\lambda\right){\bm{y}}_{i^{\prime}}$. Then, adapting Equation 1 to the unconstrained features setting, we consider the optimization problem given by

where $\lambda_{{\bm{W}}},\lambda_{{\bm{H}}}>0$ are the weight decay parameters. It is reasonable to consider decay on the features ${\bm{h}}_{ii^{\prime}}^{\lambda}$, a practice that is frequently observed in prior work (Zhu et al., 2021; Zhou et al., 2022), due to the implicit decay to the last-layer features from the inclusion of decay in the previous layers’ parameters.

The following theorem characterizes the optimal last-layer features under the assumption that the optimal classifier ${\bm{W}}$ is a simplex ETF, ie.

where ${\bm{I}}_{C}\in\mathbb{R}^{C{\times}C}$ is the identity, $\mathbf{1}_{C}\in\mathbb{R}^{C}$ is the ones vector, $\mathbf{U}\in\mathbb{R}^{d\times C}$ is a partial orthogonal matrix (satisfying $\mathbf{U}^{\top}\mathbf{U}={\bm{I}}_{C}$), and $m\in\mathbb{R}\setminus\{0\}$ is its multiplier.

Note that we make this assumption as it holds in practice based on our empirical measurements illustrated in Figure 5.

The proof of Theorem 3.1 can be found in Appendix A.1

In Figure 6, we plot the last-layer features obtained from Theorem 3.1, numerically solving for the values of $K$ and $K_{\lambda}$ that satisfy their respective equations.

Similar to the empirical results in Figure 1, the density of different-class mixup points decreases as lambda approaches 0 and 1. However, the theoretically optimal features exhibit channels arranged in a hexagonal pattern, differing from the empirical features observed in the FashionMNIST and CIFAR10 datasets. In particular, in the empirical representations, there is a more pronounced elongation of different-class features as the mixup parameter $\lambda$ approaches $0.5$. In attempt to understand these differences, we introduce an amplification of these same features in the directions of the classifier rows not corresponding to the mixed-up targets, with increasing amplifications as $\lambda$ gets closer to 0.5 (details of the amplification function is outlined in Appendix A.2). This results in the plot on the right that behaves more similarly to the other empirical outcomes, while achieving a very close (though marginally larger) loss when compared to the true optimal configuration (the loss values are indicated below each plot in Figure 6). This demonstrates that the features have some degree of flexibility while remaining in close proximity to the minimum loss.

To enhance our comprehension of the differences between theoretical features (depicted in Figure 6) and empirical features (illustrated in Figure 1), we performed an experiment employing mixup within the training framework detailed in Section 2, but fixing the classifier as a simplex ETF. The resulting last-layer features are visualized in Figure 7. Prior work (Zhu et al., 2021; Yang et al., 2022; Pernici et al., 2022) have explored the effects of fixing the classifier, but not in the context of mixup. Our observations reveal that when the classifier is fixed as a simplex ETF, the empirical features tend to exhibit a more hexagonal shape in its different-class mixup features, aligning more closely with the theoretically optimal features. Moreover, comparable generalization performance is achieved when compared to training with a learnable classifier under the same setting.

Based on these results, a possible explanation for the variation in configuration is that during the training process, the classifier is still being learned and requires several epochs to converge to a simplex ETF, as depicted in Figure 5. During this period, the features may traverse regions that lead to slightly suboptimal loss, as there is flexibility in the features’ structures without much degradation to the loss performance (depicted in Figure 6).