Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective

A classification model typically consists of a feature extractor that maps inputs into features and a classification layer that uses those features to make decisions. The representation space of the model refers to the space where the output of the feature extractor, or more specifically, the output of the penultimate layer of the model resides. It usually has high dimensionality (e.g., 2048 for ResNet50 and 768 for Swin-T), making it challenging to visually analyze its characteristics. To address this, we insert a linear layer between the feature extractor and the classification layer, mapping the output features into 2D (or 3D) vectors, and adjust the input dimension of the classification layer accordingly. As a result, the new representation space is 2D (or 3D), which facilitates visual examination. We train this modified model as described in Appendix A.1 and visualize the 2D (or 3D) representation space as described in Appendix B.

Figure LABEL:fig:1 shows the visualization results for ResNet50 (He et al., 2016), MobileNetV2 (Howard, 2017), Swin-T (Liu et al., 2021), and DeiT-T (Touvron et al., 2021) on the CIFAR-10 dataset (Krizhevsky et al., 2009). Regardless of the differences in model structures, it can be observed that the decision regions are divided into circular sectors, i.e., cone-like shapes, centered around the origin, with features radially distributed within these regions.

In addition, these characteristics remain consistent across different weight initializations as well. Figure LABEL:fig:2 shows the process of how the decision regions change during training of ResNet50 on CIFAR-10 with four different weight initializations (see Appendix A.1 for details). In Figure LABEL:fig:2_a, we use the default Kaiming uniform weight initialization (He et al., 2015). Before training, the features are distributed close to the origin, as the model’s weights are initialized as small values, and some decision regions are divided around it. For other decision regions, due to random weight initialization, they have not yet been assigned to regions in the representation space. In Figure LABEL:fig:2_b, we set the initial weights of the feature extractor in such a way that the features are positioned far from the origin while the weights of the classification layer are initialized as in Figure LABEL:fig:2_a so that the decision regions remain divided around the origin. In Figure LABEL:fig:2_c, we set the initial weights of the classification layer so that the decision regions are divided far from the origin (but the initial features are located around the origin due to the default initialization of the feature extractor). Finally, in Figure LABEL:fig:2_d, we adjust the initial weights of the entire model so that shifting the center of the decision regions is shifted and the features are distributed far from the origin. It can be observed from the figure that in all four cases, the decision regions eventually evolve into cone-like shapes around the origin as training progresses.

The cone-shaped decision regions are also consistently observed when regularization using soft labels is applied (see Figure LABEL:fig:10 in Appendix D).

In this section, we demonstrate that decision regions in the original, high-dimensional representation space (e.g., 2048 for ResNet50) are also divided into cone shapes centered around the origin. One simple way to verify this is to gradually move a correctly classified feature linearly toward the origin and observe when it becomes misclassified for the first time. This process is illustrated in Figure 4. If the decision regions are cone-shaped, the classification result will remain consistent until the feature arrives at the origin. Actually, the intersection point of the cone-shaped decision regions does not precisely coincide with the origin, but is close to the origin. Thus, the moment that the misclassification occurs will be only at the final stage of the linear movement. On the other hand, if the regions are not cone-shaped, meaning another class region lies between the feature and the origin, the feature will become misclassified early during the movement.

We verify this for ResNet50, MobileNetV2, Swin-T, and DeiT-T on the test sets of CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), and the validation set of ImageNet (Russakovsky et al., 2015) (see Appendix A.2 for training details). For each feature in the representation space, we linearly move it toward the origin over 100 uniform steps. If the index of the first misclassified step is close to 100, it suggests that the decision region is likely cone-shaped. The results are shown in Table 1. Since nearly all indices are over 98, we can confirm that decision regions are indeed divided into cone shapes in high-dimensional representation spaces as well. Further verification can be found in Appendix C.

To understand the shape of the decision regions, we investigate what happens when a feature passes through the classification layer. In other words, we examine the factors that influence the calculation of logits. A feature is assigned to the class showing the largest logit value by the classification layer. Given a feature $\mathbf{f}$ for an input image, the logit $y_{c}$ for class $c$ can be expressed as follows.

where $\mathbf{w}_{c}$ and $b_{c}$ are the weight vector and the bias of the classification layer for class $c$, respectively, and $\theta$ is the angle between $\mathbf{f}$ and $\mathbf{w}_{c}$. Therefore, the elements that can affect the prediction result for an image are $||\mathbf{w}_{c}||$, $\cos\theta$, and $b_{c}$.

Note that $||\mathbf{w}_{c}||$ and $b_{c}$ are commonly applied to all data, so they are trained not to vary much across different classes (Papyan et al., 2020). In addition, we find that most bias values are very close to zero at the end of training and thus have minimal impact on the ranking of logit values (further details are provided in Appendix C). The only remaining factor that can play a decisive role in the classification result is the cosine similarity between the feature vector $\mathbf{f}$ and the weight vector $\mathbf{w}_{c}$ for class $c$. In other words, the classification result depends on the alignment between the class’s weight vector and the feature vector. Thus, decision boundaries are formed based on the degree of alignment, and consequently, the decision regions take the shape of cones centered at the origin, aligned with the weight vectors. This explains why different model structures (differing only in feature extractors, while using the same classification layers as in Equation 1) and varying weight initializations do not alter the characteristic cone-shaped decision regions forming around the origin.

In a low-dimensional space, it is easy to identify the approximate distribution of features, as shown in Figure LABEL:fig:1. In a high-dimensional space, however, visualization becomes challenging, making it difficult to understand how features are distributed. To overcome this, we establish criteria for analysis of feature distributions. In particular, we quantify how close a feature is to the decision boundary based on its confidence value.

In Figure LABEL:fig:4_a, we present the confidence contours of decision regions in the 2D representation space. Suppose that we want to move a correctly classified feature with high confidence (red dot) to reduce its confidence. This can occur in two ways: 1) by moving radially toward the origin (blue dot) or 2) by moving toward the nearest decision boundary (green dot). Therefore, we determine how far a specific feature is from the decision boundary using two criteria: 1) the root mean square (RMS) of the feature (we use RMS as the feature magnitude to compensate for different dimensionalities across models) and 2) the cosine similarity of the feature with the class center (crosses in Figure LABEL:fig:4_b), where the classification loss is minimal for that class. The class center for a certain class is found by optimizing an arbitrary feature vector in the representation space to achieve the lowest loss for that class using the gradient descent method. Figure LABEL:fig:4_c shows the loss and gradient directions in the 2D representation space, where the point from which the gradients flow out corresponds to the class center. Discussions on different candidates for the class center can be found in Appendix E, where we verify that the minimum loss point is the best option for measuring proximity to the decision boundary based on confidence.

We show the relationship of the confidence vs. the RMS of features and the cosine similarity of features with the class center in the top row of Figure LABEL:fig:5. As expected, the smaller the RMS of features or the lower the cosine similarity is, the lower the confidence is, indicating proximity to the decision boundary.

In Figure LABEL:fig:6, we present the results of training with and without regularization (label smoothing, Mixup, and CutMix) in the 2D representation space. Training with regularization results in two notable changes. First, the RMS of features significantly decreases (top row of Figure LABEL:fig:5 and Figure LABEL:fig:6), bringing them closer to the origin. Second, from the middle row of Figure LABEL:fig:5, we observe that the proportion of data with high cosine similarity between the feature and the class center increases in the regularized models. These changes are examined in detail below.

Decrease in RMS. When one-hot encoded labels are used for training, the weights are updated to maximize the difference between logits. To achieve great logit separation between classes, it is beneficial to have large feature vectors (Wang et al., 2017). However, with soft labels, the deviation between logit values should be reduced, as large feature vectors would increase the loss. Therefore, a model trained with hard labels learns to generate features farther from the origin (with large RMS), while a model trained with soft labels produce features closer to the origin (with small RMS). To confirm this, we visualize the cross-entropy loss and gradient directions for hard and soft labels in a 2D representation space in Figures LABEL:fig:7_a and LABEL:fig:7_b, respectively. The point with the smallest loss is marked with a white cross. We can see that for hard labels, the location of the cross mark is far from the origin, while for soft labels, it is located near the origin.

Increase in cosine similarity. In Figure LABEL:fig:8, we show the confidence contours in the 2D representation space of ResNet50 trained with and without regularization on CIFAR-10. For both models, we compare how well a feature needs to be aligned with the class center (crosses) to achieve a certain confidence level. Specifically, we search for features with confidence higher than 0.99 but with the worst alignment to their class center in terms of cosine similarity, in both clockwise and counterclockwise directions (red stars). By connecting these features to the origin (red lines) and observing the angle between the lines, we can see that the angle in the regularized model is smaller (Figure LABEL:fig:8_b). This occurs because, near the origin, the region for achieving a certain confidence value (e.g., 0.99) or higher becomes narrow. Since this region contains the class center, a feature close to the origin must be well-aligned with the class center to reach a given confidence level. Conversely, a feature located far from the origin can still achieve high confidence without being as closely aligned with the class center as a feature located near the origin (Figure LABEL:fig:8_a). Therefore, in the regularized models producing features with small RMS, the cosine similarity of features with the class center is relatively high compared to the baseline models.

In the bottom row of Figure LABEL:fig:5, miscalibration due to overconfidence in the baseline training is suppressed using regularization. This can be explained in relation to the RMS decrease mentioned in Section 4.2.

When the magnitude of a feature $\mathbf{f}$ is decreased (red dot becoming the blue dot in Figure LABEL:fig:4_a) by a factor of $T$ due to training with regularization, its corresponding logit for class $c$ can be expressed as $\frac{{\mathbf{w}_{c}^{T}\mathbf{f}}}{T}+b_{c}$. In Section 3, we demonstrated that, due to the cone-shaped decision boundaries, vectors located on the line connecting a feature and the origin are mostly classified into the same class as the feature (Table 1). Furthermore, in Section 4.1, we showed that features with smaller RMS have lower confidence values. Therefore, if the magnitude of a feature vector is scaled down and the feature moves closer to the origin, the confidence of the feature will decrease, but the prediction will remain unchanged. In other words, by scaling the features through regularization using soft labels, the model’s calibration is adjusted without affecting accuracy. See Appendix G for additional related experiments.

In fact, the effect of feature scaling due to regularization is similar to a post-processing technique known as temperature scaling. Temperature scaling adjusts calibration by scaling the logit values by a factor of $T$, resulting in the logit expression $\frac{{\mathbf{w}_{c}^{T}\mathbf{f}+b_{c}}}{T}$, which is similar to the case of feature scaling. Mathematically, there is a difference of $\frac{T-1}{T}b_{c}$, but as discussed in Section 3.3, most bias values are close to zero and do not affect the ranking of logits.

How does the use of regularization lead to better robustness against gradient-based adversarial attacks? To explain this, we examine the gradient directions in the 2D space of ResNet50 on CIFAR-10 in Figure LABEL:fig:7_a. Note that the gradients shown in the figure is used to perturb the data in FGSM (and iterative FGSM), which is the most basic gradient-based attack. In areas with high cosine similarity to the class center (marked with a cross), the gradient directions point toward the origin. However, in areas with low cosine similarity, the gradients are nearly orthogonal to the class center vector, heading toward the decision boundary. Specifically, we show two samples in Figure LABEL:fig:7_a: circles represent the features of correctly classified clean images, and triangles represent features after perturbation by an amount of $\epsilon=8/255$. The feature vector well-aligned with the class center vector (green sample) can remain within the decision region after being perturbed, as the gradient points toward the origin. On the other hand, the feature vector poorly aligned with the class center vector (orange sample) moves toward the nearby decision boundary and easily becomes misclassified.

To verify this in the original high-dimensional representation space, we apply the same attack to ResNet50 on ImageNet. The results are shown in the middle row of Figure LABEL:fig:5, where the blue bars represent the histogram of cosine similarity between features and their class centers, and the orange line shows the attack success rate for each confidence bin. For regularized models, the number of features with high cosine similarity to the class center increases, and these features exhibit lower attack success rates, which is consistent with the results observed in the 2D representation space. Further discussions on adversarial robustness are provided in Appendix H.

In Table 2, we present comprehensive results for various models (ResNet50, Swin-T, MobileNetV2 (Howard, 2017), EfficientNet-B1 (Tan, 2019), and ViT-B/16 (Dosovitskiy et al., 2021)) trained with different methods on the ImageNet dataset (see Appendix A.2 for training details). We consistently observe that, when regularization is applied, the RMS of features decreases, and the cosine similarity between features and class centers increases. These changes result in reduced overconfidence in predictions (leading even to underconfidence in some cases) and improved robustness to adversarial attacks.