Adjusting Logit in Gaussian Form for Long-Tailed Visual Recognition

Input augmentation increases sample diversity in the data space. The classical augmentation methods [1] encompass operations such as flipping, rotating, cropping, padding, etc. Most recently, Wang et al. [21] proposed rare-class sample generator (RSG) that augments tail classes by utilizing encoded variation information obtained from head classes. M2m [22] establishes a well-balanced dataset through the translation of samples from head classes to tail classes, facilitated by an auxiliary pre-trained classifier.

Feature augmentation serves to enhance data diversity within the feature space. Knowledge transfer is a promising technology. For instance, Yin et al. [23] exemplified knowledge transfer by leveraging the intra-class variance derived from head classes in an encoder-decoder-based network to augment the features of tail class samples. Liu et al. [24] employed the transfer of angular variance, computed from head classes, to enrich the intra-class diversity within tail classes. Moreover, recent applications in addressing long-tailed data incorporate the use of class activation maps (CAM) [25]. Chu et al. [26] utilized CAM to decompose the features into a class-generic and a class-specific component. Then, tail classes are augmented by fusing the class-specific components obtained from the tail classes with the class-generic components of the head classes. Also, Zhang et al. [27] exploited CAM to obtain the foreground in an image and then augment the obtained foreground object by flipping, rotating, jittering, etc. The augmented foreground is then covered on the unchanged background to obtain a new informative image.

Those methods mentioned above require either an increase in data size or model complexity to solve the issues in long-tailed distribution, resulting in additional computational costs.

Recently, two-stage methods have been proposed and empirically demonstrated their efficacy. For example, Cao et al. [13] proposed LDAM-DRW, wherein features are learned in the initial stage, and a deferred re-weighting (DRW) strategy is employed to refine the classifier in the subsequent stage. While it markedly enhances long-tailed prediction accuracy, the theoretical underpinnings of the deferred DRW strategy remain unclear. Following this, Kang et al. [12] precisely identified out that the learning process of representation and classifier can be decoupled into two separate stages. The first stage performs representation learning on the original long-tail data. The second stage fixes the parameters of the backbone network and re-trains the classifier using class-balanced sampling data. Several studies [14, 15, 28] have further refined this strategy. For example, Zhang et al. [15] proposed an adaptive calibration function to calibrate the predicted logits of different classes, aligning them with a balanced class prior to preparation for the second stage. Zhong et al. [28] proposed class-based soft labels to address varying degrees of overconfidence in the predicted logit of each class, which can improve the classifier learning in the second stage. Another alternative approach is proposed by Zhou et al. [11], wherein the network structure is bifurcated into two branches. One branch focuses on learning the representation of head classes, while the other is tailored for tail classes. This structure incorporates feature mixup [29] into a cumulative learning strategy, yielding state-of-the-art results. Subsequently, Wang et al. [30] introduced contrastive learning into this bilateral-branch structure, further enhancing the performance of long-tailed classification.

More recently, researchers have explored the use of mixture of experts (MoE) methods to enhance performance by integrating multiple models into the learning framework. The fundamental concept behind these approaches is to introduce diversity to the data or models, which enables experts to concentrate on different portions of the data or allows experts with different structures to analyze the data. BBN [11] proposes a two-branched classifier that learns both the long-tailed and inverse distributions simultaneously, with a smooth transition of focus between them. BAGS [31], LFME [32], and ACE [33] divide the long-tailed data into different sub-splits and fit multiple experts on them. ResLT [34] designs residual structured classifiers that allow experts to specialize in different parts of the long-tailed data and complement each other. RIDE [35] and TLC [36] employ multiple experts, each trained on different augmented data, to independently learn the long-tailed distribution. The predictions of all experts are then gradually integrated to reduce overall model variance or uncertainty. SHIKE [37] investigates the impact of feature depth on data of varying scales in long-tailed visual recognition. The authors propose a new architecture, which incorporates features from different layers of a neural network to exploit the rich information present at different depths of a network. NCL [38] adopts multiple complete networks to learn the long-tailed data individually and uses self-supervised contrastive strategy [39] to collaboratively transfer knowledge among each individual expert.

Re-weighting the loss function is one of the most intuitive ways to improve the attention of DNN model on tail classes. In the literature, sample-wise re-weighting [40, 41] introduces the fine-grained coefficients into the loss function to make the model pay more attention to the difficult samples. Furthermore, class-wise re-weighting [42, 18, 43] assigns the standard CE loss with category-specific parameters that are inversely proportional to the class sizes. These methods can alleviate the data imbalance to a certain extent. However, when the imbalance ratio is very high, large weights may cause overfitting to the tail classes. Besides that, another side effect of assigning higher weights to difficult samples/tail classes is overly focusing on harmful samples (e.g., abnormal samples or mislabeled data) [44].

Loss function can also be modified by adjusting the logit. Menon et al. [45] proposed logit adjustment (LA), which is consistent in minimizing the balanced error. The logit shifting in LA of different classes is based on label frequencies of training data. By contrast, LADE [46] post-processes the model prediction by disentangling the training set distribution from the prediction. This method does not require the test set to be a uniform distribution. Also, DisAlign [15] adjusts the logit by calibrating the distribution of model prediction to a balanced one by minimizing the expected KL divergence. Overall speaking, these three methods can well adjust the classifier but do not take into account the distorted embedding space. Alternatively, re-margining methods [13, 47, 48] address long-tailed data by leaving large relative margins for tail classes during training. For example, label-distribution-aware margin (LDAM) loss [13] utilizes Rademacher complexity to theoretically prove that the margin should be inversely proportional to a quarter power of class sizes. The hard margin on target logit helps make the samples within a class more compact but the strict margin constraints increase the risk of overfitting and cannot actually expand the tail class coverage area in embedding space.

This section defines the notation used throughout this paper.

Suppose a feature point and a small area around it belong to the same type. It is reasonable that the adjacent points around a feature can be regarded as similar to it, and can naturally be considered as the same class.

Although both GCL-E and GCL-A calibrate the distorted embedding space well, the problem of classifier bias still remains to be addressed.

In the following, we analyze the reasons for the biased classifier. Eq. (13) implies that the sample of the target class $y$ punishes the classifier weights $\textbf{w}_{j}$ of non-target class $j,j\neq y$ w.r.t. $p_{j}$. In general, the number of training instances in head classes is enormously greater than in tail classes. Therefore, the classifier weights of tail classes receive much more penalty than positive signals during training. Consequently, the classifier will be biased towards the head classes and the predicted logits of the tail classes will be seriously suppressed, resulting in low classification accuracy of the tail classes [43, 56, 57]. We call this problem of the cross-entropy loss function in long-tailed learning negative gradient over-suppression. A straightforward approach to cope with it is to make the sample numbers of each class equal [58] to balance the negative gradients. To achieve this goal, we can make the tail classes over-sampling and then re-train the classifier. The sampling rate of each class is $\frac{1}{C}$. Then, the class-balanced sampling rate $q^{cb}_{j}$ of each sample $x$ from class $j$ is calculated by:

This strategy is called classifier re-training (cRT) [12]. It can also be combined with the effective number [18]. We can replace the actual sample number $n_{j}$ of class $j$ with the so-called effective number $n^{en}_{j}$, the effective sampling rate $q^{en}_{j}$ of each sample from class $j$ is given by:

where $n^{en}_{j}$ is calculated by:

with hyper-parameter $\beta\in[0,1)$. Algorithm 1 summarizes the overall training procedure of the proposed method.

In backward propagation, the gradients on logit $z_{i}$ are calculated by:

where $p_{i}=\dfrac{e^{z_{i}}}{\sum_{j=1}^{C}e^{z_{j}}}$. We take the binary case to illustrate without loss of generality. Suppose the input image is from class 1. The gradient on $z_{1}$ is calculated by:

It indicates that the gradient of the target class rapidly approaches zero with the increase of the target logit. This phenomenon is called softmax saturation [59, 60]. This inopportune early gradient vanishing weakens the validity of training samples and impedes model training. Therefore, softmax can only slightly separate various classes, and lacks the impetus to evenly distribute each class in the embedded space. We can also observe that there are many overlapping areas among each class in Fig. 2. Especially under the circumstances of long-tailed classification, the tail class features are insufficient to cover the ground truth distribution in embedding space. The early gradient vanish caused by soft saturation exacerbates the squeezing of the embedding distribution in tail class.

Different from the original softmax loss function, the logit difference ($\Delta_{y-j}$) obtained by GCL of Eq. (6) between the target and non-target classes is calculated by:

In case the target class is a tail class, $\delta_{y}-\delta_{j}>0$, which decreases the softmax saturation and thereby helps increase the validity of tail class samples. Eq. (8) has the same effect. Thus, Eq. (6) and Eq. (8) can automatically balance the sample validity of different classes and provide incentives for the model to make each class more separable. They achieve the aim of calibrating the distorted embedding space.

Compared with the prior work that enlarges the inter-class separability via the “hard margin”, e.g. see [49, 13, 60], Eq. (6) and Eq. (8) are equivalent to adding a “soft” margin. That is, the farther away from the class anchor, the lower the probability that the point belongs to this class. Fig. 3 schematically shows the comparison of the prior hard margin and the proposed soft margin. Hard margins will cause the samples to shrink toward the class anchor if the margin is too large. In addition to this, hard margins can lead to overfitting because they prohibit outliers, which can impair the robustness ability of the model. The proposed soft margin provides a smooth transition area, allowing the outliers to appear near the target class with a lower probability. This is both intuitively and theoretically more reasonable.

The cloud size $\delta^{*}_{j}$ may also take different expression forms, where the superscript $*$ indicates the adopted specific form. Cao et al. [13] obtained the optimal trade-off of the hard margin ($m_{i}$) and the class size via Rademacher complexity. They have proved that $m_{i}\propto n_{i}^{-1/4}$. The exponent should be ${-1/3}$ derived from Wei et al. [61]. Inspired by these works, we can set the cloud size in power function form:

where $n_{max}$ is the sample number of the most frequent class. $k$ can be ${1/3}$ or ${1/4}$. Menon et al. [45] used the Fisher consistency with respect to the balanced error and obtained that $m_{i}\propto\log(1/n_{j})$. Therefore, we can also set the cloud size in logarithmic form:

We also experimentally demonstrate the effectiveness of the cloud size in different expression forms in Section 5.4.

In short, GCL in the form of either normalized Euclidean distance or angular distance can achieve the following three advantages: 1) reduce the softmax saturation and thereby increase the sample validity of tail classes; 2) avoid overfitting and improve robustness through randomly sampling the values in Gaussian distribution; 3) enlarge the margin of class boundary for tail classes and thus calibrate the distortion of the embedding space. The slight disparity between the two forms lies in the procedural approach: GCL-E incorporates class-based perturbance onto features prior to logit calculation, whereas GCL-A is equivalent to sampling disturbed feature points subsequent to determining their distance from the class anchor. In addition, we systematically illustrate two versions of GCL and their distinctions from previous methods, exemplified by CE and LDAM [13], as depicted in Fig. 4.

The softmax has a time complexity of $\mathcal{O}(C)$, which is linear with the dimension of logit. It is the same as cross-entropy loss $\mathcal{L}_{CE}$ and $\mathcal{L}_{GCL}$ in both forms. The main difference in time complexity comes from the calculation of logit. For the original normalized logit (which is denoted as $\tilde{z}_{j}=s\cdot\cos\theta_{j}$), its main computational cost is vector multiplication. It contains $D\cdot C$ multiplications and $(D-1)\cdot C$ additions. Thus, the time complexity of computing $\tilde{z}_{j}$ is $\mathcal{O}(DC)$. Eq. (6) shows that GCL-E only adds $C$ scalar additions to $\tilde{z}_{j}$. As a result, computing $\tilde{z}^{GCL-E}_{j}$ has $\mathcal{O}(DC)$ time-complexity. For GCL-A, we first expand Eq. (8) to $\tilde{z}^{GCL-A}_{j}=s\cdot(\cos\theta_{j}\cos\delta_{j}\|\varepsilon\|-\sin\theta_{j}\sin\delta_{j}\|\varepsilon\|)$. The sine value can be obtained from the corresponding cosine value. Compared to $\tilde{z}_{j}$, GCL-A adds an additional $2C$ multiplications and $C$ subtractions. Computing $\tilde{z}^{GCL-A}_{j}$ also has $\mathcal{O}(DC)$ time-complexity. It is obvious that GCL in both forms imposes a negligible additional burden on the training process.

We use five benchmarks: CIFAR-10-LT and CIFAR-100-LT, ImageNet-LT, iNaturalist 2018, and Places-LT.

The parameters that need to be pre-set are the Gaussian distribution parameters $(\mu,\sigma^{2})$. For GCL-E, the maximum cloud size cannot exceed 1 because $\cos\theta_{i}\in[-1,1]$. Gaussian distribution has a probability of $99.7\%$ falling in $[\mu-3\sigma,\mu+3\sigma]$, we therefore set $\mu=0$ and $\sigma=\frac{1}{3}$. We further clamp $\varepsilon$ to $[-1,1]$ to prevent the cloud size from exceeding 1. For GCL-A, we first constrain the range of $\varepsilon$ to $[-1,1]$ in the same way as the cosine form GCL. Then, we multiply $\varepsilon$ with a constant $\frac{\pi}{2}$ to limit the cloud size in angular form to $[-\frac{\pi}{2},\frac{\pi}{2}]$ based on the lemma³³3Lemma: The classes can be distributed on a hyper-sphere of dimension $D$ such that any two class centers (namely, class anchors in this paper) are at least $\pi/2$ apart if the number of classes $C$ is less than twice the feature dimension $D$. proposed by Ranjan et al. [51]. Moreover, we normalize $\delta_{i}$ by $\delta_{i}\triangleq\delta_{i}/\max(\delta_{i}),i=1,2,...,C$ to ensure that maximum value of $\delta_{i}$ does not exceed 1. For data augmentation techniques, we follow Zhong et al. [28], except for basic augmentation such as image flip, rotation, and random crop, only mixup [67] are adopted in all experiments to ensure fair comparisons.

PyTorch [68] is utilized to implement the backbone network training. We adopt the SGD optimizer with a momentum of 0.9, coupled with a multi-step learning rate schedule. All models are trained from scratch, except for ResNet-152, which is pre-trained on the original balanced version of ImageNet-1K. For the first stage, we select ResNet-32 as the backbone network and follow the experimental settings in Cao et al. [13] for CIFAR-10/100-LT. For the experiments conducted on large-scale datasets, namely, ImageNet-LT, iNatralist 2018, and Places-LT, we mainly follow Kang et al.’s settings [12] except for the learning rate schedule. For the second stage, i.e., re-balancing the classifier, we follow Kang et al.’s setting [12] for all datasets.

We conduct a series of experiments to further analyze the proposed method.

Another rationale arises from the discrepancy in logits restrictions caused by varying imbalance ratios. Excessively strict logit constraints may lead the model astray. Without loss of generality, we use the most frequent class (denoted by subscript ‘head’) and the least frequent class (denoted by subscript ‘tail’) to analyze. For an input image that is tail class, GCL-A necessitates:

Considering $\delta=0.5$ as an example, when $\theta_{head}<\frac{\pi}{2}$, $\theta_{tail}$ being negative satisfies the requirements of the loss function, which could mislead the model training. The requirement that the angle between non-target classes and the target weight be greater than $\frac{\pi}{2}$ is overly stringent. For highly imbalanced datasets, namely iNaturalist 2018 and Places-LT, the discrepancies in perturbations between tail and head classes are more pronounced, which contributes to this phenomenon. In datasets with a smaller imbalance ratio, the disparities in perturbations are comparatively smaller, making this restriction relatively weaker. The majority of classes can adhere to their respective soft margin restrictions. However, opting for a smaller $\delta$ might result in the added perturbation being less conspicuous, thereby leading to less differentiation between classes. For GCL-E, an input image belonging to the tail class should satisfy the following inequality:

When $\delta=0.5$, $\theta_{head}>\frac{\pi}{3}$ will cause $\theta_{tail}$ to be negative. In contrast, the constraints imposed by GCL-E are more lenient, resulting in a slight decrease in performance on datasets characterized by a low imbalance ratio compared to GCL-A. Nonetheless, this relaxation does not predispose the model to erroneous interpretations stemming from excessively stringent restrictions.

Moreover, from another perspective, the selection of the perturbation magnitude $\delta$ holds a pivotal role for GCL-A. Additionally, cloud size selection should extend beyond mere class size considerations, with each variant of GCL potentially requiring its optimal strategy for cloud size selection. It is conceivable that the logarithmic form of cloud size utilized for GCL-A does not constitute the optimal choice. We leave these as our future study.