Delving Deep into Simplicity Bias for Long-Tailed Image Recognition

Long-tailed image recognition is a fundamental research topic in machine learning and computer vision, where it aims to overcome the data imbalance challenge [16, 1, 4]. In general, existing long-tailed image recognition methods can be roughly separated into the following paradigms. 1) Class re-balancing strategies: Re-balancing strategies, e.g., data re-sampling [17, 18] and loss re-weighting [19, 20], are conventional solutions for handling long-tailed distributed data. The popular re-sampling methods involve over-sampling by simply repeating data of minority classes [17, 2, 21] and under-sampling by abandoning data of dominant classes [18, 2, 1]. But with re-sampling, duplicated tailed samples might lead to over-fitting upon minority classes [22, 4], while discarding precious data will inevitably impair the generalization ability of deep networks. On the other hand, re-weighting belongs to another line of class re-balancing strategies, which works by allocating large weights for training samples of tail classes in loss functions, e.g., [20, 23]. 2) Decoupled learning: It is a recent trend towards effective long-tailed image recognition, which decouples the image representation learning and classifier learning to improve long-tailed classification performance. Specifically, Zhou et al. [24] and Kang et al. [25] decoupled classifier and representation learning and found that uniform sampling could benefit representation learning while classifier learning favors class-balance sampling. Later, Wang et al. [26] further introduced supervised contrastive learning into decoupled learning to improve the discriminative ability of image representations. Recently, Alshammari et al. [27] developed a sequential decoupled learning method with weight balancing strategies and achieved promising performance. 3) Ensembling: Handling different parts of the long-tailed data with multiple experts shows promising performance for long-tailed image recognition, which works essentially as ensembling by integrating these experts to obtain the optimal model over the entire dataset  [28, 29, 30]. Concretely, [29] distilled multiple teacher models into a unified model, and each teacher focused on a relatively balanced group such as many-shot, medium-shot and few-shot classes. While for [31], it learned multiple distribution-aware experts by a dynamic expert routing module.

In the literature, simplicity bias (SB), i.e., the tendency of supervised neural networks to find simple patterns, was proposed to analyze the generalization of neural networks [8, 10]. In particular, in [8], it originally showed that neural networks trained with SGD are biased to learning the simplest predictive features in the data, while overlooking the complex but equally-predictive patterns. Inspired by this, Teney et al. [9] demonstrated that SB can be mitigated through a diversity constraint and explored how to improve out of distribution generalization by overcoming SB. In this paper, to our best knowledge, we are the first to investigate the SB problem in the long-tailed image recognition task, and propose methods that can effectively mitigate SB in long-tailed distribution learning.

Self-supervised learning (SSL) has attracted increasing attention thanks to its ability to learn meaningful data representations without human annotation. It is capable of adopting self-defined pseudo labels as supervision and employs the learned representations for various downstream tasks, e.g., object detection, segmentation in computer vision. In particular, contrastive learning has recently become a dominant research area in SSL, which aims at learning augmentation-invariant representations. The basic idea thereof is contrasting the agreement between positive pairs, e.g., images under different augmentations, against those from negative pairs. More specifically, in SimCLR [32], a successful end-to-end model consisting of two encoders was proposed, where one encoder generates representations for positive samples and the other learns representations for negative samples. However, SimCLR required a large batch size to achieve satisfactory performance. The idea of memory bank [33] provides a potential solution for alleviating this limitation by pre-storing encodings of negative samples. Later, in MoCo [34], it followed a dictionary look-up perspective for conducting contrastive learning by designing a dynamic dictionary with a queue and a moving-averaged encoder. SwAV [35] was proposed by utilizing a clustering algorithm to group similar feature together for similarity calculation. Very recently, [36] investigated SSL under the dataset imbalance setting and studied the problem of robustness to imbalanced training of self-supervised representations. In this paper, we empirically validate that SSL can learn more comprehensive features from input images, which can mitigate simplicity bias, especially for the tail data in long-tailed distribution. Moreover, we further propose a novel SSL method tailored for imbalanced data to boost the quality of representations learned from long-tailed data.

We present the dataset and preliminary results in the following, as well as discussing the investigation of simplicity bias in long-tailed image recognition.

As aforementioned, SB does appear in long-tailed image recognition and has a more dramatic effect on the tail data. To alleviate this problem, we propose to use self-supervised learning (SSL) to learn more comprehensive features/patterns, especially complex features/patterns.

As preliminary verification, we design an experiment with a baseline method incorporating SSL. Concretely, we utilize different approaches to pre-train a ResNet-32 [40] network on the long-tailed CIFAR-100 [5] dataset with imbalance ratio of 100, and then fix the backbone but fine-tune the classifier via class-balanced sampling. In this way, we compare the generalization ability of the learned image representations, which can implicitly reflect the degree of SB [8].

These pre-trained approaches as well as the classification accuracy obtained thereof are presented in Table II. More specifically, when comparing to pre-training with cross-entropy, we further equip self-supervised learning into pre-training by applying MoCo-v2 [41] in a multi-task learning framework, which is denoted by “Ours (CE w. SSL)”, as our SSL baseline in Table II. As shown, combining SSL, the classification accuracy obtains an obvious improvement on the tail data (about 3% improvements). In addition, we also report the results of only using SSL for pre-training, and we can see that it achieves inferior classification accuracy. These phenomena indicate that supervised learning and self-supervised learning are complementary to each other in this case.

Furthermore, we also perform the SSL baseline method on MNIST-CIFAR-LT and show the learning curves with cross-entropy and our SSL baseline in Figure 2. As shown, after applying SSL, the final classification accuracy of CIFAR-only has been improved. These observations in Figure 2 show that SB has been greatly alleviated by the self-supervised fashion. Also, we visualize the activations by performing our SSL baseline in Figure 1c. It can also be clearly observed that the network tends to attend more complex patterns of the CIFAR part thanks to SSL.

We follow the classic SSL framework, e.g., MoCo [34, 41], to realize the holistic-level self-supervised learning in our 3LSSL method. More specifically, for an image $\mathcal{I}$, we augment it in different ways and obtain $x$ and $x^{\prime}$ as inputs of the encoder and momentum encoder in SSL. Then, after performing two projectors, the image representations of $x$ and $x^{\prime}$ are obtained, i.e., $\bm{f}_{x}$ and $\bm{f}_{x^{\prime}}$. Different from previous SSL methods using InfoNCE as the loss function [34, 41], we employ $\ell_{2}$-normalization upon $\bm{f}_{x}$ and $\bm{f}_{x^{\prime}}$ and calculate the cosine similarity to drive the network training, which is as follows:

where we omit $\ell_{2}$-normalization for simplification. The holistic-level SSL is presented as the flow with black arrows in Figure 3, which aims to enhance the learning of comprehensive patterns in the holistic level w.r.t. $x$.

Regarding the update of momentum encoder in our method, we follow [34] by conducting

where $t$ represents the current step, and $t-1$ means the previous step. $\theta_{q}$ and $\theta_{k}$ represent the parameters of encoder and momentum encoder, respectively. $\alpha$ is the updating rate of the momentum encoder, which is fixed as $0.999$.

Through the visualized activations in the preliminary experiments on MNIST-CIFAR-LT, cf. Figure 1c vs. Figure 1b, it verifies that holistic SSL (i.e., the SSL baseline in Section 3.2) is able to make models learn more sufficient patterns (i.e., alleviating simplicity bias) than vanilla models to some extent. However, there are still some important but complex image regions that are ignored due to SB, e.g., the upper part of the ship or the head of the truck in Figure 1c. To solve the problem that holistic-level SSL might ignore complex patterns, we develop a partial-level SSL by leveraging a masking manner to force the models to learn more comprehensive information from the partial level w.r.t. the whole image. The partial-level SSL is illustrated as the flow with yellow arrows in Figure 3.

Concretely, for the input $x$, we can obtain its deep feature map $\bm{X}\in\mathbb{R}^{h\times w\times d}$ by performing the encoder, where $h$, $w$ and $d$ represent the height, width and depth of the feature map, respectively. Then, we employ an off-the-shelf localization approach, i.e., Class Activation Mapping (CAM) [42], upon $\bm{X}$ with the re-balanced classifier $\bm{W}$ (elaborated lated in Section 4.3) to indicate the class-specific discriminative image regions used by CNNs to identify that class. It could also expose the implicit attention of CNNs w.r.t. a target class on an image. Obeying the notations in [42], we conduct CAM by projecting back the weights of the output layer on to the convolutional feature maps as

where $w_{k}^{c}$ is the weight of $\bm{W}$ w.r.t. class $c$ for neural unit $k$ in the last convolutional layer of the encoder, and $f_{k}(i,j)$ represent the activation of unit $k$ at spatial location $(i,j)$. Thus, $M_{c}(i,j)$ directly indicates the importance of the activation at spatial grid $(i,j)$ leading to the classification of an image to class $c$, and $\bm{M}_{c}\in\mathbb{R}^{h\times w}$ corresponds to the so-called class activation map. Intuitively, $\bm{M}_{c}$ is simply a weighted linear sum of the presence of these visual patterns at different spatial locations. By further simply upsampling $\bm{M}_{c}$ to the size of the input image, it can identify the image regions most relevant to the particular class. Therefore, to achieve the goal of our partial-level SSL as aforementioned, we deploy $\bm{M}_{c}$ as a mask for filtering out those class-specific image regions that are most likely to be attended by deep networks, and force it to learn more comprehensive information from the remaining image regions. In our implementation, we normalize $\bm{M}_{c}$ by

where $\hat{{M}_{c}}(i,j)$ forms the normalized mask $\hat{\bm{M}_{c}}$ whose values are in the range of $\left[0,1\right]$. $\min(\bm{M}_{c})$ and $\max(\bm{M}_{c})$ are the minimum and maximum value of $\bm{M}_{c}$, respectively.

After that, the input w.r.t. the partial-level SSL $x^{p}$ is obtained by

where $\bm{J}$ is the unit matrix with all $1$ elements, $\odot$ is the element-wise product, and ${\rm upsample}(\cdot)$ represents the upsampling operation to the size of original image resolution. Thus, as shown by the yellow arrows in Figure 3, the embedding of $x^{p}$, i.e., $\bm{f}_{x^{p}}$, is obtained through the momentum encoder followed by the projector. Similar to the loss in the holistic-level SSL, we calculate the similarity between the $\ell_{2}$-normalized $\bm{f}_{x^{p}}$ and $\bm{f}_{x}$ by

to realize the partial-level SSL in our method.

To further consider the severe SB on tail data, we enhance the similarity-based representation learning of tail classes by borrowing pseudo samples from other classes informed by classifier outputs, which is termed as the augmented-level SSL. As the flow of blue arrows shown in Figure 3, we first build a re-balanced classifier $\bm{W}$ upon $x$ and then produce an augmented queue containing augmented samples $x^{q}$ from $x^{\prime}$ for achieving the aforementioned goal.

More specifically, we equip a supervised component by training a classifier upon the encoder’s output for recognizing $x$. It also complies with the conclusion that supervised learning and self-supervised learning complement each other in the case of long-tailed recognition, cf. Table II. The classification loss w.r.t. $\bm{W}$ hereby is the cross-entropy loss function on the ground truth and the calibrated predicted logits which are obtained by adding up re-balanced factors of $\{\ln(N_{c}/\sum_{j}N_{j})\}$ to original predicted logits (where $N_{c}$ is the number of samples in class $c$).

To enhance the similarity-based representation learning of the tail data $x$, in the augmented-level SSL, we use $\bm{W}$ to predict the class affiliation $c^{\prime}$ of $x^{\prime}$ and select samples from class $c^{\prime}$ as pseudo positives of $x$. If we denote the embedding of such pseudo positive as $\bm{f}_{c^{\prime}}$, we add another similarity-based loss on such a pair $(\bm{f}_{x},\bm{f}_{c^{\prime}})$. By doing this, we augment the tail classes by providing chances to see more semantic-relevant samples for representation learning and consequently learn features that generalize better.

Furthermore, in order to achieve better accuracy and make the training more stable, we develop an augmented queue strategy to pre-store embeddings $\bm{f}_{y}$ (i.e., a general form of $\bm{f}_{c^{\prime}}$ as aforementioned) and their predicted pseudo labels $y$. In concretely, for such a size-$L$ augmented queue $\left\{(\bm{f}_{y^{l}}^{l},y^{l})\right\}_{l=1}^{L}$, we initially set the feature vectors as $\bm{0}$ and their predictions as $-1$. After obtaining a pseudo-label $y$ of $x^{\prime}$ at each time step, the embedding-pseudo label pair in the earliest time step will be removed, and it adds the newly combing embedding-pseudo label pair to the queue. Then, we collect all embeddings associated with class $c^{\prime}$ in the augmented queue to obtain an aggregated feature embedding, which is formulated by

where $\Omega=\{l|y^{l}==c^{\prime}\}$. After that, the augmented-level SSL is optimized by computing the similarity between the $\ell_{2}$-normalized $\bm{f}_{x^{q}}$ and $\bm{f}_{x}$, which also follows the holistic- and partial-level SSL:

Overall, the final loss function of our 3LSSL method is optimized by

where $\mathcal{L}_{cls}$ denotes the classification loss w.r.t. $\bm{W}$. All trade-off parameters are $1$ for all experiments, which reveals the good potential of our method in practice and its simplicity.

We conduct experiments on five long-tailed image recognition benchmark datasets, including long-tailed CIFAR-10/long-tailed CIFAR-100 [12], ImageNet-LT [13], Places-LT [14] and iNaturalist 2018 [15]. Furthermore, we also report the results on the constructed MNIST-CIFAR-LT dataset. For quantitative comparisons, we evaluate the methods on the corresponding balanced validation datasets, and report the top-1 accuracy over all classes, which is denoted as “All”. Besides, we follow [25, 13] to split the classes into three subsets and report the average accuracy in these three subsets, i.e., “Many”-shot ($>$100 images), “Medium”-shot (20$\sim$100 images), and “Few”-shot ($<$20 images), which are also termed as head, medium and tail data, respectively.

• •

  Long-tailed CIFAR-10/long-tailed CIFAR-100: Both CIFAR-10 and CIFAR-100 [12] contain 60,000 images, 50,000 for training and 10,000 for validation with category number of 10 and 100, respectively. For fair comparisons, we use the long-tailed versions of CIFAR datasets as the same as those used in [5] with controllable degrees of data imbalance. We use an imbalance factor $\beta$ to describe the severity of the long tail problem with the number of training samples for the most frequent class and the least frequent class, e.g., $\beta=\frac{N_{max}}{N_{min}}$. Imbalance factors we use in experiments are $10$, $50$ and $100$.

• •

  ImageNet-LT: The ImageNet-LT dataset [13] is a long-tailed version of the original ImageNet-2012 [43], which is constructed by sampling a subset following the Pareto distribution with the power value as $6$. It contains 115.8K large-scale images from 1,000 categories. The number of training samples for each class ranges from 5 to 1,280. The validation set is the same as ImageNet 2012.

• •

  Places-LT: The Places-LT dataset is also a large-scale long-tailed dataset artificially created from the balanced Places-2 [14] dataset. It contains 184.5K images from 365 diverse scene categories. The distribution of labels in the training set is also extremely long-tailed, where the sample number of each class ranges from 5 to 4,980.

• •

  iNaturalist 2018: iNaturalist 2018 [15] is a large-scale real-world dataset with 437.5K images from 8,142 categories. It naturally follows a severe long-tailed distribution with an imbalance factor of 512. Besides the extreme imbalance, it also faces the fine-grained problem [44, 45, 46]. In this paper, the official splits of training and validation images are utilized for fair comparisons.

In our 3LSSL, the encoder and the momentum encoder are of the same architecture with the same parameter initialization. We follow previous work, e.g., [24, 47, 50, 51, 53, 54, 55, 56], to employ the corresponding backbone deep model to realize the encoders for fair comparisons. The two projectors are also of the same structure which consists of a two-layer perceptron head (hidden layer 2048-d, with ReLU). Both projectors have the same parameter initialization. While, during network training, two encoders and two projectors update its own parameters independently. The classifier component in our 3LSSL is a linear classification layer. More specifically, for fair comparisons to prior art, we follow the protocol in [24, 47, 50, 51, 53, 54, 55, 56] by employing ResNet-32 [40] as the backbone for long-tailed CIFAR-10/long-tailed CIFAR-100, ResNet-50 [40] and ResNeXt-50 [59] for ImageNet-LT, ResNet-152 [40] for Places-LT, and ResNet-50 [40] for iNaturalist 2018. Regarding other empirical settings, we follow the recent state-of-the-art, e.g., [56, 55], to conduct model training. Regarding the size $L$ of the augmented queue, we set $L=1024$ for long-tailed CIFAR and $L=8192$ for large-scale ImageNet-LT, Places-LT and iNaturalist 2018. All experiments are conducted with four GeForce RTX 3090 Ti GPUs.

In this section, we analyze and discuss our proposed 3LSSL by conducting ablation studies on the ImageNet-LT dataset.

To further analyze our 3LSSL method, we provide qualitative visualization analyses from the following aspects, i.e., showing feature embedding visualization by $t$-SNE, demonstrating the distribution of classes in the augmented queue, as well as presenting the activation visualization of the learned models.