MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognition

Deep learning has achieved remarkable progress in a range of computer vision (CV) tasks, such as image recognition [17, 14, 64, 30, 63], object detection [45, 12], and action recognition [46]. Despite advances in deep technologies and computational capability, great success is also highly dependent on well-designed large datasets such as ImageNet [13] and Places [67], where each category has sufficient and roughly balanced training samples. However, real-world data tends to be long-tailed over semantic categories [60]: a few categories contain many instances (called head categories), while most categories contain only a few instances (called tail categories). Long-tailed recognition (LTR) is challenging because it needs to deal not only with the numerous small data learning problems of the tail categories but also with the extremely unbalanced classification of all categories. Deep models trained with such long-tailed data are usually biased toward head categories on balanced testing data and perform poorly on tail categories.

To address this challenge, many approaches have explored long-tail recognition in order to learn well-performing models from long-tailed data, such as class re-balancing/re-weighting [3, 4, 24, 29, 11, 6, 54, 35], decoupling learning [23] and contrastive learning [56, 22, 49, 68, 10]. Recently, long-tailed recognition methods employing multi-expert ensemble learning [53, 51, 5, 59, 27] have achieved state-of-the-art (SOTA) performance. We summarize two key aspects of these approaches that need further improvement for boosting LTR. (1) Diverse experts experts focus on different aspects, maximizing the expertise of each [5, 51]. More diversity can support experts in improving LTR. (2) There is a heavy model variance in the prediction of the model, especially for the tail category. So, reducing model variance is essential for LTR. Previous multi-expert methods [53, 51, 5, 59, 27] focused on the above two aspects but did not handle them well. RIDE [51] utilizes a loss to moderate diversity, yet individual experts focus primarily on head categories. ACE and SADE [5, 51, 59] focus on the diverse experts, which learn classification knowledge from sub-categories or dominant categories. However, The ”tail category experts” of these methods can greatly suppress head category performance while focusing on the tail categories. Furthermore, these multi-expert methods all employ an ensemble method to reduce the final variance while ignoring the variance of each expert. Among them, NCL [27] introduces strong data augmentation [8] that provides a better generalization of the model. However, there is still a high risk of model variance in its one-hot label supervision for strongly augmented instances. To this end, we design a novel method, namely More Diverse experts with Consistency Self-distillation (MDCS), for long-tailed recognition.

Our proposed MDCS contains two key components, Diversity Loss (DL) and Consistency Self-distillation (CS). Our DL contains an adjustable distribution weight, to cater to the diversity of each expert. By adjusting the distribution weight, each expert tends to recognize different categories, such as Many-shot categories, Medium-shot categories, and Few-shot categories. It is a simple yet effective method for increasing diversity and significantly improving recognition accuracy over previous methods (discussed in Sec. 4). To reduce the model variance and avoid model overfitting instances, we look forward to providing each expert with a richer form of supervision when learning strongly augmented samples. The label-smoothing regularization [47, 37] is a straightforward way, and further MiSLAS [65] proposes label-aware smoothing for long-tailed recognition. However, the proportion of label-smoothing assignments of these methods is still instance-agnostic, and more reasonable label assignment principles remain to be explored. To this end, we design CS for each expert, which distills richer instance knowledge from predictions of weakly augmented data to regularize strongly augmented instances. Especially for a mini-batch instance, we propose Confident Instance Sampling (CIS) to select the correctly classified instances for consistency self-distillation. In this way, our proposed CIS can prevent CS from introducing biased/noisy knowledge. As illustrated in Fig. 1, the model trained with strong augmentation method [8] could reduce the model variance [51] compared with the model trained with weakly augmented instances (e.g., flipped, cropped). However, our CS trains the model on strongly augmented instances and is supervised by ”soft labels” from the predictions of weakly augmented instances, leading to lower model variance and higher recognition accuracy. These ”soft labels”, produced by prediction on weakly augmented representation, contain more knowledge than their one-hot labels. In addition, the roles of our DL and CS are mutually reinforcing and coupled: (1) Our CS is designed for each expert, which increases the diversity and recognition accuracy of a single expert, and ultimately benefits the ensemble model. (2) Without the DL, the CS cannot achieve remarkable results as the model is biased towards head categories (discussed in Sec. 6).

In the experiments, our proposed MDCS model outperforms state-of-the-art (SOTA) methods by a significant margin on five commonly used benchmark datasets. For instance, on CIFAR-100-LT with an imbalance factor of 100, our approach achieves an accuracy of 56.1%. Similarly, on ImageNet-LT with ResNeXt-50, our model achieves an accuracy of 61.8%, while on iNaturalist 2018 with ResNet-50, we achieve an accuracy of 75.6%.

Long-tailed Visual Recognition.  Conventional methods to alleviate the long-tailed problem are to design re-balancing paradigms that consist of re-sampling and re-weighting. Re-sampling methods, which over-sample tail classes or under-sample head classes, aim to achieve a more balanced data distribution. Re-sampling by simply over-sampling minority classes [3, 4, 38] and under-sampling by abandoning data for dominant classes [21, 16, 3]. However, over-sampling duplicated tailed samples might lead to the over-fitting of minority classes [3]. Under-sampling potentially loses head class information and certainly impairs the generalization ability of the DNNs. Re-weighting methods [24, 29, 11, 6, 54, 35, 42, 55] assign weights to different classes by loss modification or logits adjustment. However, some researchers observed that re-balancing methods will hurt representation learning, and decoupling representation with classifiers will lead to better features. Therefore, two-stage learning was proposed which first trains the model with original data and then fine-tunes the classifier with class-balanced data [33]. Transfer learning is another way to tackle the long-tailed problem, aiming to transfer knowledge learned from majority classes to minority classes. But knowledge transfer methods often need carefully designed structures such as memory bank [31, 50, 32]. More recently, many works try to improve the performance of long-tailed visual recognition by using contrastive learning (CL) strategy [56, 22, 49, 68, 10, 28]. For example, PaCo [10] introduces a set of class-wise learnable center to overcome bias on high-frequency classes of basic supervised contrastive learning (SCL).

Ensemble-based methods, which use multiple experts with aggregation methods, are receiving more and more attention due to their effectiveness on long-tailed recognition. LFME [53] trains different experts with different parts of the dataset and distills the knowledge from these experts to a student model. RIDE [51] optimizes experts jointly with distribution-aware diversity loss and trains a router to handle hard samples. SADE [58] proposed test-time experts aggregating method to handle unknown test class distributions. Recently proposed NCL [27] uses mutual distillation allowing every expert to learn knowledge from others. They still have shortcomings in terms of expert diversity and model variance.

Knowledge Distillation.  Knowledge distillation(KD) [2, 18, 39] was proposed for the purpose of model compression by transferring knowledge learned from a powerful teacher model to a student model. KD is performed by supervising the student model with soft labels generated by the teacher model, which also provides better generalization to the student model. KD has gradually evolved from an offline process [40, 18, 39], where the teacher model has trained ahead of time, to an online process [7, 15, 62], where teacher model and student model are trained simultaneously. Unlike offline or online KD, self-distillation [57] assumes that one model can be its own teacher, where the teacher model and student model are identical.

Consistency regularization. Consistency regularization has played a very important role in semi-supervised learning, which was first proposed by Bachman [1] and popularized by Sajjadi [43] and Laine [26]. Consistency regularization utilizes unlabeled data by assuming the model should output the same result when the inputs are similar. Specifically, given two different formations of perturbation of a training sample, the gap in the output is treated as a loss to train the model. There are various ways to generate perturbed input [36, 48]. A common method is employing two different formations of data augmentation on the same image [44]. FixMatch [44] computes an artificial label for each unlabeled sample by computing the model’s predicted class distribution given a weakly-augmented version. Unlike the above methods, our proposed consistency self-distillation first designs without extra hyper-parameters, combines consistency mechanisms that transfer the richer knowledge of weakly augmented instances to provide more supervision, and employs confidence instance sampling to remove biased/noisy knowledge. Benefiting from these well-designed components, our method effectively reduces the model variance and improves generalization ability.

The proposed MDCS consists of two parts, Diversity Loss(DL) and Consistency Self-distillation(CS), aiming to improve the diversity of experts and reduce model variance, respectively. In the following part, we first introduce the preliminaries of long-tailed recognition. Then, we elaborate on our proposed DL and CS. Finally, we show the overall loss of the training process.

In this section, we perform experiments on five widely used datasets in long-tailed recognition, including CIFAR100/10-LT [25], ImageNet-LT [31], Places-LT [31], and iNaturalist 2018 [20]. After that, we conduct ablation experiments on the CIFAR100-LT and ImageNet-LT datasets to gain further insights.

Simulating weight distributions by $\lambda$. To enrich the diversity of each expert, the Diversity Loss employs the hyper-parameter $\lambda$ to simulate different weight distributions for each expert’s training. Fig. 3 indicates how different $\lambda$ can affect the accuracy of Many-shot, medium-shot, and few-shot categories and overall accuracy. When $\lambda$ increases, the accuracy of Many-shot categories decreases while the accuracy of few-shot categories increases, which demonstrates the ability to simulate different weight distributions. Besides, when $\lambda$ gets high enough, the accuracy of few-shot classes will decrease. This is due to the few-shot group having 30 categories on CIFAR100-LT, this categorization is not fine-grained enough to cater to the effect of the extremely inversely long-tailed distribution generated.

Influence of data augmentations. The RandAug [8] is widely employed as its strong generalization for long-tailed recognition [27, 68, 10]. In this subsection, we conduct different augmentations on training samples to evaluate the effectiveness of weak-strong consistency self-distillation. The results are shown in Table 9, where we compare weak-strong distillation with weak-weak distillation and strong-strong distillation. The results of weak-strong consistency self-distillation are better than strong-strong distillation, which demonstrates that the excellence of our structure does not depend entirely on RandAugment.

Impact of a different number of experts. Our proposed MDCS is a multi-expert framework. The number of experts can be easily extended by adjusting $\lambda$ for different experts. We conduct experiments to demonstrate the power of multiple experts. As shown in Fig. 4 (a), the performance of MDCS tends to get better when the number of experts increases. The $\lambda$ for the one-expert model is $\{1\}$, for the two-expert model is $\{-0.5,2.5\}$, for the three-expert model is $\{-0.5,1,2.5\}$, for the four-expert model is $\{-0.5,0,1,2.5\}$, for the five-expert model is $\{-0.5,0,1,2,2.5\}$, for the six-expert model is $\{-1,-0.5,0,2,2.5,3\}$, and for the seven-expert model is $\{-1,-0.5,0,1,2,2.5,3\}$.

The rule for setting $\lambda$. The ensemble models are not sensitive to the hyper-parameter $\lambda$ within a reasonable range, so we can easily choose $\lambda$ just to spread across this range. When the number of branches of experts increases, we first average divide experts into the three groups to set $\lambda$. For the head group, the $\lambda\in$ [-1, 0.5], for the balanced group, the $\lambda\in$ (0.5, 1.5), for the tail group, the $\lambda\in$ [1.5, 3]. when the values of $\lambda$ for different experts fell within the above three ranges, the multi-expert model exhibits effective performance improvements. For example, we set $\{-0.5,1,2.5\}$ for three experts and $\{-1,-0.5,0,1,2,2.5,3\}$ for seven experts.

Influence of loss weight $\alpha$. The $\alpha$ is an adjusted loss weight of Consistency Self-distillation to control the contribution of the CS part in total loss. To find an appropriate for $\alpha$, A series of values experimented on the CIFAR100-LT dataset. As shown in Fig. 4 (b), the best performance is achieved when $\alpha=0.6$. The best result means a balance of supervised learning and self-distillation.

Ablation studies on all components.

In this subsection, we conduct detailed ablation studies on the CIFAR100-LT dataset to analyze every component of our MDCS. As shown in Table 10, we evaluate the proposed components including Diverse Loss (DL), weak-strong augmentation(w/RandAug), Consistency Self-distillation (CS), respectively. The ✗in DL means we use normal Softmax to conduct experiments and the ✗in w/RandAug means we employ weak-weak augmentation. As shown in Table, our proposed Diversity Loss improves the performance by 2.9%. It is a core component because, without our DL, the other components are less effective at improving performance. MDCS Employing weak-strong augmentation can improve performance from 50.7% to 54.1%, which proves the strength of RandAug [8, 27]. Eventually, when conducting our proposed CS, the performance is significantly further improved, from 54.6% to 56.1%.

In this paper, we propose a novel method, MDCS, to cater to the diversity and variance of multi-expert, leading to improved long-tailed recognition accuracy. Our MDCS contains two core components: (1) diversity loss (DL), which can effectively enhance the diversity of experts. (2) consistency self-distillation (CS), which is a novel self-distillation method for reducing the model variance. Furthermore, we propose confident instance sampling in CS to ensure unbiased knowledge. In analyses and ablation studies, we analyze the effectiveness of our proposed core components through experimental results. Moreover, the roles of our DL and CS are mutually reinforcing and coupled. Experimental evidence shows that our MDCS achieves significant performance over the SOTA methods on five popular benchmarks, including 56.1% (+1.9%) accuracy on CIFAR100-LT with an IF 100, 61.8% (+1.3%) accuracy on ImageNet-LT with ResNeXt-50, and 75.6% (+0.7%) accuracy on iNaturalist 2018 with ResNet-50.

This work was supported in part by the National Natural Science Foundation of China under Grant No.62271034, and in part by the Fundamental Research Funds for the Central Universities under Grant XK2020-03.

1. The Efficiency of Consistency Self-distillation

As illustrated in Fig. 5, previous methods [51, 5, 59] reduced the model variance only by using an ensemble of multiple experts. In contrast, our approach not only reduces the variance by ensemble but also reduces the model variance by CS for each expert. The effect of CS is not only to reduce the model variance. Each expert gets richer constraint information through weakly augmented images, which enhances the expert’s own recognition ability. As shown in Table 11, experts with stronger recognition abilities also produce more diverse ensemble models.

2. More Details Settings

We implement our method with PyTorch. Following [59, 27], we use ResNeXt-50/ResNet-50 for ImageNet-LT, ResNet-32 for CIFAR100/10-LT, ResNet-152 for Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively. Moreover, we adopt the cosine classifier for prediction on all datasets. The details settings for our method are shown in table 12.