Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition

As shown in Figure 2, SADE builds a three-expert model that comprises two components: (1) an expert-shared backbone $f_{\theta}$; (2) independent expert networks $E_{1}$, $E_{2}$ and $E_{3}$. When training the model, the key challenge is how to learn skill-diverse experts from a single and stationary long-tailed training dataset. Existing ensemble-based long-tailed methods guo2021long ; wang2020long seek to train experts for the uniform test class distribution, and hence the trained experts are not differentiated sufficiently for handling various class distributions (refer to Table 6 for an example). To tackle this challenge, we first empirically investigate existing long-tailed methods in this task. From Table 1, we find that there is a simulation correlation between the learned class distribution and the training loss function. That is, the models learned by different losses are good at dealing with class distributions with different skewness. For instance, the model trained with the softmax loss is good at the long-tailed distribution, while the models obtained from long-tailed methods are skilled in the uniform distribution.

Motivated by this finding, we develop a simple skill-diverse expert learning strategy to generate experts with different distribution preferences. To be specific, the forward expert $E_{1}$ seeks to be good at the long-tailed class distribution and performs well on many-shot classes. The uniform expert $E_{2}$ strives to be skilled in the uniform distribution. The backward expert $E_{3}$ aims at the inversely long-tailed distribution and performs well on few-shot classes. Here, the forward and backward experts are necessary since they span a wide spectrum of possible class distributions, while the uniform expert ensures retaining high accuracy on the uniform distribution. To this end, we use three different expertise-guided losses to train the three experts, respectively.

The forward expert $E_{1}$    We use the softmax cross-entropy loss to train this expert, so that it directly simulates the original long-tailed training class distribution:

where $v_{1}(\cdot)$ is the output logits of the forward expert $E_{1}$, and $\sigma(\cdot)$ is the softmax function.

The uniform expert $E_{2}$    We aim to train this expert to simulate the uniform class distribution. Inspired by the effectiveness of logit adjusted losses for long-tailed recognition menon2020long , we resort to the balanced softmax loss jiawei2020balanced . Specifically, let $\hat{y}^{k}=\frac{\exp(v^{k})}{\sum_{c=1}^{C}\exp(v^{c})}$ be the prediction probability. The balanced softmax adjusts the prediction probability by compensating for the long-tailed class distribution with the prior of training label frequencies: $\hat{y}^{k}=\frac{\pi^{k}\exp(v^{k})}{\sum_{c=1}^{C}\pi^{c}\exp(v^{c})}=\frac{\exp(v^{k}+\log\pi^{k})}{\sum_{c=1}^{C}\exp(v^{c}+\log\pi^{c})}$, where $\pi^{k}=\frac{n_{k}}{n}$ denotes the training label frequency of class $k$. Then, given $v_{2}(\cdot)$ as the output logits of the expert $E_{2}$, the balanced softmax loss for the expert $E_{2}$ is defined as:

Intuitively, by adjusting logits to compensate for the long-tailed distribution with the prior $\pi$, this loss enables $E_{2}$ to output class-balanced predictions that simulate the uniform distribution.

The backward expert $E_{3}$    We seek to train this expert to simulate the inversely long-tailed class distribution. To this end, we propose a new inverse softmax loss, based on the same rationale of logit adjusted losses jiawei2020balanced ; menon2020long . Specifically, we adjust the prediction probability by: $\hat{y}^{k}=\frac{\exp(v^{k}+\log\pi^{k}-\log\bar{\pi}^{k})}{\sum_{c=1}^{C}\exp(v^{c}+\log\pi^{c}-\log\bar{\pi}^{c})}$, where the inverse training prior $\bar{\pi}$ is obtained by inverting the order of training label frequencies $\pi$. Then, the new inverse softmax loss for the expert $E_{3}$ is defined as:

where $v_{3}(\cdot)$ denotes the output logits of $E_{3}$ and $\lambda$ is a hyper-parameter. Intuitively, this loss adjusts logits to compensate for the long-tailed distribution with $\pi$, and further applies reverse adjustment with $\bar{\pi}$. This enables $E_{3}$ to simulate the inversely long-tailed distribution (cf. Table 6 for verification).

Based on the skill-diverse learning strategy, the three experts in SADE are skilled in different class distributions. The remaining challenge is how to fuse them to deal with unknown test class distributions. A basic principle for expert aggregation is that the experts should play a bigger role in situations where they have expertise. Nevertheless, how to detect strong experts for unknown test class distribution remains unknown. Our key insight is that strong experts should be more stable in predicting the samples from their skilled classes, even though these samples are perturbed.

Empirical observation    To verify this hypothesis, we estimate the prediction stability of experts by comparing the cosine similarity between their predictions for a sample’s two augmented views. Here, the data views are generated by the data augmentation techniques in MoCo v2 chen2020improved . From Table 2, we find that there is a positive correlation between expertise and prediction stability, i.e., stronger experts have higher prediction similarity between different views of samples from their favorable classes. Following this finding, we propose to explore the relative prediction stability to detect strong experts and weight experts for the unknown test class distribution. Consequently, we develop a novel self-supervised strategy, namely prediction stability maximization.

Prediction stability maximization    This strategy learns aggregation weights for experts (with frozen parameters) by maximizing model prediction stability for unlabeled test samples. As shown in Figure 2, the method comprises three major components as follows.

Data view generation    For a given sample $x$, we conduct two stochastic data augmentations to generate the sample’s two views, i.e., $x^{1}$ and $x^{2}$. Here, we use the same augmentation techniques as the advanced contrastive learning method, i.e., MoCo v2 chen2020improved , which has been shown effective in self-supervised learning.

Learnable aggregation weight    Given the output logits of three experts $(v_{1},v_{2},v_{3})\in\mathbb{R}^{3\times C}$, we aggregate experts with a learnable aggregation weight $w=[w_{1},w_{2},w_{3}]\in\mathbb{R}^{3}$ and obtain the final softmax prediction by $\hat{y}=\sigma(w_{1}\small{\cdot}v_{1}+w_{2}\small{\cdot}v_{2}+w_{3}\small{\cdot}v_{3})$, where $w$ is normalized before aggregation, i.e., $w_{1}+w_{2}+w_{3}\small{=}1$.

Objective function    Given the view predictions of unlabeled test data, we maximize the prediction stability based on the cosine similarity between the view predictions:

Here, $\hat{y}^{1}$ and $\hat{y}^{2}$ are normalized by the softmax function. In test-time training, only the aggregation weight $w$ is updated. Since stronger experts have higher prediction similarity for their skilled classes, maximizing the prediction stability $\mathcal{S}$ would learn higher weights for stronger experts regarding the unknown test class distribution. Moreover, the self-supervised aggregation strategy can be conducted in an online manner for streaming test data. The pseudo-code of SADE is provided in Appendix B.

Theoretical Analysis    We then theoretically analyze the prediction stability maximization strategy to conceptually understand why it works. To this end, we first define the random variables of predictions and labels as $\hat{Y}\small{\sim}p(\hat{y})$ and $Y\small{\sim}p_{t}(y)$. We have the following result:

Please refer to Appendix A for proofs. According to Theorem 1, maximizing the prediction stability $\mathcal{S}$ enables SADE to learn an aggregation weight that maximizes the mutual information between the predicted label distribution $p(\hat{y})$ and the test class distribution $p_{t}(y)$, as well as minimizing the prediction entropy. Since minimizing entropy helps to improve the confidence of the classifier output grandvalet2005semi , the aggregation weight is learned to simulate the test class distribution $p_{t}(y)$ and increase the prediction confidence. This property intuitively explains why our method has the potential to tackle the challenging task of test-agnostic long-tailed recognition at test time.

Datasets    We use four benchmark datasets (i.e., ImageNet-LT liu2019large , CIFAR100-LT cao2019learning , Places-LT liu2019large , and iNaturalist 2018 van2018inaturalist ) to simulate real-world long-tailed class distributions. Their data statistics and imbalance ratios are summarized in Appendix C.1. The imbalance ratio is defined as $\max{n_{j}}$/$\min{n_{j}}$, where $n_{j}$ denotes the data number of class $j$. Note that CIFAR100-LT has three variants with different imbalance ratios.

Baselines    We compare SADE with state-of-the-art long-tailed methods, including two-stage methods (Decouple kang2019decoupling , MiSLAS zhong2021improving ), logit-adjusted training (Balanced Softmax jiawei2020balanced , LADE hong2020disentangling ), ensemble learning (BBN zhou2020bbn , ACE cai2021ace , RIDE wang2020long ), classifier design (Causal tang2020long ), and representation learning (PaCo cui2021parametric ). Note that LADE uses the prior of test class distribution for post-adjustment (although it is unavailable in practice), while all other methods do not use this prior.

Evaluation protocols    In test-agnostic long-tailed recognition, following LADE hong2020disentangling , the models are evaluated on multiple sets of test data that follow different class distributions, in terms of micro accuracy. Same as LADE hong2020disentangling , we construct three kinds of test class distributions, i.e., the uniform distribution, forward long-tailed distributions as training data, and backward long-tailed distributions. In the backward ones, the order of the long tail on classes is flipped. More details of test data construction are provided in Appendix C.2. Besides, we also evaluate methods on vanilla long-tailed recognition kang2019decoupling ; liu2019large , where the models are evaluated on the uniform test class distribution. Here, the accuracy on three class sub-groups is also reported, i.e., many-shot classes (more than 100 training images), medium-shot classes (20$\sim$100 images) and few-shot classes (less than 20 images).

Implementation details    We use the same setup for all the baselines and our method. Specifically, following hong2020disentangling ; wang2020long , we use ResNeXt-50 for ImageNet-LT, ResNet-32 for CIFAR100-LT, ResNet-152 for Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively. Moreover, we adopt the cosine classifier for prediction on all datasets. If not specified, we use the SGD optimizer with the momentum of 0.9 for training 200 epochs and set the initial learning rate as 0.1 with linear decay. We set $\lambda\small{=}2$ for ImageNet-LT and CIFAR100-LT, and $\lambda\small{=}1$ for the remaining datasets. During test-time training, we train the aggregation weights for 5 epochs with the batch size 128, where we use the same optimizer and learning rate as the training phase. More implementation details and the hyper-parameter statistics are reported in Appendix C.3.

This subsection compares SADE with state-of-the-art long-tailed methods on vanilla long-tailed recognition. Specifically, as shown in Tables 3-4, Softmax trains the model with only cross-entropy, so it simulates the long-tailed training distribution and performs well on many-shot classes. However, it performs poorly on medium-shot and few-shot classes, leading to worse overall performance. In contrast, existing long-tailed methods (e.g., Decouple, Causal) seek to simulate the uniform class distribution, so their performance is more class-balanced, leading to better overall performance. However, as these methods mainly seek balanced performance, they inevitably sacrifice the performance on many-shot classes. To address this, RIDE and ACE explore ensemble learning for long-tailed recognition and achieve better performance on tail classes without sacrificing the head-class performance. In comparison, based on the increasing expert diversity derived from skill-diverse expert learning, our method performs the best on all datasets, e.g., with more than 2$\%$ accuracy gain on ImageNet-LT compared to RIDE and ACE. These results demonstrate the superiority of SADE over the compared methods that are particularly designed for the uniform test class distribution. Note that SADE also outperforms baselines in experiments with stronger data augmentation (i.e., RandAugment cubuk2020randaugment ) and other architectures, as reported in Appendix D.1.

In this subsection, we evaluate SADE on test-agnostic long-tailed recognition. The results on various test class distributions are reported in Table 5. Specifically, since Softmax seeks to simulate the long-tailed training distribution, it performs well on forward long-tailed test distributions. However, its performance on the uniform and backward long-tailed distributions is poor. In contrast, existing long-tailed methods show more balanced performance among classes, leading to better overall accuracy. However, the resulting models by these methods suffer from a simulation bias, i.e., performing similarly among classes on various class distributions (c.f. Table 1). As a result, they cannot adapt to diverse test class distributions well. To handle this task, LADE assumes the test class distribution to be known and uses this information to adjust its predictions, leading to better performance on various test class distributions. However, since obtaining the actual test class distribution is difficult in real applications, the methods requiring such knowledge may be not applicable in practice. Moreover, in some specific cases like Forward-LT-3 and Backward-LT-3 distributions of iNaturalist 2018, the number of test samples on some classes becomes zero. In such cases, the test prior cannot be used in LADE, since adjusting logits with $\log 0$ results in biased predictions. In contrast, without relying on the knowledge of test class distributions, our SADE presents an innovative self-supervised strategy to deal with unknown class distributions, and obtains even better performance than LADE that uses the test class prior (c.f. Table 5). The promising results demonstrate the effectiveness and practicality of our method on test-agnostic long-tailed recognition. Note that the performance advantages of SADE become larger as the test data get more imbalanced. Due to the page limitation, the results on more datasets are reported in Appendix D.2.

We next examine our skill-diverse expert learning strategy. The results are reported in Table 6, where RIDE wang2020long is a state-of-the-art ensemble-based method. RIDE trains each expert with cross-entropy independently and uses KL-Divergence to improve expert diversity. However, simply maximizing the divergence of expert predictions cannot learn visibly diverse experts (cf. Table 6). In contrast, the three experts learned by our strategy have significantly diverse expertise, excelling at many-shot classes, the uniform distribution (with higher overall performance), and few-shot classes, respectively. As a result, the increasing expert diversity leads to a non-trivial gain for the ensemble performance of SADE compared to RIDE. Moreover, consistent results on more datasets are reported in Appendix D.3, while the ablation studies of the expert learning strategy are provided in Appendix E.

This subsection evaluates our test-time self-supervised aggregation strategy.

Effectiveness in expert aggregation. As shown in Table 8, our self-supervised strategy learns suitable expert weights for various unknown test class distributions. For forward long-tailed distributions, the weight of the forward expert $E_{1}$ is higher; while for backward long-tailed ones, the weight of the backward expert $E_{3}$ is relatively high. This enables our multi-expert model to boost the performance on dominant classes for unknown test distributions, leading to better ensemble performance (cf. Table 8), particularly as test data get more skewed. The results on more datasets are reported in Appendix D.4, while more ablation studies of our strategy are shown in Appendix F.

Superiority over test-time training methods. We then verify the superiority of our self-supervised strategy over existing test-time training approaches on various test class distributions. Specifically, we adopt three non-trivial baselines: (i) Test-time pseudo-labeling uses the multi-expert model to iteratively generate pseudo labels for unlabeled test data and uses them to fine-tune the model; (ii) Test class distribution estimation leverages BBSE lipton2018detecting to estimate the test class distribution and uses it to pose-adjust model predictions; (iii) Tent wang2021tent fine-tunes the batch normalization layers of models through entropy minimization on unlabeled test data. The results in Table 9 show that directly applying existing test-time training methods cannot handle well the class distribution shifts, particularly on the inversely long-tailed class distribution. In comparison, our self-supervised strategy is able to aggregate multiple experts appropriately for the unknown test class distribution (cf. Table 8), leading to promising performance gains on various test class distributions (cf. Table 9).

Effectiveness on partial class distributions. Real-world test data may follow any type of class distribution, including partial class distributions (i.e., not all of the classes appear in the test data). Motivated by this, we further evaluate SADE on three partial class distributions: only many-shot classes, only medium-shot classes, and only few-shot classes. The results in Table 10 demonstrate the effectiveness of SADE in tackling more complex test class distributions.