Imbalanced Semi-supervised Learning with Bias Adaptive Classifier

Imbalanced SSL involves a labeled dataset $\mathcal{D}_{l}=\{(\mathbf{x}_{n},\mathbf{y}_{n})\}_{n=1}^{N}$ and an unlabeled dataset $\mathcal{D}_{u}=\{\mathbf{x}_{m}\}_{m=1}^{M}$, where $\mathbf{x}_{n}$ is a training example and $\mathbf{y}_{n}\in\{0,1\}^{K}$ is its corresponding label. We denote the number of training examples of class $k$ within $\mathcal{D}_{l}$ and $\mathcal{D}_{u}$ as $N_{k}$ and $M_{k}$, respectively. In a class-imbalanced scenario, the class distribution of the training data is skewed, namely, the imbalance ratio $\gamma_{l}:=\frac{\max_{k}N_{k}}{\min_{k}N_{k}}\gg 1$ or $\gamma_{u}:=\frac{\max_{k}M_{k}}{\min_{k}M_{k}}\gg 1$ always holds. Note that the class distribution of $\mathcal{D}_{u}$, i.e., $\{M_{k}\}_{k=1}^{K}$ is usually unknown in practice. Given $\mathcal{D}_{l}$ and $\mathcal{D}_{u}$, our goal is to learn a classification model that is able to correctly predict the labels of test data. We denote a deep classification model $\Psi=f_{\phi}^{\rm cls}\circ f_{\theta}^{\rm ext}$ with the feature extractor $f_{\theta}^{\rm ext}$ and the linear classifier $f_{\phi}^{\rm cls}$, where $\theta$ and $\phi$ are the parameters of $f_{\theta}^{\rm ext}$ and $f_{\phi}^{\rm cls}$, respectively, and $\circ$ is function composition operator.

With pseudo-labeling techniques, current state-of-the-art SSL methods (Xie et al., 2020b; Sohn et al., 2020; Zhang et al., 2021a) generate pseudo-labels for unlabeled data to augment the training dataset. For unlabeled sample $\mathbf{x}_{m}$, its pseudo-label $\hat{\mathbf{y}}_{m}$ can be a ‘hard’ one-hot label (Lee et al., 2013; Sohn et al., 2020; Zhang et al., 2021a) or a sharpened ‘soft’ label (Xie et al., 2020a; Wang et al., 2021). The model is then trained on both labeled and pseudo-labeled samples. Such a learning scheme is typically formulated as an optimization problem with objective function $\mathcal{L}=\mathcal{L}_{l}+\lambda_{u}\mathcal{L}_{u}$, where $\lambda_{u}$ is a hyper-parameter for balancing labeled data loss $\mathcal{L}_{l}$ and pseudo-labeled data loss $\mathcal{L}_{u}$. To be more specific, $\mathcal{L}_{l}=\frac{1}{|\hat{\mathcal{D}}_{l}|}\sum_{\mathbf{x}_{n}\in\hat{\mathcal{D}}_{l}}{\rm H}\left(\Psi(\mathbf{x}_{n}),\mathbf{y}_{n}\right),$ where $\hat{\mathcal{D}}_{l}$ denotes a batch of labeled data sampled from $\mathcal{D}_{l}$, ${\mathrm{H}}$ is cross-entropy loss; $\mathcal{L}_{u}=\frac{1}{|\hat{\mathcal{D}}_{u}|}\sum_{\mathbf{x}_{m}\in\hat{\mathcal{D}}_{u}}\mathbbm{1}(\max(p_{m})\geq\tau){\rm H}\left(\Psi(\mathbf{x}_{m}),\hat{\mathbf{y}}_{m}\right),$ where $p_{m}={\rm softmax}(\Psi(\mathbf{x}_{m}))$ represents the output probability, and $\tau$ is a predefined threshold for masking out inaccurate pseudo-labeled data. For simplicity, we reformulate $\mathcal{L}$ as

where $\lambda_{i}=\lambda_{u}\mathbbm{1}(\max(p_{i})\geq\tau)$. The pseudo-labeling framework has achieved remarkable success in standard SSL scenarios. However, under class-imbalanced setting, the model is easily biased during training, such that the generated pseudo-labels can be even more biased and severely degrades the performance of minority classes. Moreover, due to the confirmation bias issue (Tarvainen & Valpola, 2017; Arazo et al., 2020), the model itself is hard to rectify such a training bias.

Our goal is to enhance the existing pseudo-labeling SSL methods by making full use of both labeled and unlabeled data, while protecting the linear classifier from the training bias (imbalance bias and pseudo-label bias). To this end, we design to learn a bias adaptive classifier that equips the linear classifier with a bias attractor in order to assimilate complicated training bias.

The proposed bias adaptive classifier: As shown in Fig. 2. The bias adaptive classifier (denoted as $F$) consists of two modules: the linear classifier $f_{\phi}^{\rm cls}$ and a nonlinear network $\Delta f_{w}$ (dubbed bias attractor). The bias attractor is implemented by imposing a residual transformation on the output of the linear classifier, i.e., plugging $\Delta f_{w}$ after $f_{\phi}^{\rm cls}$ and then bridging their outputs with a shortcut connection.

Mathematically, the bias adaptive classifier $F$ can be formulated as

where $\mathbf{z}=f_{\theta}^{\rm ext}(\mathbf{x})\in\mathbb{R}^{d}$, $\mathbf{I}$ denotes identity mapping, and $\Delta f_{w}$ with parameters $\omega$ is a multi-layer perceptron (MLP) with one hidden layer in this paper. We design such a bias adaptive classifier for the following two considerations. On the one hand, the bias attractor $\Delta f_{w}$ adopts a nonlinear network which can assimilate complicated training bias in theory due to the universal approximation properties ¹¹1In theory, an MLP can approximate almost any continuous function (Hornik et al., 1989).. Since the whole bias adaptive classifier (i.e., classifier with the bias attractor) is required to fit the imbalanced training data, we hope the bias attractor could indeed help the linear classifier to learn the unbiased class conditional distribution (i.e., let the classifier less effected by the biases.). By contrast, in the original classification network, a single linear classifier is required to fit biased training data such that it is easily misled by class imbalance and biased pseudo-labels. On the other hand, the residual connection conveniently makes the bias attractor a plug-in module, i.e., assimilate the training bias during training and be removed in the test stage, and it also has been proven successful in easing the training of deep networks (He et al., 2016; Long et al., 2016).

Learning bias adaptive classifier: With the proposed bias adaptive classifier $F_{\omega,\phi}$, the modified classification network can be formulated as $\hat{\Psi}=F_{\omega,\phi}\circ f_{\theta}^{\rm ext}$. To make full use of the whole training data ($\mathcal{D}_{l}\cup\mathcal{D}_{u}$) for better representation learning, we can minimize the following loss function:

This involves an optimization problem with respect to three parts of parameters $\{\theta,\phi,\omega\}$, which can be jointly optimized via the stochastic gradient decent (SGD) in an end-to-end manner. However, such a training strategy poses a critical challenge: there is no prior knowledge on $f_{\phi}^{\rm cls}$ to predict unbiased label prediction and on $\Delta f_{w}$ to assimilate the training bias. In other words, we cannot guarantee the training bias to be exactly decoupled from the linear classifier $f_{\phi}^{\bm{c}ls}$. To address this problem, we take $\omega$ as hyper-parameters associated with $\phi$ and design a bi-level learning algorithm to jointly optimize the network parameters $\{\theta,\phi\}$ and hyper-parameters $\omega$.

We illustrate the process in Fig. 2 and Algorithm 1. In each training iteration $t$, we update the network parameters $\{\theta,\phi\}$ by gradient descent as

where $\alpha$ is the learning rate. Note that we herein assume that $\omega$ is only directly related to the linear classifier $f^{\rm cls}_{\phi}$ via $\phi^{t+1}(\omega)$, which implies that the subsequent optimization of $\omega$ will not affect the feature extractor $f^{\rm ext}_{\theta}$. We then tune the hyper-parameters $\omega$ to make the linear classifier $\phi^{t+1}(\omega)$ generalize well towards each class, and we thus minimize the following loss function of the network $\Psi=f_{\phi^{t+1}(\omega)}^{\rm cls}\!\!\circ\!f_{\theta^{t+1}}^{\rm ext}$ over a class-balanced set (dynamically sampled from the labeled training set):

where $\mathcal{B}\subset\mathcal{D}_{l}$ is a batch of class-balanced labeled samples, which can be implemented by class-aware sampling (Shen et al., 2016). This loss function reflects the effect of the hyper-parameter $\omega$ on making the linear classifier generalize well towards each class, we thus optimize $\omega$ by

where $\eta$ is the learning rate on $\omega$. Note that in Eq. 6 we need to compute a second-order gradient $\nabla_{\omega}\mathcal{L}^{bal}$ with respect to $\omega$, which can be easily implemented through popular deep learning frameworks such as Pytorch (Paszke et al., 2019) in practice.

In summary, the proposed bi-level learning framework ensures that 1) the linear classifier $f_{\omega}^{\rm cls}$ can fit unbiased class-conditional distribution by minimizing the empirical risk over balanced data via Eq. 5 and 2) the bias attractor $\Delta f_{\omega}$ can handle the implicit training bias by training the bias adaptive classifier $F_{\omega,\phi}$ over the imbalanced training data by Eq. 3. As such, our proposed $\Delta f_{w}$ can protect $f_{w}^{\rm cls}$ from the training bias. Additionally, our L2AC is more efficient than most existing bi-level learning methods (Finn et al., 2017; Ren et al., 2018; Shu et al., 2019). To illustrate this, we herein give a brief complexity analysis of our algorithm. Since our L2AC introduces a bi-level optimization problem, it requires one extra forward passes in Eq. 5 and backward pass in Eq. 6 compared to regular single-level optimization problem. However, in the backward pass, the second-order gradient of $\omega$ in Eq. 6 only requires to unroll the gradient graph of the linear classifier $f^{\rm cls}_{\phi}$. As a result, the backward-on-backward automatic differentiation in Eq. 6 demands a lightweight of overhead, i.e., approximately $\frac{\#{\rm Params}(f^{\rm cls}_{\phi})}{\#\rm{Params}(\Psi)}\times$ training time of one full backward pass.

Note that the update of the parameter $\omega$ aims to minimize the problem Eq. 5, we herein give a brief debiasing analysis of our proposed L2AC by showing how the value of bias attractor $\Delta f(\cdot)$ change with the update of $\omega$. We use notation $\frac{\partial f}{\partial\omega}|_{\omega^{t}}$ to denote the gradient operation of $f$ at $\omega^{t}$ and superscript $T$ to denote the vector/matrix transpose. We have the following proposition.

This shows that the update of the parameter $\omega$ affects the value of the bias attractor, i.e, the value of $\Delta f_{i}$ is adjusted according to the interaction $G_{i}$ between $\mathbf{x}_{i}$ and $\mathcal{B}$. If $G_{i}>0$ then $\Delta f_{i,k}$ is increased, otherwise $\Delta f_{i,k}$ is decreased, indicating L2AC adaptively assimilate the training bias.

Dataset. We follow the same experiment protocols as Kim et al. (2020a). In detail, a labeled set and an unlabeled set are randomly sampled from the original training data, keeping the number of images for each class to be the same. Then both the two sets are tailored to be imbalanced by randomly discarding training images according to the predefined imbalance ratios $\gamma_{l}$ and $\gamma_{u}$. We denote the number of the most majority class within labeled and unlabeled data as $N_{1}$ and $M_{1}$, respectively, and we then have $N_{k}=N_{1}\cdot\gamma_{l}^{\epsilon_{k}}$ and $M_{k}=M_{1}\cdot\gamma_{u}^{\epsilon_{k}}$, where $\epsilon_{k}={\frac{k-1}{K-1}}$. We initially set $N_{1}=1500$ and $M_{1}=3000$ following Kim et al. (2020a), and further ablate the proposed L2AC under various labeled ratios in Section 4.4. The test set remains unchanged and class-balanced.

Setups. The experimental setups are consistent with Kim et al. (2020a). Concretely, we employ Wide ResNet-28-2 (Oliver et al., 2018) as our backbone network and adopt Adam optimizer (Kingma & Ba, 2015) for 500 training epochs, each of which has 500 iterations. To evaluate the model, we use its exponential moving average (EMA) version, and report the average test accuracy of the last 20 epochs following Berthelot et al. (2019b). See Section D.1 for more details.

Results under $\gamma_{l}=\gamma_{u}$. We evaluate the proposed L2AC based on two widely-used SSL methods: MixMatch (Berthelot et al., 2019b) and FixMatch (Sohn et al., 2020), and compare it with the following methods: 1) The Vanilla model merely trained with labeled data; 2) Recent re-balancing methods that are trained with labeled data by considering class imbalance, including: Re-sampling (Japkowicz, 2000), LDAM-DRW (Cao et al., 2019) and cRT (Kang et al., 2020); 3) Recent imbalanced SSL methods, including: DARP (Kim et al., 2020a), CReST+ (Wei et al., 2021), ABC (Lee et al., 2021), SaR (Lai et al., 2022) and DASO (Oh et al., 2022). Please refer to Appendix C for more details about these methods. The main results are shown in Table 1. It can be observed that L2AC significantly improves MixMatch and FixMatch at least 9% absolute gain on bACC and at least 14% on GM for all settings. This implies that our L2AC benefits the two baselines by learning an unbiased linear classifier. Moreover, our L2AC consistently surpasses all the comparison methods over both evaluation metrics. Take the extremely imbalanced case of $\gamma=150$ for example, compared with the second best comparison method, our L2AC achieves up to 3.6% bACC gain and 4.9% GM gain upon MixMatch, and around 3.0% bACC gain and 3.6% GM gain upon FixMatch.

Results under $\gamma_{l}\neq\gamma_{u}$. The class distribution of labeled and unlabeled data can be arguably different in practice. We herein simulate two typical scenarios following Oh et al. (2022), i.e., the unlabeled set follows an uniform class distribution ($\gamma_{u}=1$) and a reversed long-tailed class distribution against the labeled data ($\gamma_{u}=100$ (reversed)). Note that CReST+ (Wei et al., 2021) fails in this case as they require the class distribution of unlabeled data as prior knowledge for training. As shown in Table 1, L2AC can consistently improve both MixMatch and FixMatch by a large margin. An interesting observation is that the baselines MixMach and FixMatch under $\gamma_{u}=1$ perform much worse than that under $\gamma_{u}=100$, even if more unlabeled data are added for $\gamma_{u}=1$. This is mainly because the models under imbalanced SSL setting have a strong bias to generate incorrect labels for the tail classes, which will impair the entire learning process. As a result, more unlabeled tail class samples under $\gamma_{u}=1$ lead to more severe performance degradation. On the contrary, our L2AC can eliminate the influence of the training bias and predict high-quality pseudo-labels for unlabeled data, such that it achieves significant performance gain in both settings.

Dataset: To make a more comprehensive comparison, we further evaluate L2AC on CIFAR-100 (Krizhevsky et al., 2009) and STL-10 (Coates et al., 2011). For CIFAR-100, we create labeled and unlabeled sets in the same way as described in Sec. 4.1 and set $N_{1}=150$ and $M_{1}=300$. STL-10 is a more realistic SSL task that has no distribution information for unlabeled data. In our experiments, we set $N_{1}=450$ to construct the imbalanced labeled set and adopt the whole unknown unlabeled set (i.e., $M=100$k). It is worth noting that the unlabeled set of STL-10 is noisy as it contains samples that do not belong to any of the classes in the labeled set.

Results: As shown in Table 2, L2AC achieves the best performance over GM and competitive performance over bACC compared to current state-of-the-art ABC (Lee et al., 2021) and DASO (Oh et al., 2022). This implies that our approach obtains a relatively balanced classification performance towards all the classes. While for SLT-10, a more realistic noisy dataset without distribution information for unlabeled data, L2AC significantly outperforms ABC and DASO on both bACC and GM, which demonstrates that it has greater potential to be applied in the practical SSL scenarios.

Dataset: SUN397 (Xiao et al., 2010) is an imbalanced real-world scene classification dataset, which originally consists of 108,754 RGB images with 397 classes. Following the experimental setups in Kim et al. (2020a), we hold-out 50 samples per each class for testing because no official data split is provided. We then construct the labeled and unlabeled dataset according to $M_{k}/N_{k}=2$. The comparison methods includes: Vanilla, cRT (Kang et al., 2020), FixMatch (Sohn et al., 2020), DARP (Kim et al., 2020a) and ABC (Lee et al., 2021). More training details are presented in Section D.2.

Results: The experimental results are summarized in Table 3. Compared to the baseline FixMatch (Sohn et al., 2020), our proposed L2AC results in about 4% performance gain over all evaluation metrics, and outperforms all the SOTA methods. This further verifies the efficacy of our proposed method toward the real-world imbalanced SSL applications.

What about the performance under various label ratios? To answer this question, we vary the ratios of labeled data (denoted as $\beta$) on CIFAR-10 and STL-10 to evaluate the proposed method. For CIFAR-10, we define $\beta=N_{1}/(N_{1}+M_{1})$ and set the imbalance ratio $\gamma=100$. For STL-10, since it does not provide annotations for unlabeled data, we re-sample the labeled set from labeled data by $\beta=N_{1}/500$ and set the imbalance ratio $\gamma_{l}=10$. As shown in Table 4, our L2AC consistently improves the baseline across different amounts of labeled data on both CIFAR-10 and STL-10. For example, STL-10 with $\beta=5\%$ contains very scarce labeled data, where only 25 and 3 labeled samples belong to the most majority and minority classes, respectively. In such an extremely biased scenario, our L2AC significantly improves FixMatch by around 16% over bACC and 37% over GM.

How does L2AC perform on the majority/minority classes? To explain the source of performance improvements, we further visualize the confusion matrices on the test set of CIFAR-10 with $\gamma=100$. Noting that the diagonal vector of a confusion matrix represents per-class recall. As shown in Fig. 3, our L2AC provides a relatively balanced per-class recall compared with the baseline FixMatch (Sohn et al., 2020) and other imbalanced SSL methods, e.g., DARP (Kim et al., 2020a) and ABC (Lee et al., 2021). It can also be observed that FixMatch easily tends to misclassify the samples of the minority classes into the majority classes, while our L2AC largely alleviates this bias. These results reveal that our L2AC provides an unbiased linear classifier for the test stage.

Could L2AC improve the quality of pseudo-labels? Qualitative and quantitative experiment results have shown that the proposed L2AC can improve the performance of pseudo-labeling SSL methods under different settings. We attribute this to the fact that L2AC can generate unbiased pseudo-labels during training. To validate this, we show per-class recall of pseudo-labels for CIFAR-10 with $\gamma_{l}=\gamma_{u}=100$ and $\gamma_{l}=100,\gamma_{u}=100$ (reversed) in Fig. 5. It is clear that L2AC significantly raises the final recall of the minority classes. Especially for the situation where the distribution of labeled and unlabeled data are severely mismatched, as shown in Fig. 5(b), our L2AC considerably improves the recall of the most minority class by around 60% upon FixMatch. Such a high-quality pseudo-label estimation probably benefits from a more unbiased classifier with a basically equal preference for each class.

Ablation analysis. We conduct an ablation study to explore the contribution of each critical component in L2AC. We experiment with FixMatch on CIFAR-10 under $\gamma_{l}=\gamma_{u}=100$ and $\gamma_{l}=100,\gamma_{u}=100$ (reversed). 1) We first verify whether the bias attractor is helpful. To this end, we apply the proposed bias attractor to FixMatch. It can be seen from Table 5 that the bias attractor helps improve the performance to a certain extent, indicating that the bias attractor is effective yet not significant. As analyzed in Section 3.2, there is no prior knowledge for the bias attractor to assimilate the training bias in this plain training manner. 2) Next, we study how important the role bi-level optimization plays in our method. Instead of using a bi-level learning framework, we disengage the hierarchy structure of our upper-level loss and lower-level loss, and reformulate a single level optimization problem as $\mathcal{L}+\lambda\mathcal{L}^{bal}$. The results in Table 5 show that such a degraded version of L2AC provides substantial performance gain over FixMatch w/ bias attractor while still inferior to our L2AC. This demonstrates the effect of the proposed bias adaptive classifier and bi-level learning framework on protecting the linear classifier from the training bias.