An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning

In Semi-Supervised Few-Shot Learning (SSFSL), we have a large-scale dataset $\mathcal{D}_{base}$ containing many-shot labeled data from each base class in $\mathcal{C}_{base}$, and a small-scale dataset $\mathcal{D}_{novel}$ consisting of few-shot labeled data as a support set $\mathcal{S}$ from the category set $\mathcal{C}_{novel}$, as well as a certain number of unlabeled data $\mathcal{U}$ acquired also from $\mathcal{C}_{novel}$. Note that, $\mathcal{D}_{novel}$ is disjoint from $\mathcal{D}_{base}$ for generalization test. The task of SSFSL is to learn a robust classifier $f(\cdot;\theta)$ based on both $\mathcal{S}$ and $\mathcal{U}$ for making predictions on new queries $\mathcal{Q}$ from $\mathcal{D}_{novel}$, where $\mathcal{D}_{base}$ is utilized as auxiliary data.

Regarding the basic semi-supervised few-shot classification setting, it generally faces the $N$-way-$K$-shot problem, where only $K$ labeled data from $\mathcal{S}$ and $U$ unlabeled data from $\mathcal{U}$ per class are available to learn an $N$-way classifier. In this setting, queries in $\mathcal{Q}$ are treated independently of each other, and are not observed in $\mathcal{U}$. It is referred to as inductive inference.

For another important setting in SSFSL, i.e., transductive inference, the query set $\mathcal{Q}$ is observed also during training and joint with $\mathcal{U}$.

We compare our MUSIC with state-of-the-art methods in the literature in Table 1. As shown, our simple approach outperforms the competing methods of both generic few-shot learning and semi-supervised few-shot learning by a large margin across different few-shot tasks over all the datasets. Beyond that, we also report the results of solely using the pseudo-labeled negative or positive samples generated by our MUSIC, which is denoted by “Ours (only neg)” or “Ours (only pos)” in that table. It is apparent to observe that even only using negative pseudo-labeling, MUSIC can still be superior to other existing FSL methods. Moreover, compared with the results of only using positive pseudo-labeling, the results of only using negative are worse. It reveals that accurate positive labels still provide more information than negative labels [10].

In the transductive setup, it is available to access the query data in the inference stage. We also perform experiments in such a setup and report the results in Table 2. As seen, our approach can still achieve the optimal accuracy on all the four datasets, which justifies the effectiveness of our MUSIC. Regarding the comparisons between (only) using negative and positive pseudo-labels, it has similar observations as those in Table 1.

In real applications, it might not be realistic to collect a clean unlabeled set without mixing any data of other classes. To further validate the robustness of MUSIC, we conduct experiments with the distractive setup, i.e., the unlabeled set contains distractive classes which are excluded in the support set. In that case, positive pseudo-labels are more prone to error, while negative pseudo-labels have a much lower risk of error. Table 2 presents the comparison results and shows that our approach can perform as the best solution in all distractive semi-supervised few-shot classification tasks.

In order to analyze the performance in the case of different unlabeled samples, we perform our MUSIC under the variety-unlabeled semi-supervised setup and compare with state-of-the-arts, e.g., ICI [36], LST [17] and PLCM [9]. As shown in Figure 2, our approach significantly outperforms over these methods in different $K$-shot tasks of SSFSL. It further validates the effectiveness and generalization ability of our MUSIC.

As assumed previously, in such an extremely label-constrained scenario, e.g., 1-shot learning, it might be hard to learn an accurate classifier for correctly predicting positive pseudo-labels. In this sub-section, we conduct ablation studies by alternatively performing negative and positive pseudo-labeling to verify this assumption. In Table 4.3, different settings denote different orders of negative and positive pseudo-labeling in SSFSL. For example, “neg $\rightarrow$ pos $\rightarrow\cdots$” represents that we firstly obtain negative pseudo-labels by our MUSIC (without using the final positive labels) and update the model, and then we obtain positive pseudo-labels²²2The method of positive pseudo-labeling here is a baseline solution, which trains a classifier with cross-entropy and obtains the positive pseudo-label by the highest logits above a certain threshold (e.g., $0.7$). and update model, and so on. Regarding the iteration time, it is relevant to the number of $K$ in the $K$-way classification. In concretely, for $5$-way classification, our MUSIC returns the most confident negative pseudo-label in the current iteration and excludes it for the next iteration. Thus, after four times of “neg $\rightarrow$ pos”, all negative pseudo-labelings are finished and the results can be reported. Similarly, “pos $\rightarrow$ neg $\rightarrow\cdots$” means that we get the positive pseudo-labels first, followed by the negative ones. As the results shown in Table 4.3, we can see that obtaining negative pseudo-labels first obviously achieves better results than positive first, which shows that labeling negative pseudo-labels first can lay a better foundation for model training, and further answers the question in this sub-section as “YES”.

In our MUSIC, to further improve the probability confidence and then promote pseudo-labeling, we equip the minimum-entropy loss (MinEnt). We here test its effectiveness and report the results in Table 4.3. It can be found that training with MinEnt (i.e., the proposed MUSIC) brings 0.2$\sim$0.3% improvements over training without MinEnt in SSFSL.

We hereby verify the effectiveness and necessity of the reject option $\delta$ in MUSIC. The $\delta$ in our approach acts as a safeguard to ensure that the obtained negative pseudo-labels are as confident as possible. We present the results in Table 6, and can observe that MUSIC with $\delta$ achieves significantly better few-shot classification accuracy than MUSIC without $\delta$. Additionally, even without $\delta$, our approach can still perform well, i.e., the results being comparable or even superior to the results of state-of-the-arts.

As aforementioned, our approach works as a successive exclusion manner until all negative pseudo-labels are predicted, and eventually obtaining positive pseudo-labels. As pseudo-labeling conducting, it is interesting to investigate how the performance changes as the iteration progresses. We report the corresponding results in Figure 3. As shown, on each task of these two datasets, our approach all shows a relatively stable growth trend, i.e., 0.5$\sim$2% improvements over the previous iteration.

In this sub-section, we explicitly investigate the error rates of both negative and positive pseudo-labels predicted by our approach. We take 5-way-5-shot classification on miniImageNet and CUB as examples, and first present the pseudo-labeling error rates of negative labels in Table 7. Since the task is 5-way prediction, there are totally four iterations of negative pseudo-labeling in MUSIC reported in that table. Except for error rates, we also detailedly report the number of wrong labeled samples in each iteration, as well as the total number of labeled samples. Note that, in the third and forth iterations of negative pseudo-labeling, the total number of labeled samples are less than the number of unlabeled data (i.e., 250), which is due to the reject option in MUSIC. That is to say, those samples cannot be pseudo-labeled with a strongly high confidence. Meanwhile, we also see that, as the pseudo-labeling progresses, the error rates slowly increase, but the final error rate of negative labeling is still no higher than 6.7%. It demonstrates the effectiveness of our approach from a straightforward view.

On the other side, Table 8 compares the positive pseudo-labeling error rates, and also reports the proportion of labeled samples in the total number of unlabeled samples. Regarding ICI [36] and iLPC [15], although they designed tailored strategies to ensure the correctness of pseudo-labels, e.g., instance credibility inference [36] and label cleaning [15], these methods still have high pseudo-labeling error rates (over 25%). Compared with them, our approach has significantly low error rates, i.e., about 10%. Meanwhile, we also note that our MUSIC only predicts about 80% of the unlabeled data, which can be regarded to be relatively conservative. However, it reveals that our approach still has a large room for performance improvement. Moreover, Table 8 also shows that, even if our approach removes the reject option strategy, its error rates are still lower than those of state-of-the-arts.

Additionally, we visualize the positive pseudo-labels with high confidence by $t$-SNE [32] in Figure 4. Compared with these methods, we can obviously find that the positive samples with high confidence predicted by our MUSIC are both more centralized and distinct. This also explains the satisfactory performance of our approach when using the positive pseudo-labels and using the positive alone (cf. Table 1 and Table 2) from the qualitative perspective.

In this sub-section, we are still interested in investigating what kind of data distribution the pseudo-labeled samples are to further analyze how our approach works well. As shown in Table 9, we present the averaged number of both negative and positive pseudo-labeled samples in all 600 episodes of 5-way-5-shot classification tasks on miniImageNet. It is apparent to see that the pseudo-labeled samples present a very clearly balanced distribution, which aids in the modeling of classifiers across different classes in SSFSL.