Semi-supervised Contrastive Learning with Similarity Co-calibration

The similarity distribution $p_{s}$ over the classes is obtained as follows. Given a list of $n$ few shot labeled samples $\bm{X}_{c}=\{x_{i}\}_{i=1}^{n}$ for class $c$, we compute the feature representation $\bm{v}_{i}$ of $x_{i}$ with current encoder $E_{q}$ where $\bm{v}_{i}=E_{q}(x_{i})$. The feature representation of class $c$ is simply obtained by averaging the features, i.e., $\bm{\tilde{v}}_{c}=\sum\bm{v}_{i}/nZ$, where $Z$ is a normalizing constant value. Given an unlabeled sample $x^{\prime}$ with normalized feature $\bm{\tilde{v^{\prime}}}$, we can get its cosine similarity distribution $p_{s}=\{s_{1},s_{2}...,s_{c}\}$ with $C$ predefined prototypes, where $s_{k}$ denotes the similarity between ($\bm{\tilde{v^{\prime}}}$, $\bm{\tilde{v}}_{k}$).

Inspired by [2], we introduce a form of fairness prediction $\hat{p}_{s}$ by maintaining a running average of distribution $p_{s}$ (average vector of $p_{s}$ over the last $t$ batches, here we simply set $t=128$) over the course of training. Overall, $\hat{p}_{s}$ can be treated as a global adjustment among the scores for each class, and thus avoid biased prediction $p_{b}$ that conducting over a single image $u_{b}$. Then the calibration is simply conducted by scaling $p_{b}$ with $\hat{p_{s}}$ and then re-normalizing to form a valid probability distribution $\hat{p}_{b}$. We use $\hat{p}_{b}$ as calibrated pseudo labels for optimization following Eq. 2.

The similarity distribution $p_{s}$ requires a predefined class specific prototype generated by the few labeled samples, and each prototype $\bm{\tilde{v}}_{c}$ is simply computed using the available ground truth labels belonging to class $c$. However, this representation is easily biased to the few shot samples, which is insufficient to model the real distribution. To solve this issue, we propose a prototype mixture strategy to refine the feature representation of each class in a robust way. The new samples are generated via a simple mixup [36] operation, conducted between the mined samples and those with ground truth labels. In this way, based on the continuity of feature space, we are able to ensure that the generated samples do not drift away too much from the real distribution of that class, while enriching the feature representation in a robust way. Specifically, given the few shot labeled sample set $\bm{X}_{c}$ for class $c$ and a list of nearest neighborhood unlabeled samples $\Omega_{c}$ that belong to class $c$, we randomly select a ground truth sample $x_{c}\in\bm{X}_{c}$, and a nearest sample $x_{nearest}\in$ $\Omega_{c}$, and mix them to generate a new sample $\hat{x}$:

where $\lambda$ is the combination factor and is sampled from beta distribution Beta$(\alpha,\alpha)$ with parameter $\alpha$. In our implementation, we simply set $\alpha=1$, and $\hat{x}$ will be added to $\bm{X}_{c}$ only use for calculating prototype for class $c$. The purpose of this step is to smooth the noise caused during the sample selection process. We visualize the benefits of this step in Fig. 3. Benefiting from the mixed samples, the feature representation is refined to better represent the class prototype. In the experimental section, we would validate its effectiveness.

In order to efficiently make use of the unlabeled samples in a robust way, inspired by [28], we adjust the original contrastive loss from [12] in three aspects. First, we extend it to allow for multiple positives for each forward propagation; second, we introduce a margin value $m$ which enforces class separability for better generalization, which we find is robust for data mining; third, considering that the mined samples may not be reliable, especially during the initial training stages, we introduce a soften factor $\alpha_{p}$ for each mined positive sample, and adaptively penalize the discrepancy according to its similarity to the corresponding class. For ease of expression, we make a deformation of original contrastive loss by using $s_{p}$ to represent the similarity of positive samples while $s_{n}$ represents the similarity of negative samples, and denote $\gamma=1/\tau$. After introducing margin $m$ and soften factor $\alpha_{p}$, the loss $\mathcal{L}_{ctr}$ can be reformulated as:

Designed with the above three properties, the loss is well suited for semi-supervised learning. In the following, we would analyze its advantages in detail.

For this branch, the positive samples are simply selected from the pseudo label loss branch. Specifically, given an unlabeled sample $u_{b}$ with pseudo labels $p_{b}$, which is predicted by the $fc$ layers of the cross entropy term. We assign it to class $c$ if $argmax\,\,p_{b}=c$, denoting that this sample is most similar with class $c$. Then we pulled $u_{b}$ together with those samples that have ground truth label $c$. In this way, the unlabeled samples are pulled towards the specific instances corresponding to that class, and the representation is more compact. In practice, all the positive samples are maintained in the key encoder $E_{k}$. Benefit from the moment contrast mechanism in MoCo[12], the encoder $E_{k}$ is asynchronously updated and add more positive keys doesn’t increase the computation complexity.

A critical issue when using cosine similarity for positive sample selection is that at the initial training stages, the model is under-fitted, and the accuracy of using the nearest neighbor is very low. As a result, the incorrect positive samples would be assigned to a class, causing the model to update in the wrong direction, which is harmful for generalization. To solve this issue, we propose a self-paced learning method to enhance the optimization flexibility and it is robust to the noisy samples. This is achieved by assigning each selected positive sample a weight factor $\alpha_{p}$, which is determined by the similarity between the current sample and the predefined most similar class prototype. It is equivalent to adding a penalty term to the loss function, and with the increased accuracy of positive sample selection, the penalty becomes larger, and in turn enforces the compact representation.

We now step deep into the optimization objective Eq. (3.2) to better understand its properties. Towards this goal, we first make some approximate transformations using the following formulations:

From above formulas, we can transform the Eq.  (3.2) to:

The specific derivation process will be shown in the Supplementary. From the above equations, we have the following observations: (1) According to Eq. (10), the objective is to optimize the upper bound of positive pairs and lower bound of negative pairs, and introducing of $\alpha_{p}$ makes this process more flexible; (2) Based on the properties of Eq. (6) and Eq. (7), whose gradient is the $softmax$ function, the positive and negative pairs obtain equal gradient, regardless of the number of positive and negative sample pairs. This gradient equilibrium ensures the success of our multi-positive sample training.

As recommended by [26], we use Wide ResNet-28-2[34] as backbone, as well as training protocol, $\mu=4$, $B=64$, $\tau=0.95$, and all loss weights (e.g. $\lambda_{pl}$) are set as 1. The projection head is a 2-layer MLP that outputs 64-dimensional embedding. The models are trained using SGD with a momentum of 0.9 and a weight decay of 0.0005. We train our model for 200 epochs, using learning rate of 0.03 with a cosine decay schedule. For the additional hyperparamaters, we set the number of negative samples $K=4096$, $\gamma=5$. Besides, we use two different augmentations on unlabeled data, The $DA_{a}$ here is the standard crop-and-flip, and we use RandAugment[8] combined with the augmentation strategy in [5] as our another augmentation $DA_{b}$. More details about hyperparameters will be shown in Supplementary.

We report the result of different labels settings with our method in Table 1, and compare with some well-known approaches. The improvement is more substantial when fewer labeled samples are available. We achieve $10.29\%$ error rate with only 40 labels, which is 3.5% better than the previous best performed method Fixmatch[26] with more robust capability.

For model training, we use ResNet-50[14] as the backbone, and use the SGD optimizer with weight decay of 0.0001 and momentum as 0.9. The model is trained on 32 Tesla V100 GPUs with batch size of 1024 for unlabeled data and 256 for labeled data, and the initial learning rate is 0.4 with cosine decay schedule. The data augmentation follows the same strategy as [5]. More hyperparameters details are shown in Supplementary.

We compare our method with several well-known semi-supervised learning approaches, which can be roughly categorized into two aspects,, i.e., the conventional semi-supervised learning strategy that makes use of cross entropy loss for data mining, as well as contrastive learning based methods that first pre-train the models without using any image-level labels, followed by directly fine-tuning with few shot labeled data. As shown in Table 2, We achieve top-1 accuracies of $60.2\%$ and $72.1\%$ with $1\%$ and $10\%$ labeled data, respectively, which significantly improves the baseline MoCo v2 [7]. Under $1\%$ setting, our method is also better than previous best performed method simCLR v2. It should be noted that simCLR v2 [6] makes use of extra tricks such as more MLP layers, which has been demonstrated to be effective for semi-supervised learning, while we simply follow the MoCo v2 baseline that makes use of two MLP layers. Under $10\%$ setting, our method is $0.6\%$ better than the previous best performed Fixmatch in top-1 accuracy, and $1.8\%$ higher in top-5 accuracy. We also find that previous semi-supervised works rarely report accuracy with $1\%$ labels due to their poor performance. The reason is that the extremely few labeled data is too biased to represent the corresponding class, and such error can be magnified during the data mining procedure using cross entropy loss, while our method is robust to the bias, and especially effective when few shot labels are available.

Our approach targets at pulling nearest class samples together with calculated prototypes, this is similar to the idea of some clustering based methods. So we use intra-class similarity as an indicator to measure our learned features. We compare our method with two classic clustering-based learning methods, SwAV[4] and Deepcluster v2 [3], and the results are shown in Table 4. The intra-class similarity is calculated by averaging cosine distance among all intra-class pairwise samples, and we report the average of 1000 classes on the ImageNet validation set. Our method consistently outperforms these two methods both in 1% and 10% labels available, which validates that it is better for discrimination when more class priors are available.

Table 5 provides some interesting results. If we use pretrained backbone and $fc$ head to predict unlabeled training data, we can get top-1 accuracy of 53.2%, while the accuracy drops to 45.3% if we use the nearest class in feature space as its label. However, the overlap of their correct prediction is only 60.1%, in another word, most of the labels that one method predicts wrong are correct in another method, which is the motivation of our co-calibration mechanism. We believe this will help the model learn complementary information during training, and it can be seen that our method gains 2.4% benefits with this operation.

We calculate the average accuracy of the training samples which can get the correct positive samples by assigning the nearest category before and after adding the mixture module. The results are shown in Fig. 5. We can see that this module brings about 8% accuracy improvement at most. In addition, the model with prototype mixture always keeps high accuracy in selecting positive samples, and more correct positive samples are undoubtedly beneficial to improve the performance.

The effect of this coefficient is equal to $\tau$ in [7, 5]. With too large $\gamma$, it will penalize more about the discrepancy, which will break the semantic structure of the embedding distribution, as it is shown in Figure 4. We set it in 10 in our ImageNet experiment.

We observe the influence of two weights $\lambda_{pl}$ and $\lambda_{ctr}$ on the performance and report the result in the Figure 4, where $\lambda_{pl}=5$ and $\lambda_{ctr}=10$ gives the best performance. But in the 10% labels setting, we need to increase the $\lambda_{pl}$ to 10 and decrease the $\lambda_{ctr}$ to 5 to get the best result. We think it is because the fewer labeled data, the worse the performance on predicting pseudo labels, which need a larger $\lambda_{ctr}$ to strengthen the contrastive regularization.

The comparison results of introducing $\alpha_{p}$ are shown in Table 6. It can be seen that introducing adaptive weight $\alpha_{p}$ is beneficial for improving classification accuracy. It brings about $4.9\%$ gains with fixed $\alpha_{p}$ (set as 1). This is mainly due to the reason that $\alpha_{p}$ assigns a lower penalty to the positive samples that are less similar to the assigned class, which reduces the influence of these relatively noisy samples during model training.

Inspired by [32], we set $m$ as negative because the pseudo labels are not always correct during training. Figure 4 shows the result of different settings, we observe that both positive and negative margins can improve the quality of learned features, so it is beneficial to design a less stringent decision boundary for better class separability.