Enhancing Self-Training Methods

Supervised deep learning models trained with clean one-hot ground truth labels generally employ a categorical cross-entropy loss known as a hard loss (using a known, single ground truth label for each example and associated softmax prediction). ST techniques must handle both clean ground truth labels and noisy pseudo-labels generated from the softmax prediction vectors of the teacher model on unlabeled data. Previous work [34, 1] has shown that using a soft loss (used in NS) with the entire softmax vector of pseudo-label predictions as targets in a full cross-entropy loss works better than a hard loss, with a softmax distribution over all training classes, helping to reduce overfitting to noisy targets.

ST is an iterative technique that trains a new student model for every iteration. Two main methods exist for initializing the student model for each ST iteration. Fresh-training (used in NS) initiates every student model from scratch (i.e., newly initialized model weights). Conversely, fine-tuning uses the weights of the model from any previous iterations with the best accuracy (on validation data) to initialize the student model and then fine-tune the weights during the current iteration.

Student models in ST learn from the labeled ground truth subset and the unlabeled subset with noisy pseudo-labels. ST commonly uses a randomly collected (uniform) mini-batch (used in NS) that tends to overfit due to their bias towards selecting a larger number of noisy pseudo-labeled than labeled data during training, as the unlabeled subset sizes are usually much larger than the labeled subset. We propose a custom SplitBatch approach that collects a user-specified split of labeled and pseudo-labeled examples for every mini-batch. Our approach uses bootstrapping to select examples from the limited labeled subset containing only clean labels, providing an additional regularization effect to counter the noise from the pseudo-labels. The user controls the hyperparameter that sets the ratio of labeled to pseudo-labeled examples in a mini-batch, making the approach adaptive to differently sized labeled/pseudo-labeled subsets. For example, a larger ratio of labeled to pseudo-labeled examples can be used for datasets having a large number of labeled samples.

With a split batch of labeled and pseudo-label examples, the loss function should similarly engage a split loss. In this work, we combine a labeled loss $L_{lab}$ and a pseudo-labeled loss $L_{pslab}$ with equal contributions into a custom MixedLoss function

$$L_{mix}=\lambda_{b}L_{lab}+(1-\lambda_{b})L_{pslab}$$

(1)

where $\lambda_{b}$ is a hyperparameter set to 0.5 to balance the loss between the labeled and pseudo-labeled examples in all of our experiments.

ST methods can easily generate class-unbalanced and low-confidence pseudo-labeled subsets that can also increase confirmation bias. NS uses a naïve softmax-thresholded class balancing technique that first uses the uncalibrated softmax scores of pseudo-label predictions to threshold high-confidence predictions (softmax scores for the argmax class > 0.3). NS then samples a user-specified number of thresholded pseudo-labeled examples that have the highest softmax confidence across every class, oversampling examples from classes not having enough pseudo-labeled examples (less than the user-specified count per-class). This method requires a handcrafted softmax threshold and per-class sampling count for every dataset.

Our previously defined SplitBatch sampler can be adapted to also dynamically re-weight and sample pseudo-labeled examples using two different sample weightings. The first weighting uses inverted per-class lengths ($\frac{1}{N_{c}}$ where $N_{c}$ is the number of pseudo-labeled examples belonging to class $c$). This method assigns larger weights to classes with a lower number of pseudo-labels which thus will be oversampled during training. The second set of sample weightings uses the per-class normalized softmax confidence scores

$$normalizedSoftmax=\frac{max(\tilde{y})}{max(S_{c})}$$

(2)

where $\tilde{y}$ is the complete pseudo-label softmax vector prediction by the teacher model having argmax predicted class $c$ for a given unlabeled data sample $\tilde{x}$, and $S_{c}=\{max(\tilde{y_{1}}),...,max(\tilde{y}_{N_{c}})\}$ is the set of max softmax scores for all pseudo-label predictions belonging to class $c$. These normalized softmax scores scale the weights per-class to avoid oversampling from classes with higher softmax confidence. We average the two sampling weights (class-length and the normalized softmax-confidence-based weights) and distribute them across the unlabeled examples to dynamically obtain the final sampling weights for every dataset without the need for handcrafted thresholding and class-balancing.

NS uses the naïve softmax thresholding approach (described above), employing softmax scores as a metric to determine pseudo-label confidence. However, modern deep neural networks are known to be poorly calibrated [9], implying that the softmax prediction probabilities do not accurately represent the true likelihood of the predictions. Hence, the uncalibrated softmax score is a poor confidence metric for rejecting noisy samples and thus can increase confirmation bias.

Alternatively, we propose adding a temperature-scaling calibration [9] step in the ST pipeline to the current teacher model for generating calibrated pseudo-label softmax predictions. We use a grid search over 400 linearly spaced temperature values between 0.05 and 20 and choose the optimal value, denoted by $\tau$, with the lowest Expected Calibration Error [21] on the validation data. We then apply $\tau$ to soften/sharpen the softmax pseudo-label predictions of the teacher model to get a full softmax vector of pseudo-labels

$$\tilde{y}=softmax\left(\frac{\mathit{f_{T}}\left(\tilde{x}\right)}{\tau}\right)$$

(3)

where $\mathit{f_{T}}\left(\tilde{x}\right)$ denotes the output logits of the teacher model for a given unlabeled sample $\tilde{x}$ and $\tilde{y}$ is the calibrated pseudo-label softmax vector.

We next propose using entropy thresholding of the calibrated softmax pseudo-label vector rather than simply thresholding the softmax score for the argmax class to determine if the pseudo-label is acceptable. We calculate the normalized entropy (dividing by $log\left(N_{c}\right)$ for $N_{c}$ classes) of the calibrated pseudo-labels of validation data and then run a grid search over 500 threshold values between 0 and 1. We then calculate the true-positive rate (TPR) and false-positive rate (FPR) for each of the possible entropy thresholds on the validation data. We perform ROC analysis [7] by plotting the TPR against the FPR at the various thresholds and selecting the optimal threshold with the lowest Euclidean distance to the top left corner (optimal/perfect classification) of the ROC curve. This method for pseudo-label selection can dynamically adapt to different datasets, unlike the naïve approach of using handpicked softmax thresholds for each dataset.

Lastly, the NS approach uses a smaller-sized initial teacher model trained on the clean labels and a larger student model (and thus a larger subsequent teacher) trained jointly on the labeled and pseudo-labeled examples. As previously mentioned, the NS student model incorporates model noise (Dropout) and data noise (strong data augmentation techniques). A natural alternative to their approach is a same-sized teacher-student model, where the teacher and student have the same model size but use stronger data augmentation techniques (SAMix+RandAugment). We compare both small and large same-sized teacher-student models.

Given the above-listed design choice alternatives, we now compare them in an experimental setting to enhance the basic ST pipeline.

We created custom labeled/unlabeled subsets from various benchmark datasets (SVHN [22], CIFAR-10 [13], and CIFAR-100 [13]) following the standard subset sizes from previous SSL work [1], as shown at the top of Table 2. For each dataset, we also created a validation subset with ground-truth labels for hyperparameter tuning and evaluating model performance during training, and a corresponding test subset for evaluating model inference. We first evaluated the experiments described in Table 1 on the three datasets and constructed the enhanced approach using the best component choices. We further evaluated the generalization performance of the resulting enhanced approach with different labeled/unlabeled dataset splits and model sizes on additional datasets (CINIC-10 [5], TinyImageNet [15]) as shown at the bottom of Table 2. We also compared the performance of our resulting enhanced approach with the original NS method on all the datasets. Note that every dataset except SVHN is class-balanced. Finally, we extended the enhanced approach with a basic Open Set detection technique to help filter out (suppress) additional/unwanted classes in a custom-built Open Set version of CIFAR-10/CIFAR-100 with 110 separate classes.

We trained a supervised baseline model for each dataset on only the labeled data subset for three runs initialized with different random seeds for each experiment in the roadmap (Exp. 1 to 6). We reported the best mean test set score obtained across three student iterations (unless otherwise mentioned for experiments below). We used a ResNet(R)-18 [12] for SVHN and a WideResNet(WRN) 28-2 [37] + SAMix for the CIFAR datasets, unless otherwise mentioned. SAMix was not used on SVHN as certain mixing data augmentation policies were expected not to be appropriate for digit classification datasets (e.g., crops, flips). We also applied RandAugment to each model/dataset with the hyperparameter settings for each dataset given in the original work [4]. RandomCrop and RandomHorizontalFlip were included in the data augmentation policy for the CIFAR datasets only. We trained the initial teacher model for 400 epochs on the labeled subsets of all datasets following the suggested training and hyperparameter settings for SAMix [16]. Each student iteration was trained for 100 epochs (which actually includes more mini-batches per-epoch than the teacher). We used a batch size of 100 for all experiments. Table 3 shows the mean initial teacher accuracy trained only using the labeled subset (note that these scores are expected to be lower than fully-supervised SOTA benchmarks that use the complete datasets).

We next evaluated the generalizability of the EST model with the different labeled/unlabeled subset sizes and larger model architectures. For these experiments, a single run for each experiment is reported.

As expected, the results in Table 10 show that performance increases as we add more unlabeled examples during training in both scenarios, highlighting the importance of building large unlabeled datasets for semi-supervised learning methods. Our EST approach provided significant improvements in the small labeled scenario while providing slight improvements to the large labeled scenario (as the teacher model already has enough labeled examples for better learning performance).

Real-world unlabeled data can be from an Open Set that contains data belonging to the target classes (classes from labeled training data) and data from additional non-target classes. The inclusion of non-target class examples in the unlabeled set can degrade SSL performance [10]. We propose a basic Open Set recognition technique using contrastive learning to build a feature space for all target classes, where the non-target classes should hopefully be farther away from the target classes. We used SimCLR [3] to learn a contrastive feature space from the labeled target data and the unlabeled data (contains target and non-target class examples). We used a validation set to find a mean prototype vector for each known target class and fit a Beta distribution per-class of the distances from labeled target examples to their class prototype. We then filtered out examples expected to be from any non-target class by using a per-class Beta cumulative distribution function (CDF) and a global CDF threshold (learned from validation), where examples having CDF values above the threshold for all classes were considered to be from a non-target class.

We evaluated this method on a custom-built Open Set version of CIFAR-10/100 with labeled and unlabeled subsets. The labeled subset is made up of a Closed Set with 10 target classes (CIFAR-10 subset consisting of 4K images), and the unlabeled subset contains 110 total classes with 10 target classes (CIFAR-10 subset consisting of different 42K images) and 100 non-target classes (CIFAR-100 subset consisting of 42K images). We compared the performance of the NS approach (reimplemented as suggested in [34] with a smaller WRN28-2 initial teacher and the same larger WRN28-8 student as EST, thus resulting in a performance decrease) to our EST approach (same-sized WRN28-8 teacher-student). After searching through multiple values, we found that CDF thresholds of 0.85 and 0.9 led to the best Closed Set validation accuracy for NS and EST, respectively. From the results in Table 12, we observed that our EST approach performed better than the NS approach showing that our proposed enhancements for handling noisy pseudo-labels extend to Open Set data as well by providing some basic filtering of pseudo-labels belonging to non-target classes. We also found that both the NS and EST approaches had further improvements upon training with our filtered Open Set data, with our EST approach on filtered Open Set data performing the best.