Self-semi-supervised Learning to Learn from Noisy Labeled Data

As discussed in the previous work (Arazo et al. 2019 [2]), DNN always fit clean label first, so data with clean label in general has lower loss than that with noisy label in an early stage. Hence, the author fits a two component Gaussian Mixture Model (GMM) on the loss distribution which can output samples’ probabilities of being clean and noisy. Unfortunately, there are two issues with this method. First, GMM is not good at classifying labels whose probability of being clean is close to that of being noisy. Second, DNN starts to memorize noisy labels before it learns enough information from clean labels, making the classification of labels inaccurate. Therefore, in this project, we innovate on noisy-label detection methods to deal with these two issues.

Our first innovation is the incorporation of a GMM fitted on the confidence of prediction (confirmation bias), along with one fitted on the cross-entropy loss as before. Based on our experiments, clean-labeled data shows higher confidence of its prediction than noisy-labeled data, and this difference in confidence remains throughout the training. Therefore, we can derive more knowledge on whether a label is clean or noisy though this new GMM.

Our second innovation is the ways to combine information from the two GMMs above. We have tried two different decision strategies, hard and elastic decision, as follows:

After detecting noisy labels, we innovated on the models’ objective functions so that our models can accommodate data with various probabilities to be clean. Inspired by training on partially-labeled data with self-semi-supervised learning, we proposed an approach to apply self-supervised technique on noisy labels, and in this case we adopt rotation recognition as the self-supervised learning task. Before fed into the model, each image $x$ is randomly rotated an angle from $\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\}$, and tagged with the label of rotation $y_{rot}$ besides the label of classification $y_{class}$. In our architecture, two parallel linear layers at the end of Pre-Activated ResNet18 are used for two different tasks: predicting classification label $y_{class}$ and predicting rotation label $y_{rot}$.

Overall, we aimed at preventing the model from memorizing the wrong information of noisy labels by distracting the samples detected to be noisy with the self-supervised learning task. There are two different methods to realize the distraction: regularizing the model through self-supervised loss, or separating the training set to train clean and noisy samples with the classification task and the rotation label prediction task respectively.

We explored the noisy detection capacities of our proposed decision. At each epoch, the weights computed by our decision policy will make a prediction on differentiating clean from mislabeled data. The accuracy of clean label prediction is compared among different proposed decision, shown in Figure 1.b. Hard decision outperformed among elastic decision and previous loss-only decision which indicates the salient signal from prediction confidence change improve the noisy detection capacity. As the lack of clean data problem discussed in section 3.1.1, we evaluated the percentage of predicted clean data which is around $31.5\%$ for hard decision (under 0.4 noise rate). From the test accuracy comparison (shown in Figure 1.c), we verified our hypothesis, even though hard decision performed better than elastic decision in noisy detection, elastic decision is better at promoting DNN to learn enough information from clean labels while maintaining a decent noisy label detection capacity.

Due to the augmentation (rotation) of data, there are two approaches to predict classification label for each image. We can either generate one prediction for each image without any rotation (1-image prediction) or average four predictions for each image rotated four times (4-rotation prediction). We conduct four controlled experiments and Figure 2.a shows the learning curves of several models with tuned hyper parameters on the validation set:

Baseline vs. Augmentation: to test whether the data augmentation by rotation improves over the baseline model, we trained an augmentation model using the same loss function as the baseline model, i.e., $L(y_{class},\hat{y}_{class})=CE(y_{class},\hat{y}_{class})$. The best validation accuracy the augmentation model achieved is roughly the same as the baseline model for 1-image prediction, but is around 2-percent higher for 4-rotation prediction.

Augmentation vs. Regularization: to test whether incorporating the representation learning (rotation) distracts the model from learning noisy labels, we trained the regularization model which performs obviously better then the augmentation model. The validation accuracy of the best regularization model (after parameter tuning) is increased by about 3 percents. This observation is consistent with our hypothesis that adding a regularization term weighted by the samples’ probabilities of being clean can distract the model from memorizing noisy-labelled samples in an early stage.

Regularization vs. Regularization with warm-up: warm-up in the first 20 epoch further enhances the regularization model, and this model achieved the best validation accuracy of $71.18\%$ with 4-rotation prediction and $69.34\%$ with 1-image prediction. It makes sense intuitively because in the warm-up process, the model fits the data with clean labels rapidly without any potential distractions from the rotation recognition task, and therefore generates a plausible loss distribution to form the initial probability $\lambda$ at epoch 21 when we incorporate the representation learning.

Augmentation vs. Separation: besides the augmentation model, we also train a separation model. This model is more conservative than the regularization model in the sense that the model will learn nothing from the labels predicted as noisy. Consequently, it fits less incorrect information, so its learning curve on the validation set shows no decline in at least 300 epochs. However, the validation accuracy of the separation model is lower than that of the regularization model (though still higher than that of baseline model), due to the loss of correct information when the separation model recognizes clean labels as noisy.

We attempted to combine the two approaches in 3.1 and 3.2 to achieve a better accuracy resulted from the more accurate noisy-labels detection and the self-supervised learning technique. As Figure 2.b shows, using the Loss-and-Confidence Noisy Detection with Elastic Decision instead of Loss-only Noisy Detection we used in the subsection 4.2, the validation accuracy of both regularization model and separation model has increased. The improvement is especially significant for the separation model more sensitive to the quality of noisy labels detection. Table 1 shows the evaluation of baseline model, regularization model and separation model using different noisy labels detection methods, and finally the best test accuracy is improved to over $69\%$ from $63.04\%$ (the baseline model).