Semi-Supervised Learning with Multi-Head Co-Training

Benefiting from the rich data sources and growing computing power in the last decade, the field of machine learning has been thriving drastically. The advent of public datasets with a large amount of high-quality labels has further spawned many successful deep learning methods (Deng et al. 2009; He et al. 2016; Zagoruyko and Komodakis 2016; Krizhevsky, Sutskever, and Hinton 2012). However, there could be various difficulties for obtaining label information, such as privacy, labor costs, safety or ethic issues, and requirement of domain experts (Zhou 2018; Chapelle, Schlkopf, and Zien 2010; Mahajan et al. 2018). All of these impel us to find a way of bringing unlabeled data into full play. Semi-Supervised Learning (SSL) is a branch of machine learning which seeks to address the problem (Chapelle, Schlkopf, and Zien 2010; Chapelle, Chi, and Zien 2006; Prakash and Nithya 2014; Van Engelen and Hoos 2020). It utilizes both labeled and unlabeled data to improve performance.

As one of the earliest and most popular SSL frameworks, self-training works by iteratively retraining the model using pseudo-labels obtained from itself (Lee 2013; Berthelot et al. 2019b, a; McLachlan 1975). Despite its simplicity and alignment to the task of interest (Zoph et al. 2020), self-training underperforms due to the “confirmation bias” or “error accumulation”. It means that some incorrect predictions could be selected as pseudo-labels to guide subsequent training, resulting in a loop of self-reinforcing errors (Zhang et al. 2016).

As an extension of self-training, co-training lets multiple individual models iteratively learn from each other (Zhou and Li 2010; Wang and Zhou 2017). In the early multi-view co-training setting (Blum and Mitchell 1998), there should be a natural split of features where the “sufficiency and independence” assumptions should hold, i.e., it’s sufficient to make predictions based on each view, and views are conditionally independent. Later studies gradually reveal that co-training can also be successful in the single-view setting (Wang and Zhou 2017; Dasgupta, Littman, and McAllester 2002; Abney 2002; Balcan, Blum, and Yang 2005; Wang and Zhou 2007). Despite being feasible, the single-view co-training framework has received little attention recently. We attribute it to (a) the extra computational cost, which means at least twice model parameters of its self-training counterpart, and (b) the loss in simplicity, i.e., more design choices and hype-parameters are introduced for keeping individual classifiers uncorrelated.

In this paper, we aim to facilitate the adoption of single-view co-training. Inspired by recent developments of data augmentation and its applications in SSL (Berthelot et al. 2019b; Sohn et al. 2020; Cubuk et al. 2020; DeVries and Taylor 2017), we find that the enormous size of the augmentation search space naturally prevents base learners from converging to a consensus. Employing the stochastic image augmentation frees us from delicately design different network structures or training algorithms. Moreover, by replacing multiple individual models with a shared module followed by multiple classification heads, the model can achieve co-training in a minimal amount of extra parameters. Combining these, we propose Multi-Head Co-Training, a new algorithm that facilitates the usage of single-view co-training. The main contributions are as follows:

• •

  Multi-Head Co-Training addresses two obstacle factors of standard single-view co-training, i.e., extra design and computational cost.

• •

  Experimentally, we show that our method obtains state-of-the-art results on CIFAR, SVHN, and Mini-ImageNet. Besides, We systematically study the components of Multi-Head Co-Training.

• •

  We further analyze the calibration of SSL methods and provide insights regarding the link between confirmation bias and model calibration.

In this section, we concentrate on relevant studies to set the stage for Multi-Head Co-Training. More extensive surveys on SSL can be found in (Prakash and Nithya 2014; Van Engelen and Hoos 2020; Zhu 2005; Zhou and Li 2010; Zhou 2018; Subramanya and Talukdar 2014).

The basic assumptions in SSL are the smoothness assumption and low-density assumption. The smoothness assumption states that if two or more data points are close in the sample space, they should belong to the same class. Similarly, the low-density assumption states that the decision boundary for a classification model shouldn’t pass the high-density region of sample space. These assumptions are intuitive in vision tasks because an image with small noise is still semantically identical to the original one. A dominant paradigm in SSL is grounded on these assumptions. From this point of view, various ways of making use of unlabeled data, including consistency regularization, entropy minimization, perturbation-based methods, self-training, and co-training, are essentially similar.

Consistency regularization constrains the model to make consistent predictions across the same example under variants of noises. In a general form,

where $q$ and $p$ are the modeled distributions. Different notations are used here, indicating that they could come from different models. The target example is denoted as $x$ and its noisy counterpart is denoted as $x^{\prime}$. $D(\cdot,\cdot)$ can be any distance measurement, such as KL divergence or mean square error. SSL methods falling into this category differ in the source of noise, models for two distributions, and the distance measurement. For example, VAT (Miyato et al. 2018) generates noise in an adversarial direction. Laine & Aila (Laine and Aila 2016) propose $\Pi$-Model and Temporal Ensembling. $\Pi$-Model performs Gaussian noise, dropout, etc., to augment images. Temporal Ensembling further ensembles prior network evaluations to encourage consistent predictions. Mean Teacher (Tarvainen and Valpola 2017) instead maintains an Exponential Moving Average (EMA) of model’s parameters. ICT (Verma et al. 2019) applies consistency regularization between the prediction of interpolation of unlabeled points and the interpolation of the predictions at those points. UDA (Xie et al. 2019) replaces the traditional data augmentation with unsupervised data augmentation.

Self-training¹¹1In the field of SSL, the terminology “self-training” overlaps with “pseudo-labeling”, which refers to training the model incrementally instead of retraining in every iteration. For illustration, we use “self-training” throughout this paper, referring to a broad category of methods. favors low-density separation by using model’s own predictions as pseudo-labels. Pseudo-Labeling (Lee 2013) picks the most confident predictions as hard (one-hot) pseudo-labels. Apart from that, the method also uses auto-encoder and dropout as regularization. MixMatch (Berthelot et al. 2019b) uses the average of predictions on the image under multiple augmentations as the soft pseudo-label. Furthermore, Mixup (Zhang et al. 2017), as a regularizer, is used to mix labeled and unlabeled data. ReMixMatch (Berthelot et al. 2019a) improves MixMatch by introducing other regularization techniques and a modified version of AutoAugment (Cubuk et al. 2019). FixMatch (Sohn et al. 2020) finds an effective combination of image augmentation techniques and pseudo-labeling. One of the reasons for the popularity of self-training is its simplicity. It can be used with almost all supervised classifier (Van Engelen and Hoos 2020). Another important but rarely mentioned factor is its awareness of the task of interest. Although some other unsupervised constraints may help build general representations, it has been shown that self-training can align to the task of interest well and benefit model in a more secure way (Zoph et al. 2020). However, confirmation bias, where the prediction mistakes would accumulate during the training process, damages the performance of self-training.

As an extension of self-training, co-training alleviates the problem of confirmation bias. Two or more models are trained by each other’s predictions. In the original form (Blum and Mitchell 1998), two individual classifiers are trained on two views. What’s more, the proposed co-training algorithm requires the “sufficiency and independence” assumptions hold, i.e., two views should be sufficient to perform accurate prediction and be conditionally independent given the class. Nevertheless, later studies show that weak independence (Abney 2002; Balcan, Blum, and Yang 2005) or single-view data (Wang and Zhou 2017, 2007; Du, Ling, and Zhou 2010) is enough for a successful co-training algorithm.

Without the distinct views of the data containing complementary information, single-view co-training has to promote diversity in some other ways. It also has been shown that the more uncorrelated the individual classifiers are, the more effective the algorithm is. (Wang and Zhou 2017, 2007) Several early studies attempt to split the single-view data (Balcan, Blum, and Yang 2005; Chen, Weinberger, and Chen 2011), and some further approach a pure single-view setting by introducing diversity among the classifiers otherwise (Zhou and Goldman 2004; Goldman and Zhou 2000; Xu, He, and Man 2012). Recently, Deep Co-training (Qiao et al. 2018) maintains disagreement through a view difference constraint. Tri-Net (Dong-DongChen et al. 2018) adopts a multi-head structure. To prevent consensus, it designs different head structures and samples different sub-datasets for learners. CoMatch (Li, Xiong, and Hoi 2020) applies a graph-based smoothness regularization and also integrates supervised contrastive learning. Although have achieved a number of successes, these methods complicate the practical adoption of co-training. In Multi-Head Co-Training, employing the stochastic image augmentation frees us from delicately designing different network structures or training algorithms. Unique to other methods, no preventive measures, e.g., extra loss term or different base learners, are needed to avoid collapse. Along with the reduction of computational cost brought by the multi-head structure, the proposed method facilitates the adoption of single-view co-training.

In this section, we introduce the details of Multi-Head Co-Training. Formally, for a $C$-class classification problem, SSL aims to model a class distribution $p(y\mid x)$ for input $x$ utilizing both labeled and unlabeled data. In Multi-Head Co-Training, the parametric model consists of a shared module $f$ and multiple classification heads $g_{m}$ ($m\in\{1,\dots,M\}$) with the same structure. Let $p_{m}(y\mid x)=g_{m}(f(x))$ represent the predicted class distribution produced by $f$ and $g_{m}$. All classification heads are updated using the using a consensus of predictions from other heads. Apart from that, following the principle of recent successful SSL algorithms, this method utilizes image augmentation techniques to employ a “Weak and Strong Augmentation” strategy. It uses predictions on weakly augmented images, which are relatively more accurate, to correct the predictions on strongly augmented images. The weak and strong augmentation function are denoted as $Aug_{w}(\cdot)$ and $Aug_{s}(\cdot)$ respectively and further introduced in The “Weak and Strong Augmentation” strategy. The diagram of Multi-Head Co-Training with three heads is shown in Figure 1. The overall algorithm is shown in Algorithm 1.

The number and structure of heads in our framework can be arbitrary, but we set the head number as three and the head structure as the last residual block (Zagoruyko and Komodakis 2016) in most of the experiments. The choice is the result of a trade-off between accuracy and efficiency (illustrated in Ablation study).

Confirmation bias comes up frequently in SSL literature. However, it is hard to formulate or observe the problem. We notice that most self-training or co-training methods select pseudo-labels by some criteria, such as confidence threshold. In these cases, confirmation bias is closely related to the over-confidence of the network: Wrong predictions with high confidence are likely to be selected and then used as pseudo-labels. Thus, we link confirmation bias to model calibration, i.e., the problem of predicted probability represents the true correctness likelihood. We envision the calibration measurements be used to evaluate confirmation bias and help the design of self-training and co-training algorithms. Apart from this, the challenges of SSL and calibration could appear simultaneously in real-world. For example, in one of the applications of SSL, medical diagnosis, control should be passed on to human experts when the confidence of automatic diagnosis is low. In such scenarios, a well-calibrated SSL model is needed. To the best of our knowledge, these two problems have hitherto been studied independently.

According to our observation, SSL models have poor performance in terms of calibration due to entropy minimization and other similar constraints. We analyze FixMatch (implemented using the same code-base), as one of the typical SSL methods, and Multi-Head Co-Training on CIFAR-100 with 10000 labels from the perspective of model calibration. Several common calibration indicators are used, namely Expected Calibration Error (ECE), confidence histogram, and reliability diagram (illustrated in the supplementary material). As shown in Figure 3a, FixMatch has an average confidence of 78.75% but only 74.93% accuracy, producing over-confident results with an ECE value of 15.61. In Figure 3b, we show the results of Multi-Head Co-Training. Although our method produces over-confident predictions, the ECE value is smaller, indicating better probability estimation.

To further investigate, we apply a simple calibration technique called “temperature scaling” (Guo et al. 2017) (see the supplementary material). From the calibrated results in Figure 3c and Figure 3d, it can be observed that the miscalibration is remedied. The ECE value of calibrated FixMatch is improved to 5.27% Multi-Head Co-Training’s reliability diagrams closely recovers the desired diagonal function with a low ECE value of 2.82. It can be concluded that Multi-Head Co-Training produces good probability estimates naturally and can be better calibrated with simple techniques. Therefore, we suggest that confirmation bias is better addressed in our method from the perspective of calibration.

The field of SSL encompasses a broad spectrum of algorithms. However, co-training framework has received little attention recently because of the diversity criterion, extra computational cost. Multi-Head Co-Training pointedly addresses these. It achieves single-view co-training by integrating the individual models into one multi-head structure and utilizing the data augmentation techniques. As a result, the proposed method is 1). in a minimal amount of both parameters and hype-parameters and 2). doesn’t need extra effort to promote diversity. We present systematic experiments to show that Multi-Head Co-Training is a successful co-training method and outperforms state-of-the-art methods. The solid empirical results suggest that it is possible to scale co-training to more realistic SSL settings. In future work, we are interested in combining modality-agnostic data augmentation to make Multi-Head Co-Training ready to be applied to other tasks.

This paper is supported by the National Key Research and Development Program of China (Grant No. 2018YFB1403400), the National Natural Science Foundation of China (Grant No. 61876080), the Key Research and Development Program of Jiangsu(Grant No. BE2019105), the Collaborative Innovation Center of Novel Software Technology and Industrialization at Nanjing University.