Consistency Regularization with Generative Adversarial Networks for Semi-Supervised Learning

In the past decade, supervised classification performance improved significantly with the advent of deep neural networks (Simonyan and Zisserman 2014; He et al. 2016; Huang et al. 2017). These advancements can be chiefly attributed to the training of deep neural networks on large-scale well-annotated image classification datasets, such as, ImageNet (Deng et al. 2009). However, obtaining such datasets with large amounts of labeled data is often prohibitive due to time, cost, expertise, and privacy restrictions. Semi-supervised learning (SSL) presents an alternative, where models can learn representations from plentiful of unlabeled data, thus reducing the heavy dependence on the availability of large labeled datasets.

In recent years, Deep Generative Models (DGMs) (Kingma and Welling 2013; Goodfellow et al. 2014) have emerged as an advanced framework for learning data representations in an unsupervised manner. In particular, Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) have demonstrated an ability to learn generative model of any arbitrary data distribution and produce visually realistic set of artificial (fake) images. GANs set up an adversarial game between a generator network and a discriminator network, where the generator is tasked to trick the discriminator with generated samples, whereas the discriminator is tasked to tell apart real and generated samples. Semi-GAN (Salimans et al. 2016) is one of the earlier extension of GANs to the SSL domain, where the discriminator employs a (K+1)-class predictor with the extra class referring to the fake samples from the generator.

We first observe that semi-GAN suffers from inconsistent predictions in our experiments on the CIFAR-10 dataset. In this experiment, each unlabeled image is augmented with two different data augmentations and fed into the well-trained discriminator of a semi-GAN. Figure 1 depicts such input images on which the semi-GAN’s discriminator produces inconsistent predictions, whereas our proposed composite consistency based semi-GAN produces desired results. Although many approaches (Dai et al. 2017; Qi et al. 2018; Dumoulin et al. 2016; Lecouat et al. 2018) have been developed to improve the performance of semi-GAN, regularizing semi-GAN with consistency techniques has barely been explored in the literature. Consistency regularization specifies that the classifier should always make consistent predictions for an unlabeled data sample, in particular, under semantic-preserving perturbations. It follows from the popular smoothness assumption (Chapelle, Scholkopf, and Zien 2009) in SSL that if two points in a high-density region of data manifold are close, then so should the corresponding outputs. Based on this intuition, we hypothesize that the discriminator of semi-GAN should also produce consistent outputs on perturbed versions of the same image.

Thus, in this work we propose to extend semi-GAN by integrating consistency regularization into the discriminator. Previous works on consistency regularization focus on either local consistency that regularizes the classifier to be resilient to local perturbations added to data samples, or interpolation consistency that regularizes the classifier to produce consistent predictions at interpolations of data samples. In this work, we propose a new composite consistency regularization by exploring both of them. In summary, we make the following contributions:

• •

  We propose a new consistency measure called composite consistency, which combines both local consistency and interpolation consistency. We experimentally show that this composite consistency with semi-GAN produces best results among the three consistency-based techniques.

• •

  We propose an integration of consistency regularization into the discriminator of semi-GAN, encouraging it to make consistent predictions for data under perturbations, thus leading to improved semi-supervised classification. Experimentally, our semi-GAN with composite consistency sets new state-of-the-art performances on the two SSL benchmark datasets SVHN and CIFAR-10, with error rates reduced by 2.87% and 3.13% respectively while using the least amount of labeled data.

In a general SSL setting, we are given a small set of labeled samples $(\mathbf{x}_{l},y_{l})$ and a large set of unlabeled samples $\mathbf{x}_{u}$, where every $\mathbf{x}\in\mathbb{R}^{d}$ is a $d$-dimensional input data sample and $y\in\{1,2,...,K\}$ is one of $K$ class labels. The objective of SSL is to learn a classifier $D(y|\mathbf{x};\theta):\mathcal{X}\rightarrow\mathcal{Y}$, mapping from the input space $\mathcal{X}$ to the label space $\mathcal{Y}$, parameterized by $\theta$. In deep SSL approaches, $D(y|\mathbf{x};\theta)$ is chosen to be represented by a deep neural network.

To address the prediction inconsistency of semi-GAN (Salimans et al. 2016), we integrate consistency regularization into semi-GAN, leading it to produce consistent outputs (predictions) under small perturbations. More specifically, we incorporate consistency regularization as an additional auxiliary loss term to the discriminator, as shown in Eq.4.

$$\begin{split}\mathcal{L}_{D}&=-\mathbb{E}_{p(\mathbf{x}_{l},y_{l})}[\text{log }D(y_{l}|\mathbf{x}_{l};\theta,\xi)]\\ &\quad-\mathbb{E}_{p(\mathbf{z})}[\text{log }D(y=K+1|G(\mathbf{z};\delta);\theta)]\\ &\quad-\mathbb{E}_{p(\mathbf{x})}[\text{log }(1-D(y=K+1|\mathbf{x};\theta,\xi))]\\ &\quad+\lambda_{cons}\mathbb{E}_{p(\mathbf{x})}d[D(y|\mathbf{x};\theta,\xi),D(y|\mathbf{x};\theta^{\prime},\xi^{\prime})]\end{split}$$

(4)

where the first three terms come from original discriminator loss of semi-GAN (see Eq.1) and the fourth term is the consistency loss (see Eq.3), and the coefficient $\lambda_{cons}$ is a hyper-parameter controlling the importance of the consistency loss. Figure 2 displays our new model architecture. As shown in the figure, the discriminator $D(y|\mathbf{x};\theta)$ in semi-GAN (Salimans et al. 2016) is also treated as the student model for the consistency regularization and the consistency loss is enforced as the prediction difference between the student and teacher models for real data.

Two types of consistency regularization methods have been developed in recent years. One is the local consistency, where it encourages the classifier to be resilient to local perturbations added to data samples. Local perturbations are usually represented in the form of input augmentations (Laine and Aila 2016; Tarvainen and Valpola 2017) or adversarial noise (Miyato et al. 2018). In this work, we explore the integration of local consistency to semi-GAN and choose the consistency method Mean Teacher (MT) (Tarvainen and Valpola 2017), as our consistency regularization.

MT imposes consistency by adding random perturbations to the input of the model. As shown in Figure 3 (a), the input data are transformed with certain types of augmentation (e.g., image shifting, flipping, etc.) randomly twice. The two augmented inputs are then fed into the student model and teacher model separately, and the consistency is achieved by minimizing the prediction difference between the student model and teacher model. One key aspect of MT is that it improves the quality of the learning targets from the teacher model by forming a better teacher model. Namely, the parameters $\theta^{\prime}$ of the teacher model are maintained as an exponential moving average (EMA) of the parameters $\theta$ of the student model during training, formulated as:

$$\theta_{t}^{\prime\prime}=k\theta^{\prime}_{t-1}+(1-k)\theta_{t}^{\prime}$$

(5)

where $t$ indexes the training step and the hyper-parameter $k$ is the EMA decay coefficient. By aggregating information from the student model in an EMA manner at training time, a better teacher model can generate more stable predictions which serve as higher quality learning targets to guide the learning of the student model.

Then we explore the integration of the other type of consistency regularization, the interpolation consistency, where it encourages consistent predictions at interpolations of two data samples. It was first proposed by Interpolation Consistency Training (ICT) (Verma et al. 2019). In ICT, the interpolation is implemented using the MixUp operation (Zhang et al. 2018). Given any two vectors $u$ and $v$, we can define the MixUp operation as

$${Mix}_{\lambda}(u,v)=\lambda\cdot u+(1-\lambda)\cdot v$$

(6)

where $\lambda\in[0,1]$ is a parameter randomly sampled from Beta distribution denoted as $\lambda\sim\text{Beta}(\alpha,\alpha)$, and $\alpha$ is a hyper-parameter controlling the sampling process. With the MixUp operation, given two randomly shuffled versions of the dataset $\mathbf{x}$ after data augmentation $\xi$ represented as $\mathbf{x}_{m}$ and $\mathbf{x}_{n}$, the ICT consistency is computed as

$$\begin{split}\mathcal{L}_{ict\_cons}&=\mathbb{E}_{p(\mathbf{x}_{m},\mathbf{x}_{n}|\mathbf{x},\xi)}d[D(y_{mix}|{Mix}_{\lambda}(\mathbf{x}_{m},\mathbf{x}_{n});\theta),\\ &\quad\quad\quad\quad{Mix}_{\lambda}(D(y_{m}|\mathbf{x}_{m};\theta^{\prime}),D(y_{n}|\mathbf{x}_{n};\theta^{\prime}))]\end{split}$$

(7)

and it encourages the predictions from the student at interpolations of any two data samples (denoted as $D(y_{mix}|{Mix}_{\lambda}(\mathbf{x}_{m},\mathbf{x}_{n});\theta)$) to be consistent with the interpolations of the predictions from the teacher on the two samples (denoted as ${Mix}_{\lambda}(D(y_{m}|\mathbf{x}_{m};\theta^{\prime}),D(y_{n}|\mathbf{x}_{n};\theta^{\prime}))$), shown in Figure 3 (b).

Having already discussed consistency-based approaches, we only focus on reviewing the most relevant GAN-based SSL approaches and other categories of deep SSL approaches.

GAN-based SSL approaches: Following semi-GAN (Salimans et al. 2016), Qi et al. (2018) propose Local-GAN to improve the robustness of the discriminator on locally noisy samples, which are generated by a local generator at the neighborhood of real samples on a real data manifold. Instead, our approach attempts to improve the robustness of the discriminator from the perspective of consistency directly on real samples. Likewise, Dai et al. (2017) have proposed a complement generator. They show both theoretically and empirically that a preferred generator should generate complementary samples in low-density regions of the feature space, so that real samples are pushed to separable high-density regions and hence the discriminator can learn to correct class decision boundaries. Based on information theory principles, CatGAN (Springenberg 2016) adapts the real/fake adversary formulation of the standard GAN to the adversary on the level of confidence in class predictions, where the discriminator is encouraged to predict real samples into one of the $K$ classes with high confidence and to predict fake samples into all of the $K$ classes with low confidence, and the generator is designated to perform in the opposite. Similarly, the CLS-GAN (Qi 2019) designs a new loss function for the discriminator with the assumption that the prediction error of real samples should always be smaller than that of fake ones by a desired margin, and further regularizes this loss with Lipschitz continuity on the density of real samples. Apart from them, Chongxuan et al. (2017) design a Triple-GAN consisting of three networks, including a discriminator, a classifier, and a generator. Here, the discriminator is responsible for distinguishing real image-label pairs from fake ones, which are generated by either the classifier or the generator using conditional generation. Most of these methods attempt to improve the classification performance from the perspective of better separating real/fake samples, whereas our approach validates that improving the ability of the discriminator in itself with consistency is critical.

Other deep SSL categories: Variational Auto-Encoders (VAEs) have also been explored in the deep generative models (DGMs) domain. VAE-based SSL approaches (Kingma et al. 2014; Rezende and Mohamed 2015) treat class label as an additional latent variable and learn data distribution by optimizing the lower bound of data likelihood using a stochastic variational inference mechanism. Aside from DGMs, graph-based approaches (Atwood and Towsley 2016; Kipf and Welling 2017) have also been developed with deep neural networks, which smooth the label information on a pre-constructed similarity graph using variants of label propagation mechanisms (Bengio, Delalleau, and Le Roux 2006). Differing from graph-based approaches, deep clustering approaches (Haeusser, Mordvintsev, and Cremers 2017; Kamnitsas et al. 2018) build the graph directly in feature space instead of obtaining a pre-constructed graph from input space and perform clustering on the graph guided by partial labeled information. Furthermore, some recent advances (Wang, Li, and Gool 2019; Berthelot et al. 2019) focus on the idea of distribution alignment, attempting to reduce the empirical distribution mismatch between labeled and unlabeled data caused by sampling bias.

In this work, we identified an important limitation of semi-GAN and extended it via consistency regularizer. In particular, we developed a simple but effective composite consistency regularizer and integrated it with the semi-GAN approach. This composite consistency measure is resilient to both local perturbations and interpolation perturbations. Our thorough experiments and ablation studies showed the effectiveness of semi-GAN with composite consistency on two benchmark datasets of SVHN and CIFAR-10, and consistently produced lower error rates among the GAN-based SSL approaches.

Since composite consistency with semi-GAN is proved to be effective on real images, we plan to study the effect of enforcing composite consistency also on generated images from the generator in our future work. Though we adopted standard data augmentations to the input images in this work, we are interested in further exploration of other stronger forms of recent data augmentations (i.e., AutoAugment (Cubuk et al. 2019), RandAugment (Cubuk et al. 2020)).