On Positive-Unlabeled Classification in GAN

GAN [4] was introduced by Goodfellow et al. (2014). It consists of two neural networks: discriminator network $D$ and generator network $G$. The discriminator $D$ aims to distinguish the provided real data and the fake data generated by the generator $G$. On the other hand, the generator $G$ aims to generate fake data that can fool the discriminator $D$. Following this adversarial manner, we expect the $G$ can generate high-quality data in the end. Formally, the objective function of GAN can be written as

where $p_{data}$ indicates the distribution of real data, $\bm{z}$ is the random noise sampled from a prior distribution $p_{\bm{z}}$ (i.e. the Gaussian distribution), and $D(\bm{x})$ is the predicted probability of $\bm{x}$ to be real by the discriminator. Since the minimax objective function might lead to gradient vanishing for $G$ when $D$ can perfectly distinguish two data set. More of GAN’s variants (e.g. WGAN [1] and LSGAN [15]) transform this minimax game into a non-saturating game. In general, the objective functions of these GANs can be concluded as follows:

where $f_{1}$ is the loss of classifying input as real and $f_{2}$ is the loss of classifying input as fake.

As shown above, existing GAN variants are trained to separate the real and generated data strictly. However, this does not match the actual situation in training. Some of the generated samples can achieve higher quality and are more realistic than others. This phenomenon usually lasts until the end of the training. As a result, the quality of samples generated by $G$ is very different, and there are many high-quality samples and a considerable proportion of low-quality samples. For instance, as a well-known problem in GAN, the model collapse problem that $G$ networks often suffer can be considered as the generation space consisting of some high quality and non-repetitive samples and the rest of the repetitive samples. These duplicate samples can be considered as low-quality samples and still need to be improved. There is still a phenomenon that there is a certain proportion of unsatisfactory samples in a well-trained generation space, and the gaps in the generated samples of different quality are relatively large. Therefore, the traditional method of strictly distinguishing the real sample from the generated sample does not conform to the actual situation of the training. In this paper, we propose an algorithm that is dedicated to picking out low-quality samples from the samples generated by $G$ and promoting them, unlike traditional discriminators $D$ that are dedicated to distinguishing real samples from generated samples. Our algorithm encourages the discriminator $D$ to divide the generation space of $G$ into high-quality samples and low-quality samples so that the generator $G$ could improve the low-quality samples.

By doing so, the proposed method enjoys several desired properties: i) The discriminator pays more attention to poor quality samples, allowing the generator to focus on improving the quality of these bad samples. As a result, the quality of generated samples is more balanced, and the overall quality is expected to be enhanced, ii) The training strategy of our algorithm is more in line with the actual situation of samples generated by $G$ in the training process, so the training process is expected to be more stable, iii) More importantly, the proposed algorithm is a flexible method which means that our algorithm could be easily integrated into existing frameworks of variety GAN, and we will show this desirable feature in the following subsection, and iv) Although the proposed method changes the function of the D network, we provide some theoretical results in Section 3 which demonstrate the proposed method also enjoys the same equilibrium condition with the other GAN and provide some guarantee for performance.

Above, we discuss the current problem existing in the GAN framework and propose to allow some good samples to be recognized as real data. In this part, we firstly introduce how we achieve this in the framework of standard GAN, and then we extend it for general GANs.

Above we conclude our objective function within the standard GAN framework. The proposed method can also be integrated into other general GAN frameworks flexibly. In this part, we combine the proposed method with other loss functions of discriminator in GAN.

In general, the objective function Eq. (2) contains two loss functions $f_{1}$ and $f_{2}$. Those concrete loss functions can be changed for a different variance of GANs, but all these loss functions are following the same concepts that $f_{1}(\bm{x})$ and $f_{2}(\bm{x})$ are trying to separate the real data from the generated data as far as possible. Similar to SGAN, we propose that the discriminator in GAN is better to focus on generated samples with low quality and recognize the high-quality samples from the generated samples. Following this concept, we implement the proposed method for the general framework of GAN with the following equation:

With the help of Eq. (11), we can now integrate the proposed method into various frameworks of GAN, such as WGAN-GP [20], LSGAN [15], and SpectualGAN [17]. This flexibility that combining with other models provides the proposed method a chance to get further improvement on existing excellent models. Loss functions corresponding to the specific model can be found in the supplementary material.

In this section, we evaluate the generation ability of the proposed method on multi-category image sets MNIST [14], FMNIST [22], and CIFAR-10 [13] and single-category image sets CAT [25] and LSUN-bedroom [24]. In this part, the resolution of images in the MNIST, FMNIST, and CIFAR-10 is 32 and is 64 in the rest datasets. We choose four representative adversarial models SGAN, HingeGAN, LSGAN, and WGAN-GP as basic frameworks and compare them with our algorithms, which are denoted as PUSGAN, PUHingeGAN, PULSGAN, and PUWGAN-GP, respectively.

Table 1 reports a comparison of the FID score obtained by the proposed method and basic models. Our models enjoy the ability to combine with most variants of GAN, which allows us to achieve the best performance on most data sets. Table 1 shows that most of our methods exceed their corresponding basic frameworks, which demonstrates the effectiveness and the flexibility of our approach. In Figure 1, we show a few images generated by the PUSGAN models. We observe that the proposed method generates high-quality images on various datasets, which is consistent with the quantitative results in Table 1.

We evaluate the stability of the proposed method on three resolution of the CAT dataset, such as 64 $\times$ 64, 128 $\times$ 128, and 256 $\times$ 256 pixels. As there are only 6654 and 2011 samples in the CAT-128 dataset and CAT-256 dataset respectively, some variants of GAN are unable to converge on these datasets. We choose SGAN, LSGAN, and WGAN-GP as basic models. We compare the proposed method with both these basic models as well as the corresponding Relativistic GAN (RaGAN). For each model, We calculate the FID score of the current model every 10,000 iterations. The results will be presented with the minimum, maximum, mean, and standard deviation (SD) of these obtained FID values. Table 2 shows the FID results for different networks in different resolutions of data sets.

For 64$\times$64 resolution dataset, all models trained by the proposed method except PULSGAN, can achieve much lower FID in minimum, maximum and mean compared with its original version and even can further achieve lower FID values than their relativistic versions, which indicates that our algorithm can effectively improve the training stability and improve the quality of the generation.

For higher resolution data sets, SGAN failed to converge in 128x128 and 256x256 resolution datasets while LSGAN will be stuck at the early stage in 256x256 resolution dataset [8]. The standard version of our model (PUSGAN) shows further stability with lower values of maximum, mean, and SD in all three resolution datasets. On the most challenging 256 resolution dataset, the proposed method achieves both the satisfactory quality and stability. In experiments, we found that although the PULSGAN can converge in the CAT-256 dataset, the convergence is much slower than other GANs. Nevertheless, PULSGAN still can achieve competitive results compared with other states of arts GANs such as WGAN-GP and RaGAN.

Overall, our algorithm presents desirable stability for all three data sets, and it can achieve similar or even better results compare to relativistic versions. It is impressive that all these GANs trained by the proposed method have improved. As a result, we conclude that the above stability experiment demonstrates that the proposed method provides could provide stability for the training progress for a variety of GANs and thus improve the quality of generated images.

As we have claimed, our approach focuses on improving low-quality samples and lead to more stable training progress. Thus our approach enjoys the ability to generalize to many training settings. To demonstrate this, we evaluate the proposed method on several hard training settings and compare it with the other GAN frameworks. In this part, we implement SGAN (RaSGAN, PUSGAN), LSGAN (RaLSGAN, PULSGAN), and HingeGAN (RaHingeGAN, PUHingeGAN) on the CIFAR-10 dataset. The experiment is conducted on one basic setting and three hard settings. The basic setting is same as above, and three hard settings are i) changing learning rate to $0.001$ (lr=.001), ii) removing Batch normalization layers in $G$ and $D$ (No BN), and iii) replacing all activation functions with Tanh in $G$ and $D$ (Tanh).

The results are showed in Table 3. In the stable setting, we can find that the PUSGAN has better performance than SGAN and its other variants. The PUHingeGAN has a huge improvement compared with the original HingeGAN with a gap of 14, and it also performs better than RaHingeGAN. On the other hands, the performance of PULSGAN is slightly worse than other LSGAN versions.

When the learning rate is increased to 0.001, all three PU versions of GANs perform well, compared with the original one. While PUSGAN and PUHingeGAN can perform better than their relativistic versions. However, by changing optimization settings such as removing batch normalization or replace ReLU activation function with Tanh activation function (No BN and Tanh in columns respectively), the performances of PUGANs will be worse. It might indicate that the PUGANs rely on optimization terms for stable training.

Generating high-resolution images is a complicated task. To demonstrate the generation ability of our algorithm, we evaluate the proposed method on CAT-128, CelebA-128, and LSUN-128 datasets with 128 $\times$ 128 pixels and the CAT-256 dataset with 256 $\times$ 256 pixels. There are 202,599 images in the CelebA-128 dataset, and 3,033,042 in the LSUN-128 dataset (only 100,000 samples are used for training). As mentioned above, the CAT-128 and CAT-256 datasets are considered as more challenging high-resolution datasets because there are only 6,645 and 2,011 samples, respectively. Images shown in Figure 2 are generated by the proposed method within the architecture of SGAN (PUSGAN), while SGAN failed to generate such high-resolution images, especially on the CAT-256 dataset. Moreover, interpolation is also an impressive feature of the generative models, which indicates that the generative model successfully learns to fit the distribution of natural images instead of overfitting to the training samples. We show high-resolution interpolation results obtained by the proposed method in Figure 3. It shows that our model generates smooth interpolation images. Figures 2 and  3 demonstrate that the proposed method could provide improvement on the quality of generated samples.

In Eq. (3), we introduce a class prior $\pi$ into our algorithm. The $\pi$ indicates the proportion of high-quality fake data in fake data, and we treat it as a hyper-parameter. In this section, we further evaluate the impact of class prior $\pi$ with PUSGAN framework on the CAT-64 dataset. The structure and training settings are the same as the previous sections. We set four different increasing patterns for $\pi$ during training. The first pattern is the basic pattern we used in previous sections. The second version will set $\pi$ to be 0.3 at the beginning, and it will be increased with 0.1 at every 10k iterations until 0.7 is reached. The second version is used to evaluate the impact of the fast growth of $\pi$. For the third and fourth patterns, $\pi$ will be fixed at 0.3 and 0.5 during the training process.

In Figure 4, we found that the fast-growing pattern achieves the worst average performance, and its FID scores remain relatively high. As a comparison, the basic pattern can reach lower FID values than the fast pattern, and it is also relatively stable in the later training stage. The fixed $\pi$ value of 0.3 could present more stability and generate a competitive result, compared with the previous two patterns. In addition, the fourth version has the worst performance at the beginning, but it was keep going better and achieved the best performance in the end, within all four patterns. The shortage of this version mainly lies in the large fluctuations and slow convergence. It is interesting that all first three patterns have similar FID values at the early stage, while the performance with a higher $\pi$ is much worse at the same stage. This may be because the proportion of high-quality samples in the $G$ network at the beginning of training is far from 0.5, and setting $\pi$ to 0.5 is against the real situation, leading to an unstable training. On the other hands, the fourth one achieves a lower FID value in the end, while others have similar values at the same time. It might show a too large (e.g. 0.7), or a too small (e.g. 0.3) values of $\pi$ reduce the performances. The result shows that the $\pi$ can affect the performance of the generation. We also present the result obtained by RaGAN for comparison. It shows that all the four versions of the model obtained by our algorithm produce both higher quality images and show better stability. As a result, the proposed method enjoys considerable tolerance for the selection of hyperparameters.

Normally, the size of real data is much smaller than the size of the generated data in the adversarial generative task. It is an interesting problem about how to make the most of this large amount of generated data. In general training progress of GAN, the batch size of real samples and generated samples are the same. Here, we try to increase the batch size of the generated data and maintain that of the actual data to take advantage of this large number of generated samples. We investigate the impact of increasing the batch size of the generated data for training. The evaluation is based on three versions of PUSGAN with different batch sizes of fake data. The first version is the basic version that the batch size of real and fake data are the same. The second and third versions will use twice and three times more fake data than real data, respectively. The structure and training settings are the same as the one we used in previous sections. We report these interesting results Figure 5.

From results, we find that the second and third versions of PUSGAN can reach their best performance at the very beginning, which proves that PUSGAN can converge faster by increasing the batch size of fake data. Although it shows that an increase in the number of false samples can provide a small performance boost and faster convergence, it seems to be detrimental to stability. Considering the stability and for a fair comparison, we insist on using the same batch size for both the real and generated data in the above experiments.