Mode Penalty Generative Adversarial Network with adapted Auto-encoder

Generative Adversarial Network (GAN) is motivated by game theory in which two players (generator and discriminator) compete with each other in a zero-sum game framework. The generator learns the target distribution via an adversarial training process with the discriminator. When the generator transforms a noise vector $z$ into a data vector $G(z)$ on the target space, the discriminator tries to estimate the probability if input sample comes from target distribution $P(x)$ or not. At the end of the training process, the generator produces outputs of $P(x)$. GAN defines this adversarial problem as follows:

The goal of GAN is to find the optimal parameter set of $G$ and $D$ in equation (1) minimizing the generator loss and maximizing the discriminator loss.

Assume that there is a set of optimal mode samples in manifold space of real data and also a set of samples from generator composing modes of generator’s manifold space. Our hypothesis is that forcing modes of generator to follow modes of real distribution encourages the minimization the distance between two manifolds and finally leads to help the learning of GANs. Real data distribution cannot be explicitly represented in original data space and we expect to transform real data to lower dimensional space for better understanding and easier manipulation. Auto-encoder is a neural network that learns discriminative representation of unlabeled input data in its learned encoded space. In other words, it learns compact and abstracted low dimensional representation of real data and is able to generate original input through decoding network. In the encoded space of auto-encoder, abstracted feature set represents aspects of original data distribution. Therefore, we expect that the encoded space provides more tractable way of defining mode and distance calculation between two manifold spaces for the training of our generative network.

There are various way for collecting target modes, for instance, sampling from the target distribution estimated using KDE, or selecting the samples from each class by defining heuristically. Since the target samples considered as modes will be used to evaluate the similarity of generated samples in the encoder space, they have to be guaranteed to be come from entire modes of real data distribution. After extracting $N$ target from real modes of target distribution and $N$ generated samples from generator, our network is able to compare real and generated manifold in the encoded space through adding new loss term to generator objective function. We construct distance loss between two manifolds through mode matching algorithm. Taking a MoG dataset as an example, the matching procedure is summarized in Figure 2. For every generated sample, we select the furthest generated sample from a centroid of real modes. Then, a real sample which has a closest distance from the selected one is chosen. Sample used in matching once are not reused. After making a generated-real sample pairs, we measure the distance the pair has for entire pair defined in equation (2) namely mode distance loss.

$X_{r}$ and $X_{g}$ denote a set of modes of real and generated distribution. $x_{r}$ and $x_{g}$ respectively denote paired real and generated mode in encoded space. And $w_{p}$ denotes a weight of penalty for missing modes which will be discussed in more detail in the next subsection. Intuitively, if the generator produces samples near only a few modes of target, then distance from missing modes are increasing. Penalty about this distance is given to the generator and it rescues the generator from bad situation when the gradient from the discriminator is almost zero. As a result, this process encourages the generator to follow manifold of real data. Next, we will discuss about

For finding complete modes of real data distribution, we focus more on missing modes than already covered mode. Model samples of real data obtained in the previous section do not have to maintain an equal importance in the distribution similarity measurement. Rather than some real samples which have been frequently recovered by the generator samples frequently, other real samples that are seldom recovered have to gain more importance in the following training iterations. Therefore, we expect to have a different focus on each samples to drive the generator in having diversity in its generated samples. For example, if our network observes that some generated samples are repeatedly close to a specific real sample, the real sample gets relieved of penalty weight. On the other hand, similarly generated samples that correspond with other distant real sample should be treated with an penalty weight and incur the generator training, which scatters the similarly generated samples to other neighbour modes. Giving heavier penalty for uncovered modes leads generator to find the uncovered mode much faster.

The basic idea of penalty weight is the quality evaluation of current real and generated samples and reflection of them in the loss calculation of training iteration. Real sample which have already been recovered by generated samples are assigned with lower penalty weight. Previously, we defined the mode distance as $Dist(X_{r},X_{g})$ which measures the distance between pairs of real and generated mode in the encoded space of auto-encoder. Now, we adjust the importance of each real samples $x_{r}$ using penalty weight.

Penalty weight $w_{p}$ in equation (3) indicates how well all generated samples contribute to the estimation of out model samples of the real data. We assign this weight for each real sample based on the degree of how much particular real sample is incarnated by generated samples in the past $k$ iterations. Calculated distance of a real sample from paired generated samples in the past iterations adds importance to the real sample. In other words, if a real sample has higher distances to all matched generated samples during the past $k$ training steps, that means this particular real sample is out of the generator interest and it gets higher penalty weight. As a result, in the following training steps, the generator is forced to consider this forgotten real sample more with higher loss value.

Finally, penalty weight for each real sample encourages generated samples to cover entire target real data samples. At each training step, this weight is updated to adjust the importance of generated and target samples. Due to the elaborate evaluation of the generated sample quality at each training step, proposed network converges to target real distribution faster without a mode collapse problem. For a set of real modes $X_{r}$ with $x_{r}$ as the element and a set of generator modes $G(Z)$ with $G(z)$, $z\sim N(0,1)$ as the element, we define the complete generator loss term in equation (4). If GAN unexpectedly generates samples from single mode of real data set, second term in equation (4) increases and force the network to find other modes. $\lambda_{p}$ is hyper-parameter adjusting the effect of penalty loss to an adversarial loss. After the generator covers almost all modes, it is turned off. By focusing on the GAN’s original purpose, therefore, generator produces much more diverse samples around each mode.

The overall training procedure of generator is summarized in Figure 3. First, we train the auto-encoder with target real data set. A set of target modes is represented in encoded space as $X_{r}$. Note that the real samples are extracted once and used for entire iterations of the training. After producing a set of modes $X_{r}$ by generator in same space, we make pairs between modes and each pair is given the mode penalty weight according to the distance.

For the analytic and controllable performance evaluation and visualization, 2 dimensional mixture of Gaussian (MoG) distribution data have been used for experimental comparison. We create four types of MoG distributions: 2D Ring (eight 2D Gaussians located in a ring), 2D Grid (25 2D Gaussians located in a grid), 2D Random (25 2D Gaussians of random amplitude and location), and 3D Cube (27 3D Gaussians in a cube). We create 20,000 real data samples from each MoG model and use for training and use 5000 generated sample for testing. Experiment is conducted with two-dimensional encoded space. We optimize both networks with the Adam optimizer with the learning rate of 1e-4, we assume number of 500 modes is enough to represent target distribution. We choose $\lambda_{p}$ is 3 which is experimentally giving better outcome. For fair comparison, we use identical network implemented in VEEGAN([13]) which is also used in other research. Generator has three layers of fully connected MLPs with 128 nodes without dropout and batch normalization, and discriminator has two layers of fully connected MLPs without dropout and batch normalization. We employ three metrics for quantitative evaluation: number of modes found, high quality sample ratio (HQS) [13], and additional distribution distance measurement Jensen-Shannon divergence (JSD). Number of Gaussian modes (8, 25, 25, and 27, respectively) found with generated samples and high quality sample ratio (HQS) are counted 20 times and mean and std. are calculated.

The number of modes found and percentage of high quality samples are not enough to tell how well generated samples are able to cover the entire real data distribution. Therefore, we measure the distance between real and generated distributions. We map the samples to canonical space and count the number of points fall in each canonical unit. By measuring Jensen-Shannon divergence (JSD) between the two distributions ($P_{r}$, $P_{g}$), we evaluate how well generator follows target real data (mixture of Gaussians) distribution. Figure 4 compares both convergence and mode detection performance on the four MoG types. Proposed method outperforms compared methodss in mode detection performance without mode collapse. For 2D Ring, proposed method already finds all modes within 5K iterations. In 2D random case, VEEGAN and Unrolled GAN fails to find modes of non-uniform locations and amplitudes, however our method finds almost identical distribution compared to target truth distribution. Proposed method also converges and finds all modes much faster than others. Figure 5 plots HQS and JSD of three methods at each training iteration in 2D Grid test. Proposed method shows outstanding convergence in both measurements. Recently, a new approach called PacGAN is proposed [8] that modifies the discriminator to make decisions based on multiple samples from the same class. This simple idea is tested on existing GANs showing outstanding performance in addressing mode collapse problem. We summarize quantitative evaluation in table 1 and our proposed method still finds slightly more modes on synthetic mixture of Gaussians data sets than all types of PacGANs even though it shows slightly lower HQS (Note that, however, std of proposed method is smaller than PacGAN).

Figure 6 plots HQS and JSD of 2D Grid test of our method at each training iteration with and without mode penalty weights $w_{p}$ and in equation (3). Importance weight achieves around 20% of HQS gain reaching almost 70% within 20 epochs even though it shows fluctuating values in very early iterations.

In this experiment, we expand MoG dataset to image space with MNIST dataset, and stacked MNIST. Evaluation on MNIST data (Figure 4) shows that proposed method generates more number of digits than original GAN. Stacked MNIST is designed for high complexity evaluation with extended number of modes. Stacked MNIST is synthesized by stacking different MNIST digits in respective colors. This synthesized dataset has 1000 modes which are the combinations of 10 classes in 3 channel. We use implemented discriminator and generator architecture of standard DCGAN. Auto-encoder has three convolutional layers with 5 by 5 filters, 2 fully connected layers for encoder part and one fully connected layer, 3 transposed convolutional layers are used for decoder. First, we train the classifier using MNIST dataset which assigns the mode to generated images. In this evaluation, we use enough generated samples and they are given to pre-trained classifier to give them a mode. For example, one sample has three channels which represent different digits. Digit for each sample in the channel is determined by MNIST classifier. Experiment is conducted with 50 dimensional encoded space. Learning rate with Adam optimizer for both network is 2e-4, the number of modes, training sample and test generated sample is 1,000, 500,000, 50,000, respectively. We experimentally choose $\lambda_{p}$ = 500.

We demonstrate proposed method encourages cover up all modes in target distribution through qualitative and quantitative evaluation. In figure 8, each vertical and horizontal axis indicates the number of founded mode out of 1000 stacked modes and the number of training steps respectively. We compare DCGAN with proposed network in first chart. The number of Mode founded by DCGAN is less than 200 and quality of generated sample is also not good as shown in figure 9. However, the first chart shows our method gives a great support to DCGAN to find uncovered modes and finally it covers all modes in target distribution. We also verified how much mode penalty weight affects finding mode through switching weight on/off. As shown in the second chart, training generator with penalty weight shows greater performance than without it. Based on the result, we confirm that focusing on discovering missing mode ensures fast convergence to covering entire distribution. Table 2 shows quantitative evaluation on stacked MNISTs with other GANs. We count the modes of generated sample and calculate KL divergence between all target modes and generated modes. Mostly, our method has a superior performance among the others except PacGAN. To give further explanation on PacGAN, as packing is a simple embedding technique on existing method, PacGAN uses WGAN and DCGAN for stacked MNIST evaluation (table 2). Without packing embedding, proposed method still shows better performance than all types of PacWGAN and competitive results over PacDCGAN. Proposed method has shown outperforming evaluation results on existing methods and embedding packing on proposed method may produce improved results.

We also qualitatively evaluate our method using CIFAR-10 dataset, which includes 32x32 color images with 10 classes collected by [7]. The architecture and hyper parameter is same to 3.2. Generated results of VEEGAN and DCGAN are collected from [13]. As shown in Figure 6, VEEGAN and DCGAN frequently include identical samples suffering from mode collapsing (see yellow marks).

Proposed method requires pre-trained auto-encoder of proper structure for target real data set. This involves additional computation cost and processing time compared to other methods. If we have a pre-trained general auto-encoder with color image set, we can simply utilize it for other applications using color real data set with or without fine tuning step.