CircleGAN: Generative Adversarial Learning
across Spherical Circles

Previous work for improving GANs concentrates on addressing the difficulty of training. These studies have been conducted in different aspects such as network architectures [11, 14, 15, 25], objective functions [13, 22] and regularization techniques [2, 11, 20, 35], which impose the Lipschitz constraint to the discriminator. SphereGAN [24] has shown that using hypersphere as an embedding space affords the stability in GAN training by the boundedness of distances between samples and their gradients. Our work also adopts a hypersphere embedding space, but proposes a different strategy in structuring and learning the hypersphere, which will be discussed in details.

The most relevant line of GAN research to ours is on the lack of sample diversity. In many cases, GAN objectives can be satisfied with samples of a limited diversity, and no guarantee exists that a model in training reaches to generate diverse samples. To tackle the lack of diversity, a.k.a. mode collapse, several approaches are proposed from different perspectives. Chen et al. [5] and Karras et al. [15] modulate normalization layers using a noise vector that is transformed through a sequence of layers. Yang et al. [32] penalize the mode collapsing behavior directly to the generator by maximizing the gradient norm with respect to the noise vector. Yamaguchi and Koyama [31] regularize the discriminator to have local concavity on the support of the generator function to increase its entropy monotonically at every stage of the training. Liu et al. [18] propose a spectral regularization to prevent spectral collapse when training with the spectral normalization, which is shown to be closely linked to the mode collapse. Our approach is very different from these in combating mode collapse.

The most straightforward way to improve the performance of GANs is to incorporate side information of the samples, typically class labels. Depending on how the class information is used, the models are categorized into two types: projection-based models and classifier-based models.

Projection-based models [19] discriminate samples by projecting them onto two embeddings, meaning ‘real’ and ‘fake’, for the corresponding class and measuring the discrepancy of the projection values. As the network architectures are combined with attention modules [33] and increased to high-capacity [4], these models improve synthesizing large-scale and high-fidelity images, achieving the state-of-the-art performance. However, since they do not learn class embeddings through an explicit classifier, it may not be easy for the methods to learn the class-specific features.

In contrast, classifier-based models [23, 26] use an auxiliary classifier with a discriminator to explicitly learn class-specific features for generation. However, when incorporating the auxiliary classifier, the diversity of generated samples often degrades severely because the generator focuses on generating easily-classifiable samples [9, 34]. To prevent the generator from being over-confident to some samples, Zhou et al. [34] introduces an adversarial loss on the classifier. But, the loss often deteriorates the classifier and hinders the models from scaling up to large datasets (e.g., ImageNet). While Gong et al. [9] propose a scalable model using twin auxiliary classifiers, they require one more classifier to prevent the other from degeneration.

Unlike these methods, we address the mode-collapse issue based on the hypersphere-based discriminators by integrating only a single auxiliary classifier without any adversarial loss.

Let samples $\mathcal{X}=\{\bm{x}\}_{i=1}^{n}$ be drawn from both data distribution $\mathcal{P}_{\mathrm{data}}$ and the generator distribution $\mathcal{P}_{\mathrm{gen}}$. The discriminator $D$ first embeds the samples $\mathcal{X}$ to $\mathcal{V}^{\mathrm{emb}}=\{\bm{v}^{\mathrm{emb}}_{i}\}_{i=1}^{n}$ and then projects them on a unit hypersphere with a learnable center $\bm{c}$: $\bm{v}_{i}=(\bm{v}^{\mathrm{emb}}_{i}-\bm{c})/\|\bm{v}^{\mathrm{emb}}_{i}-\bm{c}\|$. We denote the sets of sample embeddings on the hypersphere by $\mathcal{V}=\{\bm{v}_{i}\}_{i=1}^{n}$ . Given a unit pivotal vector $\bm{p}$, which is learnable, we can define the set of spherical circles $\Omega_{p}$ that is perpendicular to $\bm{p}$. Then, each point $\bm{v}_{i}$ (i.e., sample embedding) on the hypersphere is assigned to a corresponding circle $\omega_{m(i)}\in\Omega_{p}$, where the function $m$ represents an injective mapping from $\mathcal{V}$ to $\Omega_{p}$. As shown in the right of Fig. 1, using the sample embedding $\bm{v}_{i}$ and the pivotal vector $\bm{p}$, we define its projected vector $\bm{v}^{\mathrm{proj}}_{i}$ and its rejected vector $\bm{v}_{i}^{\mathrm{rej}}$:

where $\langle\cdot,\cdot\rangle$ indicates the inner product between two vectors. In a nutshell, each $\bm{v}_{i}$ corresponds to a spherical circle $\omega_{m(i)}\in\Omega_{p}$ that is identified with $\bm{v}^{\mathrm{proj}}_{i}$. Note that both center $\bm{c}$ and pivotal vectors $\bm{p}$ are learned in training to adapt the hypersphere to the sample embeddings.

Our strategy in learning the discriminator is to populate real samples around the longest circle in $\Omega_{p}$, i.e., the great circle, while pushing fake samples toward the shortest circles in $\Omega_{p}$, i.e., the top or bottom pole. Since longer circles occupy the larger area on the hypersphere, they may allow more diversity in representation learning, and vice versa. In this sense, discriminating real and fake samples based on their corresponding circles can naturally induce diversity to generated samples.

We propose to measure the realness score for sample embedding $\bm{v}_{i}$ based on the proximity of its corresponding circle to the great circle, which is computed by

where the score is normalized with its standard deviation $\sigma^{\mathrm{proj}}$ to fix the scale consistently through the course of training and $\sigma^{\mathrm{proj}}$ is computed as $\sqrt{{\scriptstyle\sum}_{i=1}^{|\mathcal{V}|}\|\bm{v}_{i}^{\mathrm{proj}}\|_{2}^{2}/|\mathcal{V}|}$. On the other hand, we define the diversifiability score for $\bm{v}_{i}$ by the radius of corresponding circle which can be quantified by

where $\sigma^{\mathrm{rej}}$ is computed as $\sqrt{{\scriptstyle\sum}_{i=1}^{|\mathcal{V}|}\|\bm{v}_{i}^{\mathrm{rej}}\|_{2}^{2}/|\mathcal{V}|}$.

Note that the diversifiability score increases along with the realness score. Thus we use the realness score function $s^{\mathrm{real}}$ as the discriminator output so that it guides the generator in CircleGAN training. The discrimination based on the spherical circles increases the diversity of realistic samples while suppressing the diversity of unrealistic samples, which improves training the generator. This is supported by the experimental results in Sec. 4.2 and 4.3.

We design two other variants for scoring that explicitly combine both the realness score and the diversifiability score:

The former performs a simple addition of realness and diversifiability scores, and the latter measures an angle between the pivotal vector and sample embedding. The effects of these two variants will be demonstrated in our experiment.

To train the model based on the proposed score functions, we adopt the relativistic averaged loss [13] considering its robustness and simplicity:

where $\tau$ adjusts the range of score difference for the sigmoid functions and is set to 5 for $s^{\mathrm{add}}$ and $s^{\mathrm{real}}$ and 10 for $s^{\mathrm{mult}}$. For the adversarial loss of generator $\mathcal{L}_{G}^{\mathrm{adv}}$, we set it as the inverse of the discriminator loss by changing the source of the samples.

In the following, we introduce two additional losses to improve the training dynamics. The center estimation loss improves adapting hypersphere to the sample embeddings ($\bm{v}^{\mathrm{emb}}_{i}$) and the radius equalization loss increases discriminative power of the hypersphere by enforcing one-to-one mappings from the centered embeddings ($\bm{v}^{\mathrm{emb}}_{i}-\bm{c}$) to spherical embeddings ($\bm{v}$).

We extend our method for the conditional settings and improve the sample diversity of conditional GANs within each class. The key idea is to create multiple hyperspheres for target categories and perform adversarial learning in a class-wise manner. To be concrete, we create a center and a pivotal vector for each class: $\{\bm{c}\}_{l=1}^{L}$ and $\{\bm{p}\}_{l=1}^{L}$, where $L$ is the number of categories. The embeddings are translated with their corresponding center vectors, and the scores for adversarial learning are measured based on their corresponding pivotal vectors. For the rest, in computing non-trainable parameters (e.g., standard deviations in Eqs. 2, 3, 4 and target radius in Eq. 7) and comparing scores in the loss in Eq. 5, we take all samples and associate them together irrespective of class labels due to the limited batch sizes.

We incorporate an auxiliary classifier $C$ to learn the class-specific features. The auxiliary classifier is a simple linear layer attached to the discriminator and is trained without any adversarial loss to predict the class of a sample $\bm{x}$. With this classifier, the generator makes the sample class-specific by maximizing the probability of its corresponding label $y$ :

The total losses for the conditional settings are $\mathcal{L}_{D}=\mathcal{L}_{D}^{\mathrm{adv}}+\mathcal{L}_{D}^{\mathrm{reg}}+\mathcal{L}_{D}^{\mathrm{cls}}$ and $\mathcal{L}_{G}=\mathcal{L}_{G}^{\mathrm{adv}}+\mathcal{L}_{G}^{\mathrm{cls}}$. The overall algorithm with the proposed components is presented in Algorithm 1.

As ours, SphereGAN [24] also employs a hypersphere as the embedding space for the discriminator, but constructs it with a different projection function and learns with a different objective, as shown in Fig. 2. For hypersphere projection, CircleGAN uses translation and $\ell 2$-normalization while SphereGAN uses inverse stereographic projection (ISP). The translation and $\ell 2$-normalization tends to distribute samples evenly onto the hypersphere, but ISP concentrates a large portion of samples around the north pole and maps only a small portion around the south pole. This biased projection may prevent the discriminator from fully exploiting the space in learning. Furthermore, while CircleGAN learns the center and the pivotal point adaptively to sample embeddings, SphereGAN uses a fixed coordinate system fixed with the north pole $\bm{\mathrm{N}}$ and the origin $\bm{\mathrm{O}}$. As demonstrated in Sec. 4.2, our projection method shows better performance than ISP in practice. As for the objective in training, the score function $s^{\text{mult}}_{\phi_{i}}$ is analogous to the SphereGAN objective [24]; both maximize angles between a reference vector and sample embeddings. However, while CircleGAN maximizes the angles to the great circle, SphereGAN does it to the opposite point of the reference vector. The use of circles turns out to make a significant difference in sample diversity and quality, as shown in our experiments. Finally, CircleGAN easily extends to conditional settings by creating multiple hyperspheres, whereas SphereGAN is less flexible due to its specialized projection with the fixed reference vector on the hypersphere.

The common evaluation metrics for image generation are Inception Score (IS) [26] and Frechet Incéption Distance (FID) [12]. IS measures how distinctively each sample is classified as a certain class and how similar the class distribution of generated samples is to that of the target dataset. FID measures a distance between the distribution of real data and that of generated samples in an embedding space, where the embeddings are assumed to be Gaussian distributed. While these are easy to calculate and correlate well with human assessment of generated samples, there have been some concerns about them. First, IS is computed based on class probabilities over Imagenet classes, and thus the evaluation on other datasets cannot be accurate since their class distributions are different from that of Imagenet [34]. Second, IS is highly sensitive to small changes in weights of classifiers [3] and image samples, as shown in our STL10 experiments of Sec. 4.2. Third, FID does not penalize a sample with a similar but different identity, e.g., a clear cat image generated by a dog label, which would be problematic particularly for conditional generations. Hence, in addition to IS and FID, we use two other metrics [28], GAN-train and GAN-test, for evaluation on conditional settings.

GAN-train and GAN-test evaluate diversity and quality of images, respectively. They can easily adapt to each target dataset and also consider mislabeled images in evaluation. GAN-train trains a classifier using generated images and then measures its accuracy on a validation set of the target dataset. This score is analogous to recall (the diversity of images) since the score would increase if the generated samples cover different modes of the target dataset. In contrast, GAN-test trains a classifier using a training set of the target dataset and then measures its accuracy on the generated images. This measure is not related to diversity, but to precision since high-quality samples even from a single mode can achieve a high score. To sum up, we evaluate the models of unconditional GANs with IS and FID, and along with these scores, we use GAN-train and GAN-test for conditional GANs. To measure IS and FID, we use 50K images for all experiments following original implementations.¹¹1IS: https://github.com/openai/improved-gan, FID: https://github.com/bioinf-jku/TTUR

For unconditional generation task, we evaluate our methods on CIFAR10 and STL10 [6]. CIFAR10 and STL10 consists of 50K $32\times 32$ and 100K $96\times 96$ images of 10 classes, respectively. We resize STL images to $48\times 48$ before training, following the experimental protocol of [20, 24]. We compare our methods with two Lipschitz-based models [11, 20] and one hypersphere-based model [24] (Table 1(a)). The best and second-best results are highlighted with red and blue colors, respectively.

CircleGAN models achieve the best and second-best performance in terms of all metrics on the datasets, except for IS on STL10 where SphereGAN performs the best. We suspect that the exception is due to the sensitivity of IS, as also reported in [3]; IS on the test set of STL10 is 14.8, which is significantly lower than 26.1 on the training set, but for other datasets such as CIFAR10 and CIFAR100, IS values are similar between train and test sets (CIFAR10: 11.2 vs. 11.3, CIFAR100: 14.8 vs. 14.7).

To further compare CircleGAN to SphereGAN, we conduct ablation studies on CIFAR10 with the model using angle $s^{\mathrm{mult}}$ for score function (Table 1(b)). Each component in the table is subsequently removed from the full CircleGAN model to see its effect. Here we use the FID metric, which is more stable. First and second, we remove radius equalization and center estimation losses, respectively. Third, we replace the CircleGAN objective with that of SphereGAN, which maximizes the angle to the opposite of pivotal point. Forth, we remove the score normalization. Fifth, we replace $\ell 2$-normalization with ISP of SphereGAN. The results show that the proposed components consistently improve FIDs. In particular, replacing $\ell 2$-normalization with ISP significantly deteriorates FID, which implies that ISP of SphereGAN may be problematic due to the embedding bias of samples. CircleGAN not only outperforms SphereGAN with a large margin, but also easily extends to conditional settings as demonstrated in the next experiment.

We conduct conditional generation experiments on CIFAR10, CIFAR100 [17] and TinyImagenet.²²2https://tiny-imagenet.herokuapp.com/ CIFAR100 consists of 50K $32\times 32$ images of 100 classes and TinyImageNet consists of 100K $64\times 64$ images of 200 classes. We compare our models with a projection-based model [19] and also with two classifier-based models [34, 23], one with adversarial losses on class probability [34] and the other without the losses [23]. We present the results in Table 2, 3, 4 for CIFAR10, CIFAR100, and TinyImagenet, respectively. The numbers inside the parentheses indicate the results of our models without the auxiliary classifier $C$. All the CircleGAN models outperform the other models in terms of all metrics across all the datasets. On the datasets with more classes and more diverse samples with higher resolution, the performance gains over other models become greater. It demonstrates the advantage of class-wise hypersphere placing realistic samples around the great circle in a class-wise manner.

The GAN-train and GAN-test results show that CircleGAN produces higher quality and more diverse samples than other models. In particular for the GAN-test, it remains almost the same at the highest level regardless of the datasets, which implies that every sample captures important features of its corresponding class to be classified correctly by the pre-trained classifier. We attribute this to the auxiliary classifier since all the scores drop significantly on all datasets when the classifier is detached. The overall scores without auxiliary classifier are similar to the projection-based models, which have no classifier. Hence, this supports the use of auxiliary classifier for learning the class-specific features in discriminator and correspondingly to update samples in generator.

However, the auxiliary classifier does not benefit all the other classifier-based models [11, 34], instead they perform significantly worse as the number of classes or the resolution increases. Specifically, AMGAN [34], whose discriminator is trained with an adversarial loss on class probability, shows decent performance on CIFAR10, but dramatically degrades on CIFAR100 and fails to train on TinyImagenet. The other classifier-based model having no adversarial loss also [11, 23] shows competitive results on CIFAR10 and CIFAR100, but poor GAN-train and GAN-test scores on TinyImagenet. These results suggest that utilizing spherical circles is highly effective and flexible to integrate the auxiliary classifier in conditional GANs.

For qualitative evaluation, we sample images and obtain t-SNE using pre-trained classifier for CircleGAN and Proj. SNGAN [19], which is the most competitive algorithm on TinyImagenet. We use 5 classes of images where the classes are selected by the author not to be overlapped in nature: goldfish, yorkshire-terrier, academic gown, birdhouse, schoolbus (Fig. 3). The results demonstrate that the images synthesized from CircleGAN correspond to their classes and almost overlapped to the train and the validation sets of the dataset at the 2D t-SNE space, but the images from Proj. SNGAN are vague in perceiving them as instances of their corresponding classes. Also, the distribution of t-SNE itself certifies that the samples from Proj. SNGAN are far apart from the support of the dataset. The more qualitative results can be found in supplementary material C.