Crossing-Domain Generative Adversarial Networks for Unsupervised Multi-Domain Image-to-Image Translation

GANs (Goodfellow et al., 2014) were introduced to model a data distribution using independent latent variables. Let $\bm{x}\sim p(\bm{x})$ be a random variable representing the observed data and $\bm{z}\sim p(\bm{z})$ be a latent variable. The observed variable is assumed to be generated by the latent variable, i.e., $\bm{x}\sim p_{\bm{\theta}}(\bm{x}|\bm{z})$, where $p_{\bm{\theta}}(\bm{x}|\bm{z})$ can be explicitly represented by a generator in GANs. GANs are built on top of neural networks, and can be trained with gradient descent based algorithms.

The GAN model is composed of a discriminator $D_{\bm{\phi}}$, along with the generator $G_{\bm{\theta}}$. The training involves a min-max game between the two networks. The discriminator $D_{\bm{\phi}}$ is trained to differentiate ‘fake’ samples generated from the generator $G_{\bm{\theta}}$ from the ‘real’ samples from the true data distribution $p(\bm{x})$. The generator is trained to synthesize samples that can fool the discriminator by mistaking the generated samples for genuine ones. They both can be implemented using neural networks.

At the training phase, the discriminator parameters $\bm{\phi}$ are firstly updated, followed by the update of the generator parameters $\bm{\theta}$. The objective function is given by:

The samples can be generated by sampling $\bm{z}\sim p(\bm{z})$, then $\hat{\bm{x}}=G_{\bm{\theta}}(\bm{z})$, where $p(\bm{z})$ is a prior distribution, for example, a multivariate Gaussian.

Image-to-image translation problem is a kind of image generation task that given an input image $\bm{x}$ of domain X, the model maps it into a corresponding output image $\bm{y}$ of another domain Y. It learns a mapping between two domains given sufficient training data (Isola et al., 2017). Early works on image-to-image translation mainly focused on tasks where the training data of domain $X$ are similar to the data of domain $Y$ (Gupta et al., 2012; Liu et al., 2008), and the results were often unrealistic and not diverse.

In recent years, deep generative models have shown increasing capability of synthesizing diverse, realistic images that capture both fine-grained details and global coherence of natural images (Gregor et al., 2015; Radford et al., 2015; Kingma and Welling, 2014). With Generative Adversarial Networks (GANs) (Isola et al., 2017; Zhu et al., 2017; Kim et al., 2017), recent studies have already taken significant steps in image-to-image translation. In (Isola et al., 2017), the authors use a conditional GAN on different image-to-image translation tasks, such as synthesizing photos from label maps and reconstructing objects from edge maps. However, this method requires input-output image pairs for training, which is in general not available in image-to-image translation problems. For situations where such training pairs are not given, in (Zhu et al., 2017), the authors proposed CycleGAN to tackle unsupervised image-to-image translation. With a pair of Generators $G$ and $F$, the model not only learns a mapping $G:X\rightarrow Y$ using an adversarial loss, but constrains this mapping with an inverse mapping $F:Y\rightarrow X$. It also introduces a cycle consistency loss to enforce $F(G(X))\approx X$, and vice versa. In settings where paired training data are not available, the authors showed promising qualitative results. The authors in (Kim et al., 2017) and (Yi et al., 2017) use similar idea to solve the unsupervised image-to-image translation tasks.

These approaches only tackle the problems of translating images between two domains, and have two major drawbacks. First, when applied to $n$ domains, these approaches need $n(n-1)$ generators to complete the task, which is computationally inefficient. To train all models, it would either require a significant amount of time to complete if the training is performed on one GPU, or it will require a lot of hardware and computing resources if training is run over multiple GPUs. Second, as each model is trained with only two datasets, the training cannot benefit from the data of other domains.

Our work is inspired by multimodal learning (Ngiam et al., 2011), which shows that data features can be better extracted using one modality if multiple modalities are present at feature learning time. The intuition of our method is that if we can encode the information of different domains together and generate a high-level feature space, it would be possible to decode the high-level features to build images of different domains. In this work, rather than generating images from random noise, we incorporate an encoding process into a GAN model. The image-to-image translation can be achieved by first encoding real images into high-level features, and then generating images of different domains using the high-level features through a decoding process. The encoding process and the decoding process are constrained by a weight-sharing technique that both the highest layer and the lowest layer are shared across the two encoders as well as the two generators. Sharing the high-level layers makes sure that the generated images are semantically correct, while sharing the low-level layers ensures that important low-level features be captured and transferred between domains. Our model is trained end-to-end using data from all $n$ domains.

We first describe how to apply our model to multi-domain image-to-image translation in general then illustrate it using two domains as an example. As shown in Fig. 2a, our proposed CD-GAN model consists of a pair of encoders followed by a pair of GANs. Taking domain $X$ and $Y$ as an example, the two encoders $E_{X}$ and $E_{Y}$ encode domain information from $X$ and $Y$ into a set of high-level features contained in a set $Z$. Then from a high-level feature $z$ in space $Z$, we can generate images that fall into domain $X$ or $Y$. The generated images are then evaluated by the corresponding discriminators $D_{X}$ and $D_{Y}$ to see whether they look real and cannot be identified as generated ones. For example, following the red arrows, the input image $\bm{x}$ is first encoded into a high-level feature $\bm{z}_{x}$, then $\bm{z}_{x}$ is decoded to generate the image $\hat{\bm{y}}$. The image $\hat{\bm{y}}$ is the translated image in domain $Y$. Similar processes exist for image $\bm{y}$.

Our model is also constrained by a reconstruction process shown in Fig. 2b. For example, following the red arrows, the input image $\bm{x}$ is first encoded into a high-level feature $\bm{z}_{x}$, then $\bm{z}_{x}$ is decoded to generate the image $\bm{x}\prime$, which is a reconstruction of the input image. Similar processes exist for image $\bm{y}$.

Learning with deep neural networks involves hierarchical feature representation. In order to support flexible cross-domain image translation and also to improve the training quality, we propose the use of double-layer sharing where the highest-level and the lowest-level layers of the two encoders share the same weights and so does the two generators. By enforcing the layers that decode high-level features in GANs to share weights, the images generated by different generators can have some common high-level semantics. The layers that decode low-level details then map the high-level features to images in individual domains.

Sharing weights of low-level layers has the benefit of transferring low-level features of one domain to the other, thus making the image-to-image translation more close to real images in the respective domains. Besides, sharing layers reduces the complexity of the model, making it more resistant to the over-fitting problem.

In state-of-the-art techniques, like CycleGAN, each domain is described by a specific generator, thus there is no need to inform the generator which domain the input image is generated to. However, in our model, multiple domains share two generators. For an input image, we have to include an auxiliary variable to guide the generation of image for a specific domain. The only information we have is the domain labels. To make use of this information, the inputs of the model are not images $\bm{x}$, $\bm{y}$, but image pairs $(\bm{x},\bm{l}_{y})$ and $(\bm{y},\bm{l}_{x})$ where the labels $\bm{l}_{y}$ and $\bm{l}_{x}$ inform the generators which domains to generate an image for. These image pairs are not the same as the image pairs of supervised image-to-image generation tasks, which are $(\bm{x},\bm{y})$. Thus no matter which domain images are the input, the model can always generate images of a domain of interest.

We denote the data distributions as $\bm{x}\sim p(\bm{x})$ and $\bm{y}\sim p(\bm{y})$. As illustrated in Fig. 2, our model includes four mappings, two translation mappings $X\rightarrow Z\rightarrow Y$, $Y\rightarrow Z\rightarrow X$ and two reconstruction mappings $X\rightarrow Z\rightarrow X$, $Y\rightarrow Z\rightarrow Y$. The translation mappings constrain the model by a GAN loss, while the reconstruction mappings constrain the model by a reconstruction loss. To further constrain the auxiliary variable, we introduce a classification loss by applying a classifier to classify the real or generated images into different domains. The intuition is that if images are generated with the guidance of the auxiliary variable, then it can be correctly classified into the domain specified by the auxiliary variable. Next, we introduce these model losses in more details as follows.

GAN Losses Following the translation mapping $X\rightarrow Z\rightarrow Y$, we can translate image $\bm{x}$ from domain $X$ to $\hat{\bm{y}}$ of domain $Y$ using $\bm{z}_{x}=E_{X}(\bm{x})$, $\hat{\bm{y}}=G_{Y}(\bm{z}_{x},\bm{l}_{y})$. With the purpose of improving the quality of the generated samples, we apply adversarial loss. We express the objective as:

where $G_{Y}$ tries to generate images $\hat{\bm{y}}=G_{Y}(\bm{z}_{x},\bm{l}_{y})$ that look similar to images from domain $Y$, while $D_{Y}$ aims to distinguish between translated samples $\hat{\bm{y}}$ and real samples $\bm{y}$. The similar adversarial loss for $Y\rightarrow Z\rightarrow X$ is

The total GAN loss is:

Reconstruction Loss The reconstruction mappings $X\rightarrow Z\rightarrow X$, $Y\rightarrow Z\rightarrow Y$ encourage the model to encode enough information to the high-level feature space $Z$ from each domain. The input can then be reconstructed by the generators. The reconstruction process of domain $X$ is $\bm{z}_{x}=E_{X}(\bm{x})$, $\bm{x}\prime=G_{X}(\bm{z}_{x},\bm{l}_{x})$. Similar reconstruction process exists for domain $Y$. With $l_{2}$ distance as the loss function, the reconstruction loss is:

Latent Consistency Loss With only the above losses, the encoding part is not well constrained. We constrain the encoding part using a latent consistency loss. Although $\bm{x}$ is translated to $\hat{\bm{y}}$, which is in domain $Y$, $\hat{\bm{y}}$ is still semantically similar to $\bm{x}$. Thus, in the latent space $Z$, the high-level feature of $\bm{x}$ should be close to that of $\hat{\bm{y}}$. Similarly, the high-level feature of $\bm{y}$ in domain $Y$ should be close to the high-level feature of $\hat{\bm{x}}$ in domain $X$. The latent consistency loss is the following:

Classification Loss We consider $n$ domains as $n$ categories in the classification problems. We use a network $C$, which is an auxiliary classifier, on top of the general discriminator $D$ to measure whether a sample (real or generated) belongs to a specific fine-grained category. The output of the classifier $C$ represents the posterior probability $P(c|\bm{x})$. Specifically, there are four classification losses, i.e., for real data $\bm{x}$, $\bm{y}$, and generated data $\hat{\bm{x}}$, $\hat{\bm{y}}$. For image-label pairs ($\bm{x}$, $\bm{l}_{x}$) and ($\bm{y}$, $\bm{l}_{y}$) with $\bm{l}_{x}\sim p(\bm{l}_{x})$ and $\bm{l}_{y}\sim p(\bm{l}_{y})$ our goal is to translate $\bm{x}$ to $\hat{\bm{y}}$ with label $\bm{l}_{y}$, and to translate $\bm{y}$ to $\hat{\bm{x}}$ with label $\bm{l}_{x}$. The four classification losses are:

This loss can be used to optimize discriminators $D_{X}$, $D_{Y}$, generators $G_{X}$, $G_{Y}$, and encoders $E_{X}$, $E_{Y}$.

Cycle Consistency Loss Although the minimization of GAN losses ensures that $G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})$ produce a sample $\hat{\bm{y}}$ similar to samples drawn from $Y$, the model still can be unstable and prone to failure. To tackle this problem, we further constrain our model with a cycle-consistency loss (Zhu et al., 2017). To achieve this goal, we want mapping from domain $X$ to domain $Y$ and then back to domain $X$ to reproduce the original sample, i.e., $G_{X}(E_{Y}(G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})),\bm{l}_{x})\approx\bm{x}$ and $G_{Y}(E_{X}(G_{X}(E_{Y}(\bm{y}),\bm{l}_{x})),\bm{l}_{y})\approx\bm{y}$. Thus, the cycle-consistency loss is:

Final Objective of CD-GAN To sum up, the goal of our approach is to minimize the following objective:

where $E$, $G$, and $D$ signify encoders $E_{X}$, $E_{Y}$, generators $G_{X}$, $G_{Y}$, and discriminators $D_{X}$, $D_{Y}$, and $\alpha_{0}$, $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ control the relative importance of the losses. Same as solving a regular GAN problem, training the model involves the solving of a min-max problem, where $E_{X}$,$E_{Y}$, $G_{X}$, and $G_{Y}$ aim to minimize the objective, while $D_{X}$ and $D_{Y}$ aim to maximize it.

Our proposed model has two encoder-generator pairs, but we have data from $n$ domains. To train the model using samples of all domains equally, we introduce a cross-domain training algorithm. As shown in Fig. 1b, there are 4 domains. At each iteration, we randomly select two domains $R$ and $S$, and feed training data of these two domains into the model. At the next iteration, we might take another two domains $P$ and $Q$, and perform the same training. We train the model using all data samples of $4$ domains at every epoch for several iterations. The training algorithm is shown in Algorithm  1. Cross-domain training ensures the model to learn a generic feature representation of all domains by training the model equally on independent domains.

To evaluate the scalability and effectiveness of our model, we test it on a variety of multi-domain image-to-image translation tasks using the following datasets:

Alps Seasons dataset (Anoosheh et al., 2017) is collected from images on Flickr. The images are categorized into four seasons based on the provided timestamp of when it was taken. It consists of four categories: Spring, Summer, Fall, and Winter. The training data consists of 6053 images of four seasons, while the test data consists of 400 images.

Painters dataset (Zhu et al., 2017) includes painting images of four artists Monet, Van Gogh, Cezanne, and Ukiyo-e. We use 2851 images as the training set, and 200 images as the test set.

CelebA dataset (Liu et al., 2015) contains ten thousand identities, each of which has twenty images, i.e., two hundred thousand images in total. Each image in CelebA is annotated with 40 face attributes. We resize the initial $178\times 218$ size images to $256\times 256$. We randomly select 4000 images as test set and use all remaining images for training data.

We run all the experiments on a Ubuntu system using an Intel i7-6850K, along with a single NVIDIA GTX 1080Ti GPU.

We compare the performance of our proposed CD-GAN with that of two reference models:

CycleGAN (Zhu et al., 2017) This method trains two generators $G:X\rightarrow Y$ and $F:Y\rightarrow X$ in parallel. It not only applies a standard GAN loss respectively for $X$ and $Y$, but applies forward and backward cycle consistency losses which ensure that an image $\bm{x}$ from domain $X$ be translated to an image of domain $Y$, which can then be translated back to the domain $X$, and vice versa.

DualGAN (Yi et al., 2017) This method uses a dual-GAN mechanism, which consists of a primal GAN and a dual GAN. The primal GAN learns to translate images from domain $X$ to domain $Y$, while the dual-GAN learns to invert the task. Images from either domain can be translated and then reconstructed. Thus a reconstruction loss can be used to train the model.

UNIT (Liu et al., 2017) This method consists of two VAE-GANs with a fully shared latent space. To complete the task of image-to-image translation between $n$ domains, it needs to be trained $\frac{n\times{(n-1)}}{2}$ times.

DB (Hui et al., [n. d.]) This method addresses the multi-domain image-to-image translation problem by introducing $n$ domain-specific encoders/decoders to learn an universal shared-latent space.

There is a challenge to evaluate the quality of synthesized images (Salimans et al., 2016). Recent works have tried using pre-trained semantic classifiers to measure the realism and discriminability of the generated images. The idea is that if the generated images look to be more close to real ones, classifiers trained on the real images will be able to classify the synthesized images correctly as well. Following (Zhang et al., 2016; Isola et al., 2017; Wang and Gupta, 2016), to evaluate the performance of the models in classifying generated images quantitatively, we apply the metric classification accuracy. For each experiment, we generate enough number of images of different domains, then we use a pre-trained classifier which is trained on the training dataset to classify them to different domains and calculate the classification accuracy.

The design of the architecture is always a difficult task (Radford et al., 2015). To get a proper model architecture, we adopt the architecture of the discriminator from (Isola et al., 2017) which has been proven to be proficient in most image-to-image generation tasks. It has 6 convolutional layers. We keep the discriminator architecture fixed and vary the architectures of the encoders and generators. Following the design of the architectures of the generators in (Isola et al., 2017), we use two types of layers, the regular convolutional layers and the basic residual blocks (He et al., 2016). Since the encoding process is the inverse of the decoding process, we use the same layers for them but put the layers in the inverse orders. The only difference is the first layer of the encoder and the last layer of the generator. We apply $64$ channels (corresponding to different filters) for the first layer of the encoders, but $3$ channels for the last layer of the generators since the output images have only $3$ RGB channels. We gradually change the number of convolutional layers and the number of residual blocks until we get a satisfying architecture. We don’t apply weight sharing initially. The performance of different architectures is evaluated on the Painters dataset and shown in Fig. 3. We can see that when the model has 3 regular convolutional layers and 4 basic residual blocks, the model has the best performance. We keep this architecture fixed for other datasets.

We then vary the number of weight-sharing layers in the encoders and the generators. We change the number of weight-sharing layers from 1 to 4. Sharing 1 layer means sharing the highest layer and the lowest layers in the encoder pair. Sharing 2 layers means sharing the highest and lowest two layers. The same sharing method applies for the generator pair (not including the output layer). The results are shown in table 1. We found that sharing 1 layer is enough to have a good performance.

In summary, for the testbed evaluation, we use two encoders each consisting of 3 convolutional layers and 4 basic residual blocks. The generators are composed with 4 basic residual blocks and 3 fractional-strided convolutional layers. The discriminators consist of a stack of 6 convolutional layers. We use LeakyReLU for nonlinearity. The two encoders share the same parameters on their layers 1 and 7, while the two generators share the same parameters on layers 1 and 6, which is the lowest-level layer before the output layer. The details of the networks are given in table 2. We evaluate various network architectures in the evaluation parts. We fix the network architecture as in Table 2.

We use ADAM (Kingma and Ba, 2014) for training, where the training rate is set to 0.0001 and momentums are set to 0.5 and 0.999. Each mini-batch consists of one image from domain $X$ and one image from domain $Y$. Our model has several hyper-parameters. The default values are $\alpha_{0}=10$, $\alpha_{1}=0.1$, ${\alpha}_{2}=0.1$, and ${\alpha}_{3}=10$. The hyper-parameters of the baselines are set to the suggested values by the authors.

We evaluate our model on different datasets and compare it with baseline models.

We compare the ablations of our full loss. As GAN loss and cycle consistency loss are critical for the training of unsupervised image-to-image translation, we keep these two losses as the baseline model and do the ablation experiments to see the importance of other losses.

As shown in Tabel 3, the reconstruction loss $R$ is least important with accuracy improvement of about 4.6% on Painters dataset and 3.7% on Alps Seasons dataset. The latent consistency loss $LCL$ brings the model an accuracy improvement of 26.1% on Painters dataset and 20.4% on Alps Seasons dataset. The accuracy is improved by 23.8% on Painters dataset and 15.4% on Alps Seasons dataset by the classification loss $C$.

We demonstrate our model on three unsupervised multi-domain image-to-image translation tasks.

Painting style transfer (Fig. 6) We train our model on Painters dataset and use it to generate images of size $256\times 256$. The model can transfer the painting style of a specific painter to the other painters, e.g., transferring the images of Cezanne to images of other three painters Monet, Ukiyoe and Vangogh. In Fig. 7, we also compare our model with other reference models when given the same test image.

Season transfer (Fig. 8) The model is trained on the Alps Seasons dataset. We use the trained model to generate images of different seasons. For example, we generate an image of summer from an image of spring and vice versa. In Fig. 9, we also compare our model with other reference models when given the same test image.

Attribute-base face translation (Fig. 10) We train the model on CelebA dataset for attribute-based face translation tasks. We choose 4 attributes, black hair, blond hair, brown hair, and gender. We then use our model to generate images with these attributes. For example, we transfer an image with a man wearing black hair to a man with blond hair, or transfer a man to a woman.