GMM-UNIT: Unsupervised Multi-Domain and Multi-Modal Image-to-Image Translation via Attribute Gaussian Mixture Modeling

GMM-UNIT follows the generative-discriminative philosophy. The generator inputs a content latent code $\mathbf{c}\in\boldsymbol{\mathcal{C}}\subset\mathbb{R}^{C}$ and an attribute latent code $\mathbf{z}\in\boldsymbol{\mathcal{Z}}\subset\mathbb{R}^{Z}$, and outputs a generated image $G(\mathbf{c},\mathbf{z})$. This image is then fed to a discriminator that must discern between “real” or “fake” images ($D_{\textrm{r/f}}$), and must also recognize the domain of the generated image ($D_{\textrm{dom}}$). The attribute and content latent representations need to be learned, and they are modeled by two architectures, namely a content extractor $E_{c}$ and an attribute extractor $E_{z}$. See Figure 2 for a graphical representation of GMM-UNIT for an $\boldsymbol{A}\leftrightarrow\boldsymbol{B}$ domain translation.

In addition to tackling the problem of multi-domain and multi-modal translation, we would like these two extractors, content and attribute, to be disentangled [15]. This would constrain the learning and hopefully yield better domain translation, since the content would be as independent as possible from the attributes. We expect the attributes features to be related to the considered attributes, while the content features are supposed to be related to the rest of the image. Formally, the following two properties must hold:

Sampled attribute translation

$$G(E_{c}(\mathbf{x}^{m}),\mathbf{z}^{n})\sim p_{\boldsymbol{\mathcal{X}}^{n}}\forall\;\mathbf{z}^{n}\sim{\cal N}(\boldsymbol{\mu}^{n},\boldsymbol{\Sigma}^{n}),\;\mathbf{x}^{m}\sim p_{\boldsymbol{\mathcal{X}}^{m}},\;n,m\in\{1,\ldots,K\}.$$

Extracted attribute translation

$$G(E_{c}(\mathbf{x}^{m}),E_{z}(\mathbf{x}^{n}))\sim p_{\boldsymbol{\mathcal{X}}^{n}}\quad\forall\;\mathbf{x}^{n}\sim p_{\boldsymbol{\mathcal{X}}^{n}},\;\mathbf{x}^{m}\sim p_{\boldsymbol{\mathcal{X}}^{m}},\;n,m\in\{1,\ldots,K\}.$$

The encoders $E_{c}$ and $E_{z}$, and the generator $G$ need to be learned to satisfy three main properties. Consistency: An image and its generated/extracted codes have to be consistent even after a translation from a domain $\boldsymbol{A}$ to a domain $\boldsymbol{B}$. Fit: The distribution of the attribute latent space must follow a GMM. Realism: The generated images must be indistinguishable from the real images. In the following, we discuss different losses used to force the overall pipeline to satisfy these properties.

In the consistency term, we include image, attribute and content reconstruction, as well as cycle consistency. More formally, we use the following losses:

• •

  Self-reconstruction of any input image from its extracted content and attribute vectors:

  $$\mathcal{L}_{\textrm{s/rec}}=\textstyle\sum_{n=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}}}\big{[}\|G(E_{c}(\mathbf{x}),E_{z}(\mathbf{x}))-\mathbf{x}\|_{1}\big{]}$$

• •

  Content reconstruction from an image, translated into any domain:

  $$\mathcal{L}_{\textrm{c/rec}}=\textstyle\sum_{n,m=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}},\mathbf{z}\sim{\cal N}(\boldsymbol{\mu}^{m},\boldsymbol{\Sigma}^{m})}\big{[}\|E_{c}(G(E_{c}(\mathbf{x}),\mathbf{z}))-E_{c}(\mathbf{x})\|_{1}\big{]}$$

• •

  Attribute reconstruction from an image translated with any content:

  $$\mathcal{L}_{\textrm{a/rec}}=\textstyle\sum_{n,m=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}},\mathbf{z}\sim{\cal N}(\boldsymbol{\mu}^{m},\boldsymbol{\Sigma}^{m})}\big{[}\|E_{z}(G(E_{c}(\mathbf{x}),\mathbf{z}))-\mathbf{z}\|_{1}\big{]}$$

• •

  Cycle consistency when translating an image back to the original domain:

  $$\mathcal{L}_{\textrm{cyc}}=\textstyle\sum_{n,m=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}},\mathbf{z}\sim{\cal N}(\boldsymbol{\mu}^{m},\boldsymbol{\Sigma}^{m})}\big{[}\|G(E_{c}(G(E_{c}(\mathbf{x}),\mathbf{z})),E_{z}(\mathbf{x}))-\mathbf{x}\|_{1}\big{]}$$

We note that all these losses are used in prior work [6, 15, 46, 47] to constraint the infinite number of mappings that exist between an image in one domain and an image into another one. The $\mathcal{L}_{1}$ loss is used as it generates sharper results than the $\mathcal{L}_{2}$ loss [16]. We also propose to complement the Attribute reconstruction with an isometry loss, to encourage the attribute extractor to be as similar as possible to the sampled attributes. Formally:

In the fit term we encourage both the attribute latent variable to follow the Gaussian mixture distribution and the generated images to follow the domain’s distribution. We set two loss functions:

• •

  Kullback-Leibler divergence between the extracted latent code and the model. Since the KL divergence between two GMMs is not analytically tractable, we resort on the fact that we know from which domain are we sampling and define:

  $$\mathcal{L}_{\textrm{KL}}=\textstyle\sum_{n=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}}}[\mathcal{D}_{KL}(E_{z}(\mathbf{x})\|\mathcal{N}(\boldsymbol{\mu}^{n},\boldsymbol{\Sigma}^{n}))]$$

  where $\mathcal{D}_{KL}(p\|q)=-\int p(t)\log\frac{p(t)}{q(t)}dt$ is the Kullback-Leibler divergence.

• •

  Domain classification of generated and original images. For any given input image $\mathbf{x}$, we would like the method to classify it as its original domain, and to be able to generate from its content an image in any domain. Therefore, we need two different losses, one directly applied to the original images, and a second one applied to the generated images:

  	$\displaystyle\mathcal{L}_{\text{dom}}^{D}$	$\displaystyle=\textstyle\sum_{n=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}},d_{\boldsymbol{\mathcal{X}}^{n}}}[-\log D_{\text{dom}}(d_{\boldsymbol{\mathcal{X}}^{n}}|\mathbf{x})],$
  	$\displaystyle\mathcal{L}_{\text{dom}}^{G}$	$\displaystyle=\textstyle\sum_{n,m=1}^{K}\mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\mathcal{X}}^{n}},d_{\boldsymbol{\mathcal{X}}^{m}},\mathbf{z}\sim{\cal N}(\boldsymbol{\mu}^{m},\boldsymbol{\Sigma}^{m})}[-\log D_{\text{dom}}(d_{\boldsymbol{\mathcal{X}}^{m}}|G(E_{c}(\mathbf{x}),\mathbf{z}))]$

where $d_{\boldsymbol{\mathcal{X}}^{n}}$ is the label of domain $n$. Importantly, while the generator is trained using the second loss only, the discriminator $D_{\textrm{dom}}$ is trained using both.

The realism term tries to making the generated images indistinguishable from real images; we adopt the adversarial loss to optimize both the real/fake discriminator $D_{\textrm{r/f}}$ and the generator $G$:

We quantitatively evaluate our method through image quality and diversity of generated images. The former is evaluated through the Fréchet Inception Distance (FID) [13], while we evaluate the latter through the LPIPS [44].
FID We use FID to measure the distance between the generated and real distributions. Lower FID values indicate better quality of the generated images. We estimate the FID using 1000 input images and 10 samples per input v.s. randomly selected 10000 images from the target domain.
LPIPS The LPIPS distance is defined as the $\mathcal{L}_{2}$ distance between the features extracted by a deep learning model of two images. This distance has been demonstrated to match well the human perceptual similarity [44]. Thus, following [15, 20, 47], we randomly select 100 input images and translate them to different domains. For each domain translation, we generate 10 images for each input image and evaluate the average LPIPS distance between the 10 generated images. Finally, we get the average of all distances. Higher LPIPS distance indicates better diversity among the generated images.

We first evaluate our model on a simpler task than multi-domain translation: two-domain translation (e.g. edges to shoes). We use the dataset provided by [16, 46] containing images of shoes and their edge maps generated by the Holistically-nested Edge Detection (HED) [41]. We resize all images to 256$\times$256 and train a single model for edges $\leftrightarrow$ shoes without using paired information. Figure 3 displays examples of shoes generated from the same sketch by all the state of the art models. GMM-UNIT and MUNIT generate high-quality and diverse results that are almost indistinguishable from the ground truth and the results of BicycleGAN, which is a paired (supervised) method. Although, MSGAN and DRIT++ generate diverse images, they suffer from low quality results. The results of StarGAN* confirm the findings of previous studies that only adding noise does not increase diversity [16, 27, 47]. These results are confirmed in the quantitative evaluation displayed in Table 2. Our model generates images with high diversity and quality using half the parameters of the state of the art (MUNIT), which needs to be re-trained for each transformation. Particularly, the diversity is comparable to the paired model performance. These results show that this multi-modal and multi-domain model can be efficiently applied also to simpler tasks than multi-domain problems without much loss in performance, while other multi-domain models suffer in this setting. We refer to the Supplementary for additional results on this task.

We then increase the complexity of the task by evaluating our model in a multi-domain translation setting, where each domain is composed by digits collected in different scenes. We use the Digits-Five dataset introduced in [43], from which we select three different domains, namely MNIST [19], MNIST-M [9], and Street View House Numbers (SVHN) [30]. During the training, given that all images are resized to 32$\times$32, we reduce the depth of our model and compared models. We compare our model with the state-of-the-art on multi-domain translation, and we show in Table 3 the quantitative results. We add in the Supplementary extensive qualitative results for space limit reasons.

From these results we conclude that StarGAN* fails at generating diversity, while GMM-UNIT generates images with higher quality and diversity than all the state-of-the-art models. Additional experiments carried out implementing a StarGAN*-like GMM-UNIT (i.e. setting $\boldsymbol{\Sigma}^{k}=0,\;\forall k$) indeed produced similar results. Specifically, the StarGAN*-like GMM-UNIT tends to generate for each input image one single (deterministic) output and thus the corresponding LPIPS scores are zero. We refer to the Supplementary for additional results on this task.

We also evaluate GMM-UNIT in the complex setting of multi-domain translation in a dataset of facial attributes. We use the Celebfaces Attributes (CelebA) dataset [23], which contains 202,599 face images of celebrities where each face is annotated with 40 binary attributes. We apply central cropping to the initial 178$\times$218 size images to 178$\times$178, then resize the cropped images to 128$\times$128. We randomly select 2,000 images for testing and use all remaining images for training. This dataset is composed of some attributes that are mutually exclusive (e.g. either male or female) and those that are mutually inclusive (e.g. people could have both blond and black hair). Thus, we model each attribute as a different GMM component. For this reason, we can generate new images for all the combinations of attributes by sampling from the GMM. As aforementioned, this is not possible for state-of-the-art models such as StarGAN and DRIT++, as they use one-hot domain codes to represent the domains. To be consistent with the state of the art (StarGAN) we show five binary attributes: hair color (black, blond, brown), gender (male/female), and age (young/old). These five attributes allow GMM-UNIT to generate 32 domains.

We observed that image-to-image translation is sensitive to complex background information. In fact, models are inclined to manipulate the intensity and details of pixels that are not related to the desired attribute transformation. Hence, we add a convolutional layer at the end the of decoder $G$ to learn a one-channel attention mask $\mathbf{M}$ in an unsupervised manner. Hence, the final prediction $\widehat{\mathbf{B}}$ is obtained through combining the input image $\mathbf{A}$ and its initial prediction $\widetilde{\mathbf{B}}$ through: $\widehat{\mathbf{B}}=\widetilde{\mathbf{B}}\cdot\mathbf{M}+\mathbf{A}\cdot(1-\mathbf{M})$. We also apply the attention layer to Edges $\leftrightarrow$ Shoes and Digits, but find that it provides no noticeable improvements in the results.

Figure 4 shows some generated results of our model. We can see that GMM-UNIT learns to translate images to simple attributes such as blond hair, but also to translate images with combinations of them (e.g. blond hair and male). Moreover, we can see that the rows show different realizations of the model thus demonstrating the stochastic approach of GMM-UNIT. These results are corroborated by Table 3 that shows that our model is superior to StarGAN* and DRIT++ in both quality and diversity of generated images. Particularly, the use of an attention mechanism allows our model to achieve diversity only on the part of the image that is involved in the transformation (e.g. hair and face for gender and hair translation). To demonstrate this, we compute the LPIPS distance between the background of the input image and the generated images (LPIPS_b). Table 3 that our model is the best at preserving the original background information. In Figure 9 we show the difference between the diversity we achieve and DRIT++ diversity. GMM-UNIT preserves the background while it changes the face and create diverse hair styles, while DRIT++ just changes the overall color intensity and affects parts of the image not related to the attributes, which is not desirable. Extensive results are displayed in the Supplementary.

We evaluate our model on style transfer, which is a specific task where the style is usually extracted from a single reference image. Thus, we randomly select two input images and synthesize new images where, instead of sampling from the GMM distribution, we extract the style (through $E_{z}$) from some reference images. Figure 5 shows that the generated images are sharp and realistic, showing that our method can also be effectively applied to Style transfer.

In addition, we evaluate the ability of GMM-UNIT to synthesize new images with attributes that are extremely scarce or non present in the training dataset. To do so, we select three combinations of attributes consisting of less than two images in the CelebA dataset: Black hair+Blond hair+Male+Young and Black hair+Blond hair+Female+Young.

Figure 6 shows that learning the continuous and multi-modal latent distribution of attributes allow to effectively generate images as zero- or few-shot generation. At the best of our knowledge, we are the first ones being able to translate images in previously unseen domains at no additional cost. Recent literature on zero-pair translation learning indeed scale linearly with the number of domains [39]. This ability can be of vital importance in tasks where labels are extremely imbalanced.

Finally, we show that by learning the full latent distribution of the attributes we can do attribute interpolation both intra- and inter-domains. In contrast, state of the art methods such as [21] can only do intra-domain interpolations due to their discrete domain encoding. Other works such as Chen et al. [5] are focused on explicitly learning an interpolation and use a reference image to do the same task, while we can either interpolate between two reference images or between any two points in the attribute latent space (by sampling these points/vectors), even for multiple attributes. Figure 7 shows some generated images through a linear interpolation between two given attributes, while in Supplementary we show that we can also do intra-domain interpolations.

Given that the importance of $\mathcal{L}_{s/rec}$ and $\mathcal{L}_{dom}$ was verified in previous works (i.e. CycleGAN and StarGAN), and that $\mathcal{L}_{c/rec},\mathcal{L}_{KL}$ are necessary to the model convergence, we compare GMM-UNIT with three variants of the model that ablate $\mathcal{L}_{\textrm{cyc}}$, $\mathcal{L}_{\textrm{a/rec}}$ and $\mathcal{L}_{\textrm{iso}}$ in the Digits dataset. Figure 9 shows the results of the ablation. As expected, $\mathcal{L}_{\textrm{cyc}}$ is needed to have higher image quality, and we observe that it increases the diversity because of noisy results. When $\mathcal{L}_{\textrm{a/rec}}$ is removed image quality decreases, but $\mathcal{L}_{\textrm{iso}}$ still helps to learn the attributes space. Finally, without $\mathcal{L}_{\textrm{iso}}$ we observe that both diversity and quality decrease, thus confirming the need of all these losses. For the first time from its introduction in [15], we also test for the disentangled assumption of visual content and attributes. Although we cannot test the network removing the attribute extractor $E_{z}$, we remove the content extractor $E_{c}$ and change the generator $G$ to have $\mathbf{x}$ and $\mathbf{z}$ as input. We observe that the results are similar, although the diversity decreases substantially. This means that the disentanglement approach needs to be further studied in the multiple architectures and tasks that propose it [12, 15, 40] to understand its necessity and contribution. We refer to Supplementary for the disentanglement and the additional ablation results broken down by domain.