Gated SwitchGAN for multi-domain facial image translation

Generator. As shown in Fig. 1, our generator consists of an attributes encoder switch module with parallel attribute encoder branches $E_{i},i\in(1,n)$, and a single decoder module $M$, where both of the modules are trained at the same time. Our generator takes an image $x$ and a target label vector $c_{trg}$ as input. Image $x$ is encoded by the encoder branches first, and then the encoded feature maps $F$ are multiplied with the target label vector $c_{trg}$ and summed up to generate fused features $F_{fusion}$. Both the original label vector $c_{org}$ and the target label ${c_{trg}}$ can be represented as follows:

where $B_{i}$ is the binary label of the $i$-th attribute. The feature maps $F$ input to the feature-switching operator can also be represented as follows:

For the generator, $F_{i}$ represents the feature map encoded by the $i$-th encoder $E_{i}$. Then, the feature-switching operation can be formulated as follows:

The binary label works as a switch, and only the feature maps corresponding to the non-zero component of label vector pass to the following process. As a result, our generator automatically selects the desired attribute information for multiple domain facial image translation.

It is important to learn the multi-domain joint distribution via a single generator. In one-to-one domain image translation, both the shared common latent space [38] and the content and style decomposition [30] provide special structures to learn such a joint distribution. Inspired by their works, we provide a latent space for the fused features to achieve joint distribution, and then a single decoder successfully learns the mapping from the joint distribution to the desired output. Consequently, our flexible generator is capable of adaptive switching among one-to-many or many-to-many image translation tasks and provides a better interpretability than the previous works. Note that the input label vector must have at least one non-zero value.

For SwitchGAN, the function of the generator can be formulated as follows:

To control the intensity of attribute translation and improve the performance, we introduce a gate module for the SwitchGAN generator. The gate module consists of a commonly shared encoder $E_{m}$ and the intensity of attributes $\alpha$. The shared encoder, with the same structure of the attribute encoder, is added to the SwitchGAN generator. As shown in Fig. 2, the output of the attribute encoder switch is fused with the output of the shared encoder. Then, a similar single decoder successfully learns the mapping from the joint distribution to the desired output. The generator function can be formulated as follows:

Both the SwitchGAN and Gated SwitchGAN share the same training process, while for the testing process, we introduce a vector $\alpha$ to enable the model to control the intensity of translation, which can be represented as follows:

where $\alpha_{i}$ $\in$ [0,1] represents the intensity of the $i$-th attribute. Therefore, the value of each attribute intensity can continuously vary from 0 to 1, and the generator function during testing process can be formulated as follows:

Since only the attribute encoders corresponding to the non-zero label go through the attribute encoder switch, the commonly shared encoder should learn to balance the features among different attribute encoders. In other words, the shared encoder is indirectly forced to encode the content information and the intermediate state of all attributes. As shown in Fig. 2, the content feature map can be decoded to generate a content image, which retains basic semantic content of our input image. As a result, the attribute encoder switch is more focused on the attribute information and improves the ability of translation. When we impose different values on the switch (i.e., $\alpha$ = 0.5 or 1), the output of the generator can show different intensities of attribute translation and simultaneously preserve the content information. The switch works as a gate of attributes. The higher the intensity value we impose, the clearer the attribute appearance it shows. Note that the motivation of the gate module is different from the cycle-consistent network [8, 10], i.e., the gate module is mainly designed to extract the common information and intermediate content. We demonstrate in some experiments that owing to the shared encoder, the Gated SwitchGAN can visualize the variant intensity of translation and the content information of the input image. Please refer to the supplementary material for more details of the generator.

Discriminator. We utilize the same structure of the discriminator as that of the SwitchGAN models. As shown in Fig. 2, our discriminator extracts shallow features by the common channels $D_{m}$, which is followed by a classification branch $D_{cls}$ and an attribute adversary switch $D_{adv}$. The former classifies the attributes of the input image, while the latter distinguishes whether the input image is real or fake. The discriminator function for the classification can be formulated as follows:

Innovatively, the attribute adversary switch $D_{adv}$ consists of parallel adversarial blocks $D_{adv_{i}}$, where the feature-switching operation that is similar to the generator is utilized for the outputs of adversarial blocks. The features are multiplied with the label vector and summed to generate the final output, which can be formulated as follows:

where $D_{adv_{i}}$ represents the specific adversary block for the input with the $i$-th attributes. Similar to the generator, only the features corresponding to the non-zero value of the label vector can go through the switch module.

The key idea of our discriminator is that the adversarial blocks discriminate the fake image as true and converge the model only if the synthesized image is indistinguishable from the real samples corresponding to the specific attributes. In contrast, the discriminator in former methods [11, 12] always directly discriminates the fake image as true and converges the model once the synthesized image is indistinguishable from the samples from database. We realize the importance of the neglected domain distribution information. Therefore, we provide conditional information for the discriminator to distinguish whether the input image is from real or synthesized samples. It is the reason why our model synthesizes images with more apparent attributes and higher quality.

Consequently, we provide a more effective conditional discriminator for multi-domain image translation tasks. The details of the discriminator network are illustrated in the supplementary material.

Given a facial image $x$ with an attribute $c_{org}$ and a target label vector $c_{trg}$, the generator aims to synthesize a face image with desired attribute $c_{trg}$.

Adversarial loss. To conduct the training process of attribute translation, a conditional adversarial constraint is required. The adversarial function is denoted as follows:

It can be separated as $loss_{adv_{D}}$ and $loss_{adv_{G}}$ and ensures that the generator translates the image into the desired output.

Reconstruction loss. A cycle-consistent loss [8] and a self-reconstruction loss [12] are applied to preserve the content information of the output image. The cycle-consistent loss is formulated as follows:

The self-reconstruction loss is formulated as follows:

Classification loss. A classification loss [11, 28] is applied to the generator to promote the attribute classification accuracy. The classification loss optimizes $D$ by real samples and optimizes $G$ by synthesized samples. By minimizing this objective, it forces the generated image to fall into the target domain. The classification loss function is denoted as follows:

Conditional feature matching loss. As feature matching loss [7, 39] can encourage the model to produce natural statistics at multiple scales, it has been popularly utilized to optimize the generator. Let $D_{i}(x,c_{org})$ denote the feature map extracted from the $i$-$th$ down-sampling layer of the discriminator D for input x and its corresponding label $c_{org}$, and ${\bar{D}}_{i}(x,c_{org})$ denote the global average pooling result of $D_{i}(x,c_{org})$. The feature matching loss can be formulated as follows:

where T denotes the amount of the selected feature maps in D.

Deriving benefit from the attribute adversary switch $D_{adv}$, the discriminator can converge the model more smoothly. Furthermore, we can utilize the domain-specified feature maps extracted from parallel adversary blocks $D_{adv_{i}}$ of $D_{adv}$ to optimize the generator. Specifically, we construct the domain-specified feature extractor named $D^{f}_{adv_{i}}$, by removing the last (prediction) layer from $D_{adv_{i}}$. Then, we use $D^{f}_{adv_{i}}$ to extract the domain-specified information and implement the feature-switching operation as follows:

With the fused domain-specified feature map obtained from the feature-switching operation, we finally propose a conditional feature matching loss as follows:

Domain-aware gradient penalty. The original GANs are powerful generative models but suffer from training instability. To stabilize the training process, a gradient penalty [40] is utilized in our network. Considering the input of the label vector in our discriminator structure, the loss function with the gradient penalty is modified as follows:

where $P_{\hat{x}}$ in the third part is a distribution randomly sampled from the real distribution $P_{x}$ and synthesized distribution $P_{\tilde{x}}$. The loss function of $loss_{adv_{G}}$ is modified as follows:

Full objective. As a result, to generate a plausible image with target attributes, we arrive at the objective function as follows:

where $\lambda_{cyc}$, $\lambda_{self}$, $\lambda_{c}$ and $\lambda_{cfm}$ are hyperparameters to control the importance of each loss.

Morph [41] is the largest publicly available longitudinal face database. Its Album 2 contains 55,134 images of 13,000 individuals with age labels ranging from 16 to 77 years. We separate the images into a training set with 50,020 images and a test set with 4,925 images. The images are separated into five groups with ages 11-20, 21-30, 31-40, 41-50, and 50+.

The Radboud Faces Database (RaFD) [42] is a face database with 8,040 pictures of 63 individuals displaying 8 emotional expressions. We only use the 4,824 frontal images in the database. Six individuals are randomly chosen as the test set. The images are separated into eight facial expression groups.

CelebA [43] is a large-scale face attributes dataset. It contains 202,599 face images, each with 40 attribute annotations. The dataset provides a standard list for dataset separation: 162,770 images for training, 19,868 images for validation and 19,961 images for testing. We use the given training set and validation set as our training set and the remaining 19,961 images as the test set.

All SwitchGAN models are trained through the strategy of Adam grad with $\beta_{1}=0.5$ and $\beta_{2}=0.999$. We take 300,000 steps of training for the Gated SwitchGAN models in Morph and CelebA, and 120,000 steps of training for the Gated SwitchGAN in RaFD. The parameters are set as $\lambda_{cyc}=$ $\lambda_{self}=5$ and $\lambda_{c}=$ $\lambda_{cfm}=1$ for the Gated SwitchGAN models training during one-to-many tasks. We set $\lambda_{cyc}=$ $\lambda_{self}=10$, $\lambda_{c}=2$ and $\lambda_{cfm}=1$ for the experiment of the Gated SwitchGAN in CelebA [43]. The initial learning rates for discriminator and generator are 0.0001, which linearly decays to 0 at the second half steps. The batch size for all datasets are 16. The Morph database images are detected by the MTCNN [44] and then aligned and cropped to $128\times 128$. For the RaFD and CelebA databases, the images are cropped to $256\times 256$, where the faces are centred, and resized to $128\times 128$. The same data augmentation that flips the images horizontally with a probability of 0.5 is applied to the datasets. We use the function in (18) and the hinge version [45] of GAN loss for training. For each training image, a randomly generated target label vector is used together with the image and associated attribute label vectors as input to the model for training. The generator is updated once when the discriminator is updated five times.

If the trained generator shows a good performance, the synthesized images should be photo-realistic and follow the same distribution as the real samples. The attribute accuracy and Frechet Inception Distance (FID) score [36, 37] have been popularly used to evaluate the quality of synthesized images. We also adopt them to qualitatively evaluate the quality of our results.

The translation accuracy, classified by a trained classifier, has been commonly utilized to evaluate image translation tasks. We employ it to compare the performances of different models. We utilize ResNet-18 [46] (a stable and parameter-insensitive model to report convincing results) and select the model with the best performance in the test set (MAE of 3.42 for age in Morph, accuracy of 99.54% for facial expression in RaFD, and average accuracy of 93.76% for five selected attributes in CelebA) for attribute classification. The synthesized images on the test set are tested by the classifiers. For example, when evaluating the model in Morph, each of the 4,925 test images is translated into the five age groups 11-20, 21-30, 31-40, 41-50, and 50+. As a result, 24,625 ($4,925\times 5$) synthesized images are tested by the trained age classifier to assess the translation performance.

The FID score is shown to correlate well with human judgement of visual quality by measuring the similarity between two datasets of images. It calculates the Wasserstein-2 distance between the generated images and the real images in the feature space of an Inception-v3 network. A lower FID score indicates that the distribution of generated images is more similar to that of real target images. Following the steps in [36], a feature vector is first extracted by a pretrained network [36, 37] for each attribute. We then compute FID between every generated image group and the target ground truth images and report the mean FID.

We calculate the translation accuracy and FID score for different baselines. The comparison is shown in Table I and Table II.

Morph. An age group classifier (ResNet-18) trained using the training set of Morph is used to classify the images generated by different GAN models. If the synthesized face of a target group is correctly classified by the classifier, we decide such a translation is a successful one. Table I shows the classification results for different models. While the classifier achieves 63.42%, 54.76% and 70.06% accuracy on the images generated by StarGAN, AttGAN and STGAN, respectively, it achieves higher accuracy on the images generated by the SwitchGAN (76.94%) and Gated SwitchGAN models (85.74%). A pretrained Inception-v3 model is utilized to compute the FID score. The FID scores are reported in Table I. Our Gated SwitchGAN model achieves the best FID score over the previous methods.

RaFD. As shown in Table I, the classifier achieves 92.53%, 81.68% and 87.30% accuracy on the images generated by StarGAN, AttGAN and STGAN, respectively, while our SwitchGAN and Gated SwitchGAN models achieve higher accuracy, i.e., 93.34% and 98.32%, respectively. A pretrained Inception-v3 model is again utilized to compute the FID score. Our Gated SwitchGAN model achieves the best FID score among different methods.

CelebA. Table II shows the classification results for different models. The classifiers achieve 89.23%, 78.24% and 91.01% average accuracy on the images generated by StarGAN, AttGAN and STGAN, respectively, while our SwitchGAN and Gated SwitchGAN achieve higher accuracy, i.e., 91.22% and 95.20%, respectively. A pretrained Inception-v3 model is utilized to compute the FID scores, which are reported in Table II. Our Gated SwitchGAN achieves the best average FID score, which demonstrates the superior ability of our Gated SwitchGAN model.

Our SwitchGAN and Gated SwitchGAN models are mainly compared with StarGAN, AttGAN and STGAN for age progression and regression, facial expression translation, and multiple facial attributes translation.

Morph. Age progression and regression aims to translate a facial image into the desired age group by transforming wrinkles, hairlines, hair colours, etc. Fig. 3 shows the results of different models. In Fig. 3, the example faces of a 41-year-old man translated by StarGAN, AttGAN, STGAN, SwitchGAN, and Gated SwitchGAN are visually compared. All the methods achieve reasonable age group translation. However, StarGAN, AttGAN and STGAN mainly adapt the wrinkle translation. The faces associated with ages of more than 50 years that are translated by our models present higher hair lines, much more white hair and beards to make them look older. The faces associated with ages less than 20 years that are translated by StarGAN, AttGAN and STGAN do not look young, while our SwitchGAN models regrow the black hair and shave the beard to make them look younger. Moreover, compared with SwitchGAN, the Gated SwitchGAN shows more significant changes in the beard and hair. The results clearly justify the usefulness of the SwitchGAN models.

RaFD. The facial expression translation aims to translate a facial expression into another. Fig. 4 shows the results of different models. In Fig. 4, the faces of different expressions translated by StarGAN, AttGAN, STGAN, SwitchGAN and Gated SwitchGAN are visually compared. As shown in the figure, only the mouth is appropriately changed for the surprised face translated by StarGAN, AttGAN and STGAN, while the eyes still look unsurprised. Our SwitchGAN models change both the eyes and mouth to make them look surprised. Compared to SwitchGAN, Gated SwitchGAN shows more appealing changes are made to the mouth and eyes. Similar translation can be found for the happy expression, i.e., the mouth generated by Gated SwitchGAN is more realistic.

CelebA. The attributes of black hair, blond hair, gender, moustache, and age are selected for experiments. The first two attributes aim to translate a facial image into an output with black hair or blond hair. The remaining three attributes aim to translate a facial image into an output with the opposite attributes, e.g., translating male into female, or female into male. Fig. 5 shows the results of different models. In Fig. 5, the example faces of a young woman with brown hair translated by StarGAN, AttGAN, STGAN, our SwitchGAN and Gated SwitchGAN are visually compared. The last three columns represent the joint translation of different attributes. As shown in the fourth column of Fig. 5, the young lady’s hair remains long after the translation performed by StarGAN, AttGAN and STGAN, while the SwitchGAN models shorten the lady’s hair to give it a more distinctively male appearance. As can be seen in the images of the 7th column, when the attributes of gender and age are translated at the same time, the images translated by StarGAN, AttGAN and STGAN merely manipulate the texture information of the input. The images translated by our SwitchGAN models shorten the hair and recede the hairline of the young lady to make her appear as if she were an old man. Moreover, as can be seen in images of the 6th and the last columns, the images translated by Gated SwitchGAN show less artefacts and more apparent results than SwitchGAN.

Given an input image, let $\alpha$ denote the weight value of target attributes $c_{trg}$, such that we can continuously control the attribute intensity of the output appearance by the function mentioned in Equation 7.

Morph. The age translation with different intensities, i.e., $\alpha$ = 0.25, 0.5, 0.75 and 1, are shown in Fig. 6, where the face of a man is translated into the age groups of younger than 20 years and older than 50 years by different methods. When the value of the attribute intensity is continuously changed, the images translated by StarGAN, AttGAN, and STGAN merely adapt the wrinkle translation. For the images translated by SwitchGAN, if a low value of $\alpha$ is given, artefacts and vacuity appear in the images due to the loss of encoded information in the latent space. The images translated by Gated SwitchGAN represent different age characteristics when different intensity values are imposed. For example, when the face of the input man is translated to the age group of less than 20 years old, the higher the intensity we impose, the more the black hair and the less the beard that appear in the output. When the input face is translated to the age group of more than 50 years old, the higher the intensity we impose, the more the beard, the whiter the hair and the higher the hair line that appear in the output.

RaFD. The facial expression translation with different intensities, i.e., $\alpha$ = 0.25, 0.5, 0.75 and 1, are shown in Fig. 7, where an angry face is translated into fearful and surprised expressions by different methods. When the value of the attribute intensity changes continuously, the images translated by StarGAN, AttGAN, and STGAN remain stable and obvious artefacts can be observed. For the images translated by SwitchGAN, when the value of $\alpha$ is small, artefacts appear in the images due to the loss of encoded information in the latent space. The images translated by Gated SwitchGAN are smoothly controlled by the intensity value and considerably fewer artefacts can be observed. For example, when the angry face is translated to the surprised expression, the higher the intensity we impose, the wider the mouth and eyes open. Similar results can also be found for the translation of the fearful facial expression.

CelebA. The multiple attribute translation with different intensities, i.e., $\alpha$ = 0.25, 0.5, 0.75 and 1, are shown in Fig. 8, where the face of a young woman is translated into various attributes by different methods. When the value of the attribute intensity is continuously changed, the images translated by StarGAN, AttGAN, and STGAN merely adapt the wrinkle translations. For the images translated by SwitchGAN and Gated SwitchGAN, they represent different attribute characteristics when different intensity values are imposed. For example, when the face of a young female is translated to an old man, the higher the intensity we impose, the shorter the hair and the higher the hair line that appear in the output. The images translated by Gated SwitchGAN are smoothly controlled by the intensity value and considerably fewer artefacts can be observed. While the intensities of translation for different attributes are set the same as the values in previous examples, we further show the translation results of two example faces in Figure 9, when the intensities of the attribute translation are set to four different values, i.e., 0, 0.25, 0.5 and 1. As shown in the figure, our Gated SwitchGAN can actually control the intensity of different attributes during face translation. Taking the face of the first lady as an example, when we are translating the female face to the male face, with an increasing value of $\alpha_{1}$, such that the face appears more similar to a male face (from left to right in the first row), we can actually decrease the value of $\alpha_{2}$ to reduce the amount of moustache. Similar results can also be achieved for other attributes such as hair colour and age.

We conduct ablation experiments to validate the effectiveness of the three components in our method: (1) self-reconstruction loss $\bm{L_{self}}$, (2) gate module, and (3) conditional feature-matching loss $\bm{L_{cfm}}$.

Self-reconstruction loss. We first improve the SwitchGAN by adding self-reconstruction loss defined in Equation 12 to the training process, which preserves the global structure and unmodified attributes of the input face. As shown in Table III, $\bm{L_{self}}$ can reasonably improve the FID/accuracy of SwitchGAN. We then denote the SwitchGAN with $\bm{L_{self}}$ as the baseline for the following analysis of our proposed components.

Gate module. We seek to validate that our gate module can effectively control the intensity of attribute translation and meanwhile improve the performance of image translation. We visually and quantitatively compare the performance of the baseline and baseline+Gate in Fig. 10. As shown in the figure, the example images of Morph and RaFD translated by the baseline remain stable for different intensities, and the faces are generally blurry, due to the loss of details. In contrast, our approach with the gate module (baseline+Gate) can well control the intensity of age, expression, hair/gender and age/gender translations and generates considerably fewer artefacts than the baseline.

Conditional feature matching loss. We now test the usefulness of the conditional feature matching loss $\bm{L_{cfm}}$, by combining it with gate modules and comparing with different settings. Fig. 11 shows the performance of our module with and without integrating $\bm{L_{cfm}}$. By exploiting domain information to constrain the generator, baseline+Gate+$\bm{L_{cfm}}$ effectively enhances the attribute manipulation ability and reduces artefacts in the generated images. As shown in Fig. 11 (a), the face less than 20 years old translated with $\bm{L_{cfm}}$ regrows more black hair and presents less moustache to make the face look young. In contrast, the face older than 50 translated with $\bm{L_{cfm}}$ presents whiter beard/hairs and less hair. Similarly, as shown in Fig. 11 (b) and Fig. 11 (c), the quality of faces translated with $\bm{L_{cfm}}$ for different expressions and attributes are generally higher and look more realistic than that translated without $\bm{L_{cfm}}$.

Quantitative analysis of different components. Table III lists the FID and Accuracy (Acc) of face images translated with different components, together with that of the baseline. The general feature-matching loss defined in Equation 15, $\bm{L_{fm}}$, is also included for comparison. As shown in the table, the gate module can generally improve both the accuracy and FID of baseline. Taking RaFD as an example, the gate module improves the FID/accuracy of the baseline from 27.45/95.23% to 19.14/97.22%. However, the inclusion of the $\bm{L_{fm}}$ loss function actually deteriorates the accuracy and increases the FID score of the baseline. It turns out that the combination of the gate and $\bm{L_{cfm}}$ achieves the best FID/accuracy among different settings.