LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions whilst Leveraging Human Perceptual Judgements ^††thanks: *Work done while at Affectiva

Deep learning [59] has engendered tremendous progress in automated facial analysis, with applications ranging from face verification [72, 22, 11] to expression classification [92]. However, building robust and accurate models that generalize effectively in-the-wild is still an open problem. A major part of it stems from training datasets failing to represent the “true” distribution of real world data [54, 68, 93, 6] (\egextreme lighting conditions [17, 73, 38]); or the training set may be non-uniformly distributed across classes, leading to the long-tail problem [63].

One way to mitigate the imbalance problem, shown to work in multiple domains [62, 9, 15, 100], is to introduce synthetic samples into the training set. Many approaches for generating synthetic data exist [18, 8, 42], none as successful as GANs [37] in generating realistic face images [16, 51, 52, 25]. Thus, in this work, we design a GAN model for synthesizing variations of an existing face image with the desired illumination and facial expression, while keeping the subject identity and other attributes constant. These synthetic images, when used as supplemental training data, can help build facial analysis systems that better generalize across variations in illumination and expressions.

One drawback of GAN-based face generation is the absence of an accurate and automated metric to judge the perceptual quality of synthesized images [57, 20]. In order to solve this problem, we also introduce a quality estimation model that can serve as a cheap but efficient proxy for human judgment while evaluating naturalness of synthetic face images. Instead of generating a single score for a distribution of synthetic images [107, 45, 79] or for image pairs [105, 75], our goal is to infer an image-quality score on a continuum for a single synthetic face image. With this in mind, we run an Amazon Mechanical Turk (AMT) experiment where turkers are instructed to score the naturalness of synthetic face images, generated using different 3D-model [9] and GAN-based [51, 52, 78, 2] synthesis approaches. We then build a feed forward CNN to learn representations from these images that map to their corresponding perceptual rating, using a margin based regression loss.

In addition to a traditional discriminator [37], this trained quality model is then used as an auxiliary discriminator in the synthesis framework, named LEGAN (Lighting-Expression GAN), that we propose in this paper. Instead of intertwining the two tasks [25], LEGAN decomposes the lighting and expression sub-spaces using a pair of hourglass networks (encoder-decoder) that generate transformation masks capturing the intensity changes required for target generation. The desired output image is then synthesized by a third hourglass network from these two masks.

To demonstrate the effectiveness of LEGAN, we qualitatively and quantitatively compare its synthesized images with those produced by two popular GAN based models [25, 26] using objective metrics like FID [45], LPIPS [105], SSIM [95] and face match score. We also conduct a human rater study to evaluate the perceptual quality of LEGAN’s images and the contribution of the quality based auxiliary discriminator towards hallucinating perceptually superior images. Finally, we show the efficacy of LEGAN as training data augmenter by improving the generalizability of face verification and expression recognition models.

Face Synthesis: Early approaches [18, 67] focused on stitching together similar looking facial patches from a gallery to synthesize a new face. Manipulating the facial shape using 3D models [49, 89, 63, 9] or deep features [27, 91] is another popular approach to generate new views. In recent times however researchers have pre-dominantly focused on using GANs [37] for synthesis, where an upsampling generator hallucinates faces from a noise vector, either randomly sampled from a distribution [37, 77, 51, 52] or interpreted from a different domain [71]. An existing face can also be encoded and then upsampled to obtain the desired attributes [48, 47, 96, 10, 70, 31, 35, 108, 23, 25].

Unlike these methods, we build a synthetic face quality estimation model by leveraging perceptual ratings of over 37,000 images generated using five different synthesis techniques [51, 52, 78, 1, 9]. Our quality model takes into account the variability in human judgements and generate a realism score for individual images rather than the whole set. We leverage this model as an auxiliary discriminator in the LEGAN framework for simultaneous lighting and facial expression manipulation. This novelty together with LEGAN’s feature disentanglement improves the naturalness of the hallucinated images. Additionally, we do not require external 3DMM information or latent vectors during training nor do we need to fine-tune our model during testing on input images [30, 58].

Our quality estimator model is trained with synthetic face images assembled and annotated in two sequential stages, as described below.

Stage I: We first generate 16,507 synthetic face images using the StyleGAN [52] generator. These images are then annotated by labelers using Amazon Mechanical Turk (AMT) on a scale of 0 - 10 for naturalness, where a 0 rating represents an unnatural image and 10 a hyper-realistic one. The images are then binned into two broad groups - ‘unnatural’ for AMT ratings between 0 - 5 and ‘natural’ for 5 - 10. We extract descriptors for each image from the ‘avg_pool’ layer of the ResNet50 [44] model, pre-trained on VGGFace2 [22] and train a linear SVM [28] with the extracted features of around 12,000 images from this dataset and use the rest for parameter tuning. Post training, we use this SVM as a rough estimator of naturalness.

Stage II: In this stage, we perform the same AMT experiment again with a larger set of synthetic face images, collected from the following datasets:
1. FaceForensics++[78] - we randomly sample 1000 frames from this dataset consisting of 1000 video sequences that have been manipulated with four automated face manipulation methods.
2. DeepFake[2, 3] - we use sampled frames from 620 manipulated videos of 43 actors from [1].
3. ProGAN [51] - we generate 10,000 synthetic face images of non-existent subjects by training NVIDIA’s progressively growing GAN model on the CelebA-HQ dataset [51].
4. StyleGAN [52] - we extract 100,000 hyper-realistic face images of non-existent subjects generated using the StyleGAN model that were pre-filtered for quality [4].
5. Notre Dame Synthetic Face Dataset [9] - we randomly sample 163,000 face images, from the available 2M, of synthetic subjects generated using ‘best-fitting’ 3D models.

To focus on near-frontal faces, we remove images with yaw over 15$\degree$ in either direction, estimated using [43]. Since gender information is absent in most of the above datasets, we group the synthetic images using gender predictions from a pre-trained model [60]. Our trained SVM (from Stage I) is also used to rate the coarse naturalness of the collected images, using their ResNet50 features. We ensure balance in our synthetic dataset by sampling evenly from the natural and unnatural sets, as estimated by the SVM, and the perceived gender classes. To focus solely on the facial region, the pixels outside the convex hull formed by the facial landmarks, estimated using [21], of an image are masked. After the gender, facial yaw and naturalness based filtering, and the pre-processing step, we end up with 37,267 synthetic face images¹¹1Available here: https://github.com/Affectiva/LEGAN_Perceptual_Dataset for our second AMT experiment.

Again, we ask Turkers to rate each image for naturalness on a scale of 0 - 10. Each image is shown to a Turker for 60 seconds to allow them time to make proper judgement even with slow network connection. We divide the full set of images into 72 batches such that each batch gets separately rated by 3 different Turkers. Post crowd-sourcing, we compute the mean ($\mu$) and standard deviation ($\sigma$) from the 3 scores and assign them as naturalness label for an image.

To train the quality estimation model, we use 80% of this annotated data and the rest for validation and testing. For augmentation, we only mirror the images as other techniques like translation, rotation and scaling drastically change their appearance compared to what the Turkers examined. Our model downsamples an input image using a set of strided convolution layers with Leaky ReLU [99] activation followed by two fully connected layers with linear activation and outputs a single realness scoring. Since both $\mu$ and $\sigma$ are passed as image labels, we try to capture the inconsistency in the AMT ratings (i.e. the subjective nature of human perception) by formulating a margin based loss for training. The model weights are tuned such that its prediction is within an acceptable margin, set to $\sigma$, from the mean rating $\mu$ of the image. The loss $L_{N}$ can be represented as:

$$L_{N}=\frac{1}{n}\sum_{i=1}^{n}\left\|\sigma-\left\|\mu-Q(I_{i})\right\|_{2}^{2}\right\|_{2}^{2}$$

(1)

where $n$ is the batch size, $Q(I_{i})$ is the model prediction for the $i$-th image in the batch. Since the model is trained on the mean rating $\mu$ (regression) as the target rather than fixed classes (classification), $L_{N}$ pushes the model predictions towards the confidence margin $\sigma$ from $\mu$.

We build LEGAN as a lightweight network that works with unpaired data, similar to the StarGAN family, to focus more on assessing the effect of our quality estimation model ($Q$) as a perceptual loss. Unlike [30, 58], LEGAN does not require additional networks to regress 3DMM parameters or fine-tuning during inference. We describe the architecture and objective functions of LEGAN in this section, an overview of which can be seen in Figure 3.

For the unseen, in-the-wild datasets, the performance of LEGAN is mostly superior to the other models, as shown in Tables 2 and 3 respectively. Due to the non-uniform nature of the data, especially the facial pose, most of the metrics deteriorate from Table 1. However, the boost in quality score overall suggests the high quality images from AFLW [56] and especially CelebA [61] to be visually more appealing than images from MultiPIE [39]. Some sample results have been shared in Figure 6.

For expression classification, we use a modified version of the AU-classification model from [32] (Leaky ReLU [99] and Dropout added) and manually annotated AffectNet [66] images for the (‘Neutral’, ‘Happy’, ‘Surprise’, ‘Disgust’) classes, as these 4 expressions overlap with MultiPIE [39]. The classification model is trained with 204,325 face images from AffectNet’s training split, which is highly skewed towards the ‘Happy’ class (59%) and has very few images for the ‘Surprise’ (6.2%) and ‘Disgust’ (1.6%) expressions. To balance the training distribution, we populate each sparse class with synthetic images generated by LEGAN from real images belonging to any of the other 3 classes. We use the original and augmented (balanced) data separately to train two versions of the model for expression classification. As there is no test split, we use the 2,000 validation images for testing, as done in other works [92]. We find the synthetic images, when used in training, to substantially improve test performance especially for the previously under-represented ‘Surprise’ and ‘Disgust’ classes (Table 5). This further validates the realism of the expressions generated by LEGAN.

We propose LEGAN, a GAN framework for performing many-to-many joint manipulation of lighting and expressions of an existing face image without requiring paired training data. Instead of translating the image representations in an entangled feature space like [25], LEGAN estimates transformation maps in the decomposed lighting and expression sub-spaces before combining them to get the desired output image. To enhance the perceptual quality of the synthetic images, we directly integrate a quality estimation model into LEGAN’s pipeline as an auxiliary discriminator. This quality estimation model, built with synthetic face images from different methods [51, 52, 78, 2, 9] and their crowd-sourced naturalness ratings, is trained using a margin based regression loss to capture the subjective nature of human judgement. The usefulness of LEGAN’s feature disentangling towards synthesis quality is shown by objective comparison [45, 105, 95] of its synthesized images to that of the other popular GAN models like StarGAN [25] and StarGAN-v2 [26]. These experiments also highlight the usefulness of the proposed quality estimator in LEGAN and StarGAN w/ $L_{qual}$, specifically comparing the latter to vanilla StarGAN.

As a potential application, we use LEGAN as training data augmenter for face verification on the IJB-B [97] and LFW [46] datasets and for facial expression classification on the AffectNet [66] dataset (Tables 4 and 5). An improvement in the verification scores in both datasets suggests LEGAN can enhance the intra-class variance while preserving subject identity. The boost in expression recognition performance validates the realism of the LEGAN generated facial expressions. The output quality, however, could be further improved when translating from an intense expression to another. We plan to address this by - (1) using attention masks in our encoder modules, and (2) building translation pathways of facial action units [88] while going from one expression to another. Another future goal is to incorporate a temporal component in LEGAN for synthesizing a sequence of coherent frames.

We share details of the architecture of our quality estimator $Q$ in Table 6. The fully connected layers in $Q$ are denoted as ‘fc’ while each convolution layer, represented as ‘conv’, is followed by Leaky ReLU [99] activation with a slope of 0.01.

In this section, we share the distribution of the naturalness ratings that we collected from the Amazon Mechanical Turk (AMT) experiment (Stage II). To do this, we average the perceptual rating for each synthetic face image from its three scores and increment the count of a particular bin in [(0 - 1), (1 - 2), … , (8 - 9), (9 - 10)] based on the mean score. As described in Section 3 of the main paper, we design the AMT task such that a mean rating between 0 and 5 suggests the synthetic image to look ‘unnatural’ while a score between 5 and 10 advocates for its naturalness. As can be seen in Figure 7, majority of the synthetic images used in our study generates a mean score that falls on the ‘natural’ side, validating their realism. When used to train our quality estimation model $Q$, these images tune its weights to look for the same perceptual features in other images while rating their naturalness.

To further check the overall perceptual quality of each of the different synthesis approaches used in our study [51, 52, 2, 78, 9], we separately find the mean rating for each synthetic face image generated by that method, depicted in Figure 8. It comes as no surprise for the StyleGAN [52] images to rank the highest, with a mean score over 7, as its face images were pre-filtered for quality [4]. The other four approaches perform roughly the same, generating a mean score that falls between 6 and 7.

Our loss uses the L2 norm between the predicted quality ($p$) and mean label ($\mu$) and then computes a second L2 norm between this distance and the standard deviation ($\sigma$). $\sigma$ acts as a margin in this case. If we consider $\mu$ as the center of a circle with radius of $\sigma$, then our loss tries to push $p$ towards the boundary to fully capture the subjectiveness of human perception. We also tried a hinge version of this loss: $\max\left(0,\left(\left\|\mu-p\right\|_{2}^{2}-\sigma\right)\right)$. This function penalizes $p$ falling outside the permissible circle while allowing it to lie anywhere within it. When $\sigma$ is low, both functions act similarly. We found the quality estimation model’s ($Q$) predictions to be less stochastic when trained with the margin loss than the hinge. On a held-out test set, both losses performed similarly with only 0.2% difference in regression accuracy. Experimental results with LEGAN, and especially StarGAN trained using $Q$ (Tables 1, 2, 3 in the main text), underpin the efficiency of the margin loss in comprehending naturalness. The improvements in perceptual quality, as demonstrated by LPIPS and FID, further justify its validity as a good objective for training $Q$.

As discussed in Section 3 of the main text, we hold out 10% of the crowd-sourced data (3,727 face images) for testing our quality estimation model $Q$ post training. Since $Q$ never encountered these images during training, we use them to evaluate the effectiveness of our model. We separately compute the mean naturalness score for each synthesis approach used in our study and compare this value with the average quality score as predicted by $Q$. The results can be seen in Figure 9. Overall, our model predicts the naturalness score for each synthesis method with a high degree of certainty. Some qualitative results can also be seen in Figure 10.

In this section, we list the different layers in the generator $G$ and discriminator $D$ of LEGAN. Since $G$ is composed of three hourglass networks, we separately describe their architecture in Tables 7, 8 and 9 respectively. The convolution layers, residual blocks and pixel shuffling layers are indicated as ‘conv’, ‘RB’, and ‘PS’ respectively in the tables. After each of ‘conv’ and ‘PS’ layer in an hourglass, we use ReLU activation and instance normalization [90], except for the last ‘conv’ layer where a tanh activation is used [77, 80]. The description of $D$ can be found in Table 10. Similar to $Q$, each convolution layer is followed by Leaky ReLU [99] activation with a slope of 0.01 in $D$, except for the final two convolution layers that output the realness matrix $D_{src}$ and the classification map $D_{cls}$.

To analyze the contribution of each loss component on synthesis quality, we prepare 5 different versions of LEGAN by removing (feature disentanglement, $L_{adv}$, $L_{cls}$, $L_{rec}$, $L_{qual}$, and $L_{id}$) from $G$ while keeping everything else the same. The qualitative and quantitative results, produced using MultiPIE [39] test data, are shown in Figure 11 and Table 11 respectively. For the quantitative results, the output image is compared with the corresponding target image in MultiPIE, and not the source image (\ieinput).

As expected, we find $L_{adv}$ to be crucial for realistic hallucinations, in absence of which the model generates non-translated images totally outside the manifold of real images. The disentanglement of the lighting and expression via LEGAN’s hourglass pair allows the model to independently generate transformation masks which in turn synthesize more realistic hallucinations. Without the disentanglement, the model synthesizes face images with pale-ish skin color and suppressed expressions. When $L_{cls}$ is removed, LEGAN outputs the input image back as the target attributes are not checked by $D$ anymore. Since the input image is returned back by the model, it generates a high face matching and mean quality score (Table 11, third row). When the reconstruction error $L_{rec}$ is plugged off the output images lie somewhere in the middle, between the input and target expressions, suggesting the contribution of the loss in smooth translation of the pixels. Removing $L_{qual}$ and $L_{id}$ deteriorates the overall naturalness, with artifacts manifesting in the eye and mouth regions. As expected, the overall best metrics are obtained when the full LEGAN model with all the loss components is utilized.

To check the effect of the different upsampling approaches on hallucination quality, we separately apply bilinear interpolation, transposed convolution [103] and pixel shuffling [82] on the decoder module of the three hourglass networks in LEGAN’s generator $G$. While the upsampled pixels are interpolated based on the original pixel in the first approach, the other two approaches explicitly learn the possible intensity during upsampling. More specifically, pixel shuffling blocks learn the intensity for the pixels in the fractional indices of the original image (i.e. the upsampled indices) by using a set convolution channels and have been shown to generate sharper results than transposed convolutions. Unsurprisingly, it generates the best quantitative results by outperforming the other two upsampling approaches on 3 out of the 5 objective metrics, as shown in Table 12. Hence we use pixel shuffling blocks in our final implementation of LEGAN.

However, as can be seen in Figure 12, the expression and lighting transformation masks $M_{e}$ and $M_{l}$ are more meaningful when interpolated rather than explicitly learned. This interpolation leads to a smoother flow of upsampled pixels with facial features and their transformations visibly more noticeable compared to transposed convolutions and pixel shuffling.

As discussed in the main text, we set the value of the hyper-parameter $q$ = 8 for computing the quality loss $L_{qual}$. We arrive at this specific value after experimenting with different possible values. Since $q$ acts as a target for perceptual quality while estimating $L_{qual}$ during the forward pass, it can typically range from 5 (realistic) to 10 (hyper-realistic). We set $q$ to all possible integral values between 5 and 10 for evaluating the synthesis results both qualitatively (Figure 13) and quantitatively (Table 13).

As can be seen, when $q$ is set to 8, LEGAN generates more stable images with much less artifacts compared to other values of $q$. Also, the synthesized expressions are visibly more noticeable for this value of $q$ (Figure 13, top row). When evaluated quantitatively, images generated by LEGAN with $q$ = 8 garner the best score for 4 out of 5 objective metrics. This is interesting as setting $q$ = 10 (and not 8) should ideally generate hyper-realistic images and consequently produce the best quantitative scores. We attribute this behavior of LEGAN to the naturalness distribution of the images used to train our quality model $Q$. Since majority of these images fell in the (7-8) and (8-9) bins, and very few in (9-10) (as shown in Figure 7), $Q$’s representations are aligned to this target. As a result, $Q$ tends to rate hyper-realistic face images (i.e. images with mean naturalness rating between 8 - 10) with a score around 8. Such an example can be seen in the rightmost column of the first row in Figure 10, where $Q$ rates a hyper-realistic StyleGAN generated image [52] as 8.3. Thus, setting $q$ = 8 for $L_{qual}$ computation (using trained $Q$’s weights) during LEGAN training produces the optimal results.

In this section, we share more details about the interface used for our perceptual study. As shown in Figure 14, we ask the raters to pick the image that best matches a target expression and lighting condition. To provide a basis for making judgement, we also share a real image of the same subject with neutral expression and bright lighting condition. However, this is not necessarily the input to the synthesis models for the target expression and lighting generation, as we want to estimate how these models do when the input image has more extreme expressions and lighting conditions. The image order is also randomized to eliminate any bias.

Although LEGAN is trained on just frontal face images acquired in a controlled setting, it can still generate realistic new views even for non-frontal images with a variety of expressions, as shown in Figures 16 and 17. However, as with any synthesis model, LEGAN also has its limitations. In majority of the cases where LEGAN fails to synthesize a realistic image, the input expression is irregular with non-frontal head pose or occlusion, as can be seen in Figure 15. As a result, LEGAN fails to generalize and synthesizes images with incomplete translations or very little pixel manipulations. One way to mitigate this problem is to extend both our quality model and LEGAN to non-frontal facial poses and occlusions by introducing randomly posed face images during training.

In this section, we share more qualitative results generated by LEGAN on unconstrained data from the AFLW [56] and CelebA [61] datasets in Figures 16 and 17 respectively. The randomly selected input images vary in ethnicity, gender, color composition, resolution, lighting, expression and facial pose. In order to judge LEGAN’s generalizability, we only train the model on 33k frontal face images from MultiPIE [39] and do not fine tune it on any other dataset.

For the colorization augmentation network, we use a generator architecture similar to the one used in [10] for the 128$\times$128$\times$3 resolution. The generator is an encoder-decoder with skip connections connecting the encoder and decoder layers, and the discriminator is the popular CASIANet [101] architecture. Details about the generator layers can be found in Table 14.

We train two separate versions of the colorization network with randomly selected 10,000 face images from the UMDFaces [12] and FFHQ [51] datasets. These two trained generators can then be used to augment LEGAN’s training set by randomly recoloring the MultiPIE [39] images from the training split. Such an example has been shared in Figure 18.