Inverting Generative Adversarial Renderer for Face Reconstruction

Face modeling aims at using mathematical formulas to generate locations of vertices of a face mesh. Since the introduction of the Basel Face Model [6] that used a linear combination of Gaussian distributed coefficients of a face, multiple methods have been proposed, including blender shapes [9], skeleton animations [24], B-splines [28], deformation based method [18]. Nonlinear models have also been proposed [43, 42]. However, all of the face models can only formulate the facial regions and lack the modeling of the face accessories and hairs. Some efforts have been made to model various face attributes, including hair [16], mouth [30], back of the head [33, 23], \etc. However, they are generally more complicated and require much more computation to fit or render a face image.

3D face reconstruction is the inverse process of recovering the face shapes from a monocular image. The methods can be generally categorized into two types, optimization-based methods and learning-based methods. The optimization-based methods generally try to invert the rendering or imaging process of face images by optimizing a cost function for each input image [46, 33]. Various loss functions were explored, including the re-projection loss of the detected 2D landmarks from the rendered 3D mesh [5], the photometric loss between the rendered image by a differentiable renderer and the original image [22], etc. [13] introduces a pretrained generative adversarial network to fit the texture UV map. And thanks to the differential renderers, [13] adopts photo-metric loss, as well as recognition loss to further enhance the texture and geometry quality. [49] proposes a ReDA Rasterizer for more soft and realistic rendering, and a free-form deformation layer with as-rigid-as-possible constraint to reconstruct an accurate face model.

Recently, deep learning-based methods have presented promising performance. Zhu et al.[50] proposed to directly regress the face parameters from the input face image. Starting from the cascaded face-parameter regressor [8], there are methods focusing on designing supervisions on representing the final reconstructed mesh. Video-based methods introduced additional constraints to regularize the reconstruction results. [14] assumed that the reconstructed face images from multiple frames of a video should maintain the same face identity and similar textures. Face recognition models and perceptual loss are adopted to minimize the differences between the feature maps of the multi-view images to better regularize the reconstructed face shapes.

Many 3D-based methods have been proposed to generate face images [12, 10]. Besides methods using a statistic face model to explicitly calculate the 3D mesh, face image auto-encoder has been popular since the introduction of generative adversarial network [15]. High-resolution images can be gradually generated from a low-resolution to high-resolution manner. StyleGANs [20, 21] introduced to model the latent code for image generation as the input Batch Normalization parameters. There are also methods that try to control the generated images with the input latent code [29] by adding a classifier to restrict the output patterns. However, the controllable properties of the generated face images are only limited to the modification of single neurons [37]. Some feature disentanglement networks [1] for image generation have also been proposed. However, they cannot generate realistic images conditioned on 3D information. [32] tried to disentangle the process of 2D face image generation into 3D mid-level feature generation and 3D-to-2D feature projection and generation. Other than how to generate more realistic face images, given a trained GAN model, how to effectively invert the GAN to obtain the corresponding latent code is also of importance. [4] studied how to effectively invert a GAN model, since the inverting of the GAN model might also be stuck at local minima. A regression network is trained with the generator network predicting a good initialization for inverting the GAN from the input image to recover the corresponding latent code.

In this section, we introduce the proposed Generative Adversarial Renderer $G$, which takes in a normal map $n$ and latent code $z$ and outputs a corresponding rendered image $I_{out}$.

The input feature map $f_{cxy}$ is convoluted to $f^{\prime}_{lxy}$ with the modulated kernels

The normal map is used to further modulate the feature map $f^{\prime}$ in the Normal Injection Module (NIM). However, instead of regularizing the channel dimension, the normal map is used for regularizing the spatial dimension:

where $f^{\prime\prime}_{lxy}$ is the feature maps after modulation by the input normal map, and $n_{xy}$ denotes the injecting normal values from the facial normal map $n$ from the spatial location $(x,y)$.

The feature maps $f^{\prime\prime}$ are added with a learned bias $b$ and a random Gaussian noise map $\varepsilon$ before sent to the next block. The insight here, as discussed in the original StyleGAN [20], is that the small dimension of the latent code $z$ cannot fully express all the details of a face image. The extra noise is therefore needed to properly model the extra information.

Since the proposed neural renderer targets the face geometry reconstruction, we would like the face shape to be neatly controlled by the input face normal map but not the latent code $z$. Therefore, we introduce two contrastive losses to facilitate the disentanglement and to strengthen the controllability of the input normal map (see the right side of Figure 2 (b)).

On the one hand, we would like the face structure to be fully controlled by the input normal map. We construct pairs training data, in which the paired data have identical normal maps $n_{1}=n_{2}$ but their latent codes $z_{1}$ and $z_{2}$ are different. In addition, we further adopt a facial landmark detector as a measurement of the structure consistency. Facial landmarks are essential complements to the normal maps, since normal maps focus on general structures of the surface, while the landmarks pay more attention to the facial edges and boundaries. Specifically, the facial landmark consistency loss $\mathcal{L}_{ldmk}$ is formulated as

where $L$ is the pre-trained landmark detector, $z_{1}$, $z_{2}$ are different latent codes fed into the renderer $G$.

On the other hand, we require to retain the identity of the same person when his/her pose $\theta$ and expression $\beta$ vary, while the latent code and the normal map remains the same. we use a face recognition network’s output features to measure whether the two output images correspond to a same person, or known as the identity loss $\mathcal{L}_{\rm id}$,

where $R$ is the pre-trained and fixed face recognition network, ${\theta}_{1}$ and ${\theta}_{2}$ denote the different poses fed into the renderer. The human shape $\alpha$ and the latent $z$ remain the same so that we can obtain a same person with the similar facial textures from different view points.

The facial landmark consistency loss $\mathcal{L}_{\rm ldmk}$, together with the identity loss $\mathcal{L}_{\rm id}$, form the contrastive loss, which is designed to further disentangle the input normal map and the latent code $z$.

In this section, we introduce our optimization-based framework for face geometry reconstruction, aided by the proposed neural renderer.

We first initialize these parameters, and render them with the trained and fixed neural renderer to obtain the rendered image, which is then used to calculate the loss with $I_{\rm t}$. The face geometry reconstruction loss $\mathcal{L}_{f}$ is defined as

where $G$ represents the fixed generative adversarial renderer, $\tilde{n}$ is the normal map calculated from geometry coefficients ($\alpha,\beta,\theta$), $F_{i}$ is the $i$th layer feature map by an ImageNet-pretrained VGG network to model the perceptual loss, and $\lambda_{n}$ weights the regularization term on the random noise. By minimizing the above face geometry reconstruction loss $\mathcal{L}_{f}$, we can obtain optimized geometry parameters $\alpha$, $\beta$ and $\theta$.

The renderer inverting network $V$ and the generative adversarial renderer $G$ are trained in a coupled way, where the output of the neural renderer (the generated face image $I_{\rm out}$) is input into $V$ to convert the image back to a latent code $\hat{z}$. Ideally, the reconstructed latent code $\hat{z}$ should be close to the input latent code $z$.

We design the structure of the inverting network $V$ symmetric to the GAR $G$, which is more theoretically interpretable, with Conv Layers converted to Deconv Layers, and the statistical mean and variance of feature maps are used to estimate the latent code $z$ with an MLP that has same depth and channels for each layers as the style transfer MLP. The resulting feature maps of the inverting network should be of the same spatial size as those in the corresponding layer of the neural renderer. The reconstructed latent code $\hat{z}$ is estimated based on the concatenation of each layers’ statistic means and standard variances followed by an MLP. The loss function for training the renderer inverting network is therefore

where $I_{\rm out}=G(n,z,{\rm noise})$ is the generated face image from the neural renderer, $R_{i}$ and $G_{i}$ denotes the feature map from the $i$th layer of $R$ and $G$ respectively, $\mu,\sigma$ is the mean and variance of the feature map in $R$.

where $v^{\prime}_{i}=v_{i}+W^{s}_{i}s+W^{e}_{i}e$ is the location of the vertex. $\alpha,\beta$ are the 3DMM parameters of shape and expressions, $W^{s}_{i},W^{e}_{i}$ represent the linear bases of shape and expression in 3DMM model, $k,R,t$ are the pose parameters, and $v_{i}$ is the $i$-th vertex mean position.

The renderer inverting network, together with the 3DMM parameter pre-solving, provides a good initialization for the optimization. The optimization of the 3D face shape can then be performed by minimizing the photo-metric loss between the rendered image and the input image.

The proposed face geometry reconstruction method, together with the Generative Adversarial Renderer, provides an effective approach for face editing. Specifically, given an source image $I_{s}$, the corresponding 3DMM geometry parameters $(\alpha_{s}$, $\beta_{s}$, $\theta_{s})$ can be recovered by our optimization-based framework, as well as the latent code $z_{s}$ and $\varepsilon_{s}$. All or a portion of the 3DMM parameters can be chosen for editing. By rendering the edited parameters with the Generative Adversarial Renderer, we could obtain the corresponding edited face image $I_{t}$ with realistic details. Even though the general idea is similar, the edited faces are much more appealing, owing to the proposed novel renderer.

Our algorithm is trained in a self-supervised manner, and requires no image annotated with 3DMM parameters. To train and test our proposed algorithm’s performance, the following datasets are adopted.

For all of our experiments, a given face image is aligned to our fixed template using 68 landmark locations [47, 34, 39] detected by an hourglass 2D landmark detection [31]. For the normal map estimation, we adopt SFSNet [35].

During the GAR training process, we optimize parameters using Adam solver with a 0.01 learning rate. We set our balancing factors as $\lambda_{\rm n}=2.0$, $\lambda_{\rm ldmk}=2.0$, $\lambda_{\rm id}=1.0$, $\lambda_{\rm adv}=1.0$.

For the evaluation of the Florence dataset, we uniformly sample $5$ frames of each video, and calculate the average of the vertex coordinates for evaluation following [13]. The evaluation metric for face reconstruction is the point-to-plane error of each vertex of reconstructed 3DMM meshes to the ground-truth scanned meshes.

The examples of generated images by our approach can be seen in Figure 4, which shows that the proposed Generative Adversarial Renderer can generate face images with much higher visual quality than conventional graphics-based renderers, where random hairstyle, glasses, and other attributes are well generated. To quantitatively analyze the image quality of the generated images, we randomly generated $50,000$ images with Progressive GAN [19], StyleGAN [20] and our GAR, and calculate the Frechet Inception Distance (FID) [38] between the generated images and the real image datasets. The results are presented in Table 1. It demonstrates that even though our GAR is trained with more conditioning constraints and the controllability of the network is promoted, our output images are still competitive to those of the unconditional StyleGAN in terms of image quality and diversity.

For the input conditioning normal maps in Fig. 4, we can see that the face geometry is well maintained during the rendering. This indicates that our GAR follows the important role of a renderer, faithfully converting an input normal map and a latent code to a corresponding face image.

In this section, we present the results of an ablation study on investigating different components of our proposed face reconstruction framework.

As mentioned in Section 3.2, our method is capable of face editing. As shown in Figure 6, by editing the 3DMM parameters, the rendered image would present corresponding attributes. In the first row, the pose is set to turn from left to right, so the faces in the rendered images gradually change while the identity and the expression maintain unchanged. In the second and third rows, we present the resulting images of editing facial expressions. For results please refer to the supplementary materials.