High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

Our work builds on the multi-view-consistent image synthesis ability of 3D-GAN. The state-of-the-art method, EG3D [5], proposes an expressive hybrid explicit-implicit network based on a 2D CNN generator and neural rendering. For a 3D-aware generation, each random sampled latent code is first sent to a pose-conditioned StyleGAN2 generator to learn a tri-plane 3D representation. It then learns a neural radiance field based on it and generates a low-dimensional raw image by volume rendering. Finally, a super resolution module is adopted to generate high-resolution result. After being trained on FFHQ [20], EG3D can be used for unsupervised generation of multi-view-consistent facial images. Its latent space constitutes a generative prior for facial images with consistent 3D geometry.

A closer look at EG3D latent space. EG3D [5] is an unconditioned generation network and doesn’t provide any controls over identity or expression. For generation w.r.t. to a specific facial image, one can invert the input image back into the latent space. Given the inverted latent code, novel view synthesis can be realized by changing the camera pose during generation. However, recovering the 3D geometry from a single image is an ill-posed problem. Directly inverting facial images to the generic latent space of EG3D cannot faithfully reconstruct the specific characteristics of that individual, an example is given in Figure 3. In addition, it doesn’t support face reenactment.

We aim to reconstruct 3D-aware animatable facial avatar based on a monocular video. To take advantage of the rich generative prior of 3D-GAN as well as maintain the personalized characteristics, we propose to learn a personalized generative prior. Our overall framework is given in Figure 2. Specifically, we define the personalized generative prior as a local, low-dimensional, and smooth subspace in the latent space. The low-dimensional property is expected as the facial images of a specific identity share common properties.

We consider $\mathcal{W}+$ space of 3D-GAN and assign a learnable personalized basis with $k$ vectors as $\mathbf{B}=[\mathbf{b}_{1},\cdots,\mathbf{b}_{k}]\in\mathcal{R}^{k\times d}$ in the $\mathcal{W}+$ space. The subspace that spanned by $\mathbf{B}$ is defined by

$$\textstyle\mathcal{S}_{\mathbf{B}}=\left\{\bm{w}|\bm{w}=\sum_{i=1}^{k}\bm{\alpha}_{i}\bm{b}_{i},\bm{\alpha}\in\mathcal{R}^{1\times k}\right\},$$

(1)

where $\bm{\alpha}$ represents the coefficient w.r.t. basis vectors. Rather than directly inversing each facial image $\bm{x}$ into the high-dimensional $\mathcal{W}+$ space, we project $\bm{x}$ into the low-dimensional subspace $\mathcal{S}_{\mathbf{B}}$, by learning an encoder $\bm{f}:\bm{x}\rightarrow\mathcal{R}^{1\times k}$ to regress the coefficient $\bm{\alpha}$ of $\bm{x}$. Finally, $\bm{w}=\bm{f}(\bm{x})\cdot\mathbf{B}$ is sent to 3D-GAN generator for free view synthesis, as $\hat{\bm{x}}=\mathcal{G}(\bm{f}(\bm{x}),\mathbf{B},\bm{p})$, where $\mathcal{G}$ is the 3D-GAN generator and $\bm{p}$ is the camera pose used for rendering.

Given a monocular face video $\bm{X}=\{\bm{X}^{t}\}^{T}_{t=1}\in\mathcal{R}^{T\times H\times W\times 3}$ of a individual with $T$ frames, each frame of which contains different expressions and poses. The encoder $\bm{f}$ and the basis $\mathbf{B}$ are jointly optimized for a faithfully reconstruction of $\bm{X}$. Let $\hat{\bm{X}}^{t}=\mathcal{G}(\bm{f}(\bm{X}^{t}),\mathbf{B},\bm{p}^{t})$ and $\bm{p}^{t}$ is the camera pose extracted from $\bm{X}^{t}$, we calculate the $\mathcal{L}_{2}$ loss and LPIPS loss [51] between $\bm{X}^{t}$ and $\bm{X}^{t}$,

$$\textstyle\mathcal{L}=\sum_{t=1}^{T}\mathcal{L}_{2}(\bm{X}^{t},\hat{\bm{X}}^{t})+\lambda_{lpips}\mathcal{L}_{lpips}(\bm{X}^{t},\hat{\bm{X}}^{t}).$$

(2)

During training, we further constraint the basis vectors to be orthogonal to each other. The orthogonal constraint can largely boost the disentanglement of the basis. We provide an visualization of the basis in Figure 6. For the generator $\mathcal{G}$, we adopt the pretrain model [5] that learned on FFHQ. Similar to PTI [39], during training we slightly modify the generator to better maintain the personalized characteristics. We use a two-stage training strategy: 1) freezing the parameters of the generator and updating the encoder $\bm{f}$ and the basis $\mathbf{B}$, and 2) turning on the gradient of the generator to adapt it to the personalized subspace.

Generalize to testing frames. The local and low-dimensional personalized subspace provides a good generalization to facial variations beyond the training frames. After training, the encoder $\bm{f}$ can be directly applied to testing frames with different facial expressions to generate high-fidelity facial avatar reconstruction.

In the above section, we learn an encoder to project each facial image into the personalized latent space, to provide faithful 3D-aware reconstruction. Indeed, the input signal is not limited to facial images. Here, we provide two realizations with 3DMM coefficients and audio signals as input information, respectively. After training, they can be used for 3DMM or audio driven 3D-aware face reenactment.

3DMM-driven face reenactment: we extract 3DMM expression coefficient $\bm{\beta}\in\mathcal{R}^{76}$ from each image, forming training pairs of $\{(\bm{X}^{t},\bm{\beta}^{t})\}_{t=1}^{T}$. The coefficient $\bm{\alpha}$ is learned by $\bm{\alpha}=\bm{f}_{e}(\bm{\beta})$. Audio-driven face reenactment: following [17], we use DeepSpeech [1] to extract a 29-dimensional feature for each 20ms audio clip. To eliminate the noisy signals from raw input, we concatenate the features of sixteen neighboring audio clips, resulting in $\bm{\delta}\in\mathbb{R}^{16\times 29}$ for the audio feature of the current frame. We then project $\bm{\delta}$ into the latent space by $\bm{\alpha}=\bm{f}_{a}(\bm{\delta})$.

The realization of $\bm{f}_{e}$ and $\bm{f}_{a}$ are given in the supplementary materials. We follow the training process in Sec.3.3 for the learning of $\bm{f}_{e}$, $\bm{f}_{a}$ and their corresponding basis. We compare to existing 3DMM and audio-driven face reenactment in Sec.4.3 and Sec.4.4.

Data preprocessing. The training video needs to be processed before face modeling. For each frame of the video, we use an off-the-shelf pose estimator [10] to estimate its corresponding camera intrinsic and extrinsic matrices as the input to EG3D generator. The flattened $4\times 4$ camera extrinsic matrix and flattened $3\times 3$ camera intrinsic matrix are concatenated into a 25-dimensional vector as the camera input to EG3D generator. Then, we extract the appropriately-sized crops from each frame and resize each cropped image to the resolution of $512\times 512$.

Training details. For each video, we train the network 200k iterations to obtain a personalized face model. The parameters of the generator are not optimized in the first 50k rounds. We train our model on a single Nvidia Telsa V100 GPU. We use the Adam optimizer [22] to train the network, and the learning rate is set to $3e^{-4}$, $\beta_{1}$ and $\beta_{2}$ set to $0.9$ and $0.999$, respectively. The batch size is $2$ and $\lambda_{lpips}=5$. The number of basis vectors in our method is set to $k=50$.

We first conduct experiments with RGB frames as input. We adopt the three monocular videos used in NerFACE [11] for evaluation. For each video, we extract the first 2 minutes ($\sim$ 6000 frames) for training and the left 20 seconds ($\sim$ 1000 frames) for testing. After training, we directly send each testing frame into the learned encoder to obtain its latent code, which is then sent to the generator for novel view synthesis. To better verify the face reconstruction performance, we also present the performance of PTI [39], which is an optimization-based GAN inversion method. For each testing frame, PTI optimizes both the latent code and the EG3D generator for a faithfully reconstruction.

To measure their quality, we conduct a quantitative evaluation using several common metrics, including Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index (SSIM), and the Learned Perceptual Image Patch Similarity (LPIPS) [51]. As we don’t have novel view ground truth images, we calculate these metrics under the same views of the testing frames. The numerical results are given in Table 1. Compared to PTI, we obtain superior performance under all three metrics. In Figure 3, we show some rendered images with the camera views of testing frames and two randomly picked novel views. The complete novel view comparisons are given in the supplementary materials. Our method can better maintain the personalized characteristics, such as the mouth area. More importantly, for novel view synthesis, PTI shows significant artifacts for face shapes and cannot preserve the ID, while we obtain superior multi-view-consistent results. These demonstrate that the learned personalized generative prior enables faithful face reconstruction.

We then evaluate the performance of 3DMM-driven face reenactment. We compare to the 3D-aware methods, Neural Head Avatar (NHA) [15] and NerFACE [11]. We also provide the results of FOMM [41], a 2D-based face animation method. Note that FOMM doesn’t support novel view synthesis. As in the previous section, we adopt the three monocular videos used in NerFACE. We extract the 3DMM expression coefficients from each frame for training and testing.

The average results in terms of PSNR, SSIM and LPIPS are listed in Table 2. The 3D-aware methods NerFACE, NHA and ours obtain better performance than FOMM. With the help of the high-quality prior of the generative model, our method significantly boosts the performance of 3DMM-driven face reenactment.

We also show a qualitative comparison in Figure 4, where all results are rendered under the same view of the ground truth image. It can be seen that the results of FOMM have obvious artifacts, and the identity of the animated face is altered a lot from the ground truth. The results of NerFACE are too smooth, and the details of the facial textures are not well reconstructed. NHA cannot faithfully reconstruct facial characteristics, including eyeglasses and mouth areas. In comparison, our method can better maintain the facial characteristics and generate faithful face reenactment. In Figure High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors, we present some novel view rendered images of NerFACE and ours, a complete comparison is given in the supplementary materials. The results of NerFACE are blurred and distorted. It fails to generate high-quality renderings under novel views, while our method obtains faithfully multi-view-consistent images.

Following the practice of [17], we perform audio-driven experiments on three public videos collected from YouTube. The position of the camera is fixed and the resolution of the videos is $512\times 512$. Each video is divided into two segments of the training set and testing set, with no overlap between them. We extract their audio features for training and testing.

We compare our method with two audio-driven NeRF methods: AD-NeRF [17] and DFRF [40]. DFRF is a few-shot method. To make a fair comparison to it, we use a short 20 seconds video clip for training for all methods. In addition to the image quality metrics of PSNR, SSIM and LPIPS, SyncNet [9] is further used to measure audio-visual synchronization. The average metrics of three videos are given in Table 3. In the audio-driven scenario, our method also outperforms the previous methods with a significant margin for all four evaluation metrics. Besides, we provide some rendered images in Figure 5. We highlight the mouth areas that are most significant to audio-driven face reenactment. Compared to AD-NeRF and DFRF, we obtain much better mouth shapes and teeth. We also provide some novel view results of DFRF and ours in Figure High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors. Under novel views, the distortion of DFRF is even worse.

The key to our approach is to learn a basis to represent the personalized generative prior. We conduct ablation studies based on RGB-based face reconstruction to analyze the properties of the basis vectors.

Visualization of the basis vectors. We require the set of basis vectors to be orthogonal to each other during training. In Figure 6 we make a visualization of the learned basis. We also show the visualization results for a basis learned without the orthogonal constraints. It can be seen that the basis vectors are coupled together when there is no orthogonal constraint. With the orthogonal constraint, the basis vectors are disentangled and show better semantic meanings. The quantitative comparison results are shown in Table 4, which demonstrates that the reconstruction quality is much better with the orthogonal constraint.

Number of the basis vectors. We further explore the effect of the number of basis vectors. We vary the number of basis vectors by 5, 10, 20 and 50. The quantitative comparisons in terms of PSNR, SSIM and LPIPS are given in Table 4. And Figure 7 shows some rendered images under the different value of $k$. We highlight the mouth areas for better visualization. As the number of basis vectors increases, the model is more capable of representing facial details and obtains better rendered quality.