SAFA: Structure Aware Face Animation

3D morphable model (3DMM) is a statistical model which transforms the shape and texture of a 3D face into a vector space representation [1]. Traditional 3DMMs [1, 22, 4, 2, 16] are low-dimensional linear models generated through principal component analysis (PCA). A 3D face is represented by a linear combination of those orthogonal bases with largest eigenvalues. The state-of-the-art linear model, FLAME (Faces Learned with an Articulated Model and Expressions) [16] is learned from about 3800 3D face scans. Apart from shape and expression, FLAME also models four articulated joints (jaw, neck, eyeballs) and pose-dependent corrective blendshapes, which enables more flexible and expressive full head modeling. In this paper, we adopt FLAME as a decoder to model 3D faces and use a convolutional neural network as an encoder to estimate 3DMM parameters from face images.

Recent works on face animation can be divided into three categories: (1) motion-based methods, (2) landmark-based methods and (3) 3D-based methods.

Motion-based methods explicitly models a relative motion field from source to driving during the generation process. Early motion-based methods, e.g. X2Face [39], directly estimate a dense motion field by deep neural networks. Then the source face embedding is warped by the dense motion field to generate reenactment result. Monkey-Net [28] and its follow-up work, First Order Motion Model (FOMM) [29] uses abstract self-supervised 2D keypoints to estimate multiple local sparse motions and then combines them into a dense motion field to warp the source image. One-Shot Talking Heads [38] extends the 2D keypoints to the 3D space and estimates a dense motion field in 3D, which enables more expressive and flexible motion modeling. [30] replaces the keypoints with regions and models in-plane rotation and scaling via principal component analysis (PCA). However, it still lack the capability to model out-of-plane rotations and expressional motions. Among all the motion-based works, only FOMM [29] and [30] models the occlusion using an occlusion map learned in a self-supervised way. In this paper, two categories of occluded area are perceived separately and recovered with the help of the 3D geometry embedding and the widely used inpainting technique.

Landmark-based methods refer to those works that utilize facial landmarks as conditions to synthesis animated source with generative models. Few-shot Talking Heads [42] uses landmarks as the conditional input and injects source appearance features to the generator through an adaptive instance normalization layer (AdaIn) [12]. Fast Bi-layer [41] adopts spatially-adaptive denormalization (SPADE) layers [21] to build a neural rendering system based on the landmark skeleton. Few-shot Vid2Vid [36] uses SPADE residual blocks to inject landmark features into the generator, where appearance features are adapted to driving landmarks. Even landmarks are possible to represent face poses and relative motions, it still lacks the ability to preserve identity information and to handle occlusion.

3D-based methods take advantages of the geometric prior of 3D faces, which are reconstructed by fitting 3DMM parameters to the source and driving frames. The traditional 3D method Face2face [35] transfers driving expressions to the source through deformation transfer [32] and the reenacted 3D face is rendered to synthesize videos. In the deep learning era, 3D-based methods are combined with convolutional neural networks to generate more photo-realistic videos. HeadGAN [7] uses the rendered 3D faces of the driving frames as conditional inputs to build a neural rendering generator, where both deformed source appearance features and driving audio features are injected to generate realistic animation videos. StyleRig [34] and GIF [8] manipulate face images through a pre-trained StyleGAN conditioned on 3DMM parameters. Basically, these methods can generate realistic images, but are intrinsically disadvantageous in spatial and temporal consistency on videos.

In this paper, we propose a method that combines the advantage of motion-based and 3D-based methods which can generate photo-realistic and identity-preserving reenactments even in case of large pose deviation and severe occlusion.

We use FLAME [16] as the 3D face prior model in our framework. As a statistical 3D morphable model, FLAME acts as a decoder, which takes shape (or identity) $\mathbf{\alpha}\in\mathbb{R}^{\left|\mathbf{\alpha}\right|}$, expression $\mathbf{\beta}\in\mathbb{R}^{\left|\mathbf{\beta}\right|}$ and pose $\mathbf{\theta}\in\mathbb{R}^{\mathrm{3k+3}}$ ($k$ joint rotations and one global rotation) parameters as inputs and outputs a 3D head mesh with $v=5023$ vertices and $f=9976$ faces. The model is mathematically defined as follows,

where $W(\cdot)$ is the blend skinning function, which takes the offseted template mesh $T_{P}\in\mathbb{R}^{v\times 3}$, joints position $\mathbf{J}(\mathbf{\alpha})\in\mathbb{R}^{3k}$ and blendweights $\mathbf{w}\in\mathbb{R}^{k\times v}$ as inputs.

We adopt the differentiable renderer in PyTorch3D [23] to render the reconstructed 3D face. The differentiable renderer in our case rasterizes 3D vertex attributes into the 2D image space. The rendering function $\mathcal{R}$ can be mathematically formulated as

where $I\in\mathbb{R}^{H\times W\times a}$ is the rendered 2D image. $\mathbf{c}=(s,\mathbf{t})$ represents the scale and translation parameters of the weak perspective camera model that we adopted in this paper. $A\in\mathbb{R}^{v\times a}$ represents 3D vertex attributes to fill on the image and $a$ is the attribute dimension. In the following of the paper, we use attributes of RGB texture, vertex normal and vertex offset (motion) between source and driving faces. We only keep the face region of the rendered image through a face mask in the UV space (eyes and mouth regions are not included). With the differentiable renderer, the whole framework can be trained end-to-end.

The 3D motion module contains a 3DMM encoder, a FLAME decoder and a differentiable renderer. The 3DMM encoder is pre-trained to predict the 3DMM parameters $\Phi$ given face images.

Given $\Phi_{S}$ and $\Phi_{D}$, we obtain the 3D face vertices in the source and driving images from the FLAME decoder and then compute RGB texture, normal and motion attributes of these vertices. We interpret the source image $S(*)$ and the camera projection function $\mathbf{Proj}(*,\mathbf{c})$ as functions supporting arrayed inputs and outputs for convenience. The texture attributes $A_{S,tex}\in\mathbb{R}^{v\times 3}$ of the source image are obtained as

where bi-linear interpolation is used to obtain colors from $S$. The normal attributes $A_{S,n}\in\mathbb{R}^{v\times 3}$ and $A_{D,n}\in\mathbb{R}^{v\times 3}$ of both images (Omitted in the Figure 1) are computed from the neighbor faces of each vertex. The vertex motion attributes $A_{m}\in\mathbb{R}^{v\times 2}$ are computed as

Then by using Equation 2, we rasterize the following maps:

3D reenactment $I_{r}\in\mathbb{R}^{H\times W\times 3}$ refers to the rendered driving 3D face with texture transferred from source 3D face. $I_{n,S}\in\mathbb{R}^{H\times W\times 3}$ and $I_{n,D}\in\mathbb{R}^{H\times W\times 3}$ denote the normal maps of source and driving images, respectively. $\mathcal{T}_{S\leftarrow D,3D}\in\mathbb{R}^{H\times W\times 2}$ denotes the sparse motion field of the face area from the driving image to the source image. Exemplar visualization of $I_{r}$ and $\mathcal{T}_{S\leftarrow D,3D}$ is shown in Figure 1.

We adopt the same sparse motion estimator as proposed in FOMM [29], where an Hourglass Network [20] is used. The sparse motion estimator estimates $K=10$ keypoints $\left\{p_{S,k},p_{D,k}\in\mathbb{R}^{2}\right\}_{k=1}^{K}$ and Jacobians $\left\{J_{S,k},J_{D,k}\in\mathbb{R}^{2\times 2}\right\}_{k=1}^{K}$ from the source and driving images respectively. The keypoints are represented in terms of heatmaps and Jacobians are calculated by average pooling of the four output channels. The local affine transformations are approximated by the first order Taylor expansion,

where $z\in\mathbb{R}^{2}$ is an arbitrary point on the driving frame. $K$ sparse affine motion fields $\left\{\mathcal{T}_{S\leftarrow D,k}(z)\right\}_{k=1}^{K}$ are used to warp the source image separately. Warping is simply implemented using a differentiable bi-linear sampling operator [13]. Besides the $K$ sparse motions, an identity motion field is appended to model the static background area.

The dense motion estimator is an encoder-decoder network with similar design to [29]. It takes deformed source images and a set of heatmaps as inputs, which are downsampled to $1/4$ of the original image resolution. The deformed source images include $I_{r}$, $S$, and $\left\{\tilde{S}_{k}\right\}_{k=1}^{K}$. $\tilde{S}_{k}$ is obtained by deforming $S$ using $\mathcal{T}_{S\leftarrow D,k}$. $I_{r}$ corresponds to the face motion computed from the 3D face model. $S$ corresponds to staticity of the background. $\tilde{S}_{k}$ corresponds to the 2D affine motion. The heatmaps indicate where motion might happen and also have $K+2$ instances. $K$ heatmaps corresponding to the affine motion are formulated in the same way as FOMM, which are calculated by the difference of the heatmaps centered in $p_{D,k}$ and $p_{S,k}$.

where $\sigma=0.01$. For the static background, the heatmap is represented by a zero map. For the heatmap of 3D face motion, we make use of the $z$ channel of the normal maps $I_{n,S}$ and $I_{n,D}$. The $z$ component indicates whether the 3D face is visible in the camera frame, which helps estimate not only the 3D motions but also the occlusion maps.

The dense motion module estimates $K+2$ soft masks $M_{3D}$, $M_{b}$, and $\left\{M_{k}\right\}_{k=1}^{K}$ (as shown in the bottom-left block of Figure 1). $M_{3D}$ corresponds to the rigid 3D face (in white). $M_{b}$ corresponds to the static background (in black) and $\left\{M_{k}\right\}_{k=1}^{K}$ correspond to 2D affine motion areas (in other colors). All the masks are softmaxed to guarantee their pixel-wise sum to be 1. Sparse motion fields are fused into a dense motion field through masked averaging:

Besides, the dense motion module also predicts two occlusion maps $O_{g},O_{ca}\in\left[0,1\right]^{\frac{H}{4}\times\frac{W}{4}}$ indicating the invisible region in the source image but exposed in the generated image, which are used separately in the two elaborately designed modules in the occlusion aware generator to generate geometrically realistic face appearance and perceive areas need to be inpainted.

The occlusion aware generator takes $S$, $\hat{\mathcal{T}}$, $O_{g}$, $O_{ca}$ and the driving 3DMM parameters $\Phi_{D}$ as inputs and generates the reenacted image $D_{rec}$. We adopt a U-Net [24] shaped network, which contains an encoder, a geometrically-adaptive denormalization layer (GADE), a bottleneck block, a contextual attention module and a decoder. The encoder takes the $S$ as input and outputs source feature maps of different scales. Then those feature maps are warped by $\hat{\mathcal{T}}$. The highest level warped features are ported to the GADE layer, the bottleneck block and the contextual attention module to further recover the occluded area. The GADE layer and the contextual attention module are guided by $O_{g}$ and $O_{ca}$ separately to avoid potential conflicts. Finally, the recovered feature map is fed to the decoder. Lower level warped features are also concatenated to their corresponding intermediate features in the decoder in a U-Net fashion. Figure 1 shows this procedure in the bottom-right block.

The 3DMM encoder is pre-trained before we train the whole model end-to-end. We project 68 3D facial landmarks on the surface of the reconstructed 3D face $l_{i}\in\mathbb{R}^{3}$ into 2D image space and use the 2D landmark prediction $k_{i}\in\mathbb{R}^{3}$ from FAN [3] to formulate the $L1$ landmark re-projection loss. Besides, we use $L2$ parameter loss to regulate the 3DMM shape parameters $\mathbf{\alpha}$ and expression parameters $\mathbf{\beta}$ to obtain reasonable reconstruction results.

Then, our framework is trained in a self-supervised strategy. We perform self-reenactment, where a pair of source and driving images are sampled from each video. The driving images are used as the ground truth for supervision. We adopt loss functions of two main objectives: (1) to improve the faithfulness and quality of output images; (2) to regulate the 3D motion module and the first order motion module to predict reasonable motion fields. For the first objective, we follow the same loss functions in FOMM, including perceptual loss [14] and GAN-related loss. To regulate 3D motion, we preserve the 3DMM loss function in the pre-train stage. To stabilize 2D sparse keypoints and jacobians, we adopt the equivariance constraints from FOMM. Please refer to the supplementary material for the detailed expressions of loss functions.

We use Voxceleb1 [19] dataset to train and evaluate our approach. The Voxceleb1 dataset contains 22496 videos of 1251 celebrities extracted from YouTube videos. We adopt the same pre-processing strategy and train-test split as that of the FOMM. Videos of size $256\times 256$ are cropped from the original videos covering the heads. In total, we obtain 17941 videos for training and 506 videos for testing after cropping. Besides, we also extract images from the pre-processed videos and use FAN [3] to predict 68 2D facial landmarks of each image.

We first pre-train the 3DMM encoder using the images extracted from pre-processed videos for every 10 frame. We adopt the lightweight MobilenetV2 [26] as our 3DMM encoder and modify the output fully connected layer to output 156 dimensions of FLAME parameters and 3 dimensions of camera parameters. We train the 3DMM estimator for 50 epochs with batch size of 128. Adam optimizer is used with learning rate $2e-4$ and $\beta_{1}=0.9,\beta_{2}=0.999$

For the end-to-end model training, we perform self-reenactment on the pre-processed videos. We randomly sample a pair of source and driving images from each video and use the driving image as ground truth. The 3DMM encoder is jointly fine-tuned with the animation model during end-to-end training. We train the whole model for 50 epochs and for each epoch we repeat the video list for 150 times. We adopt synchronized Batch Normalization to further improve the performance. Adam optimizer is used with learning rate $2e-4$ and $\beta_{1}=0.5,\beta_{2}=0.9$ for all network modules. We use 8 24GB NVIDIA P40 GPUs to train our model.

To quantitatively evaluate our approach and the previous state-of-the-art methods, we perform video reconstruction experiments, where the first frame of each test video is used as the source image and the rest frames are used as driving images. The following metrics are computed:

(1) L1 distance: Averaged L1 distance between the reconstructed images and the ground truth images.

(2) Average Keypoint Distance (AKD): We extract facial landmarks from the reconstructed and ground truth images using FAN [3] and compute the average distance between the two set of landmarks. AKD measures how accurately the generated face video follows the face poses of the driving video.

(3) Average Euclidean Identity Distance (AEID): We adopt the widely used face recognition network, ArcFace [6] to extract identity features from both generated and ground truth images. Then we calculate the averaged euclidean distance between the two. AEID measures how much identity information is preserved.

(4) Fréchet Inception Distance (FID): FID was proposed in [11], which measures how realistic the visual quality of the generated images is, compared with the real images. We adopt the implementation of [27], which uses a pre-trained InceptionV3 [33] network to extract features from generated and ground truth images and calculates distribution distance between the two sets of features.

(5) Peak Signal-to-Noise Ratio (PSNR): PSNR measures the image quality between generated and real images in terms of the absolute mean squared error (MSE).

(6) Structural Similarity Index (SSIM): SSIM measures the structural similarity between the reconstructed and real images, which is more robust to global illumination changes compared with PSNR.

The units of L1, AKD, and PSNR are RGB intensity, pixels, and dB, respectively. FID, AEID and SSIM have no units since they are defined perceptually.

We compare our approach with the state-of-the-art face animation methods FS-Vid2vid [36], FS Talking Heads [42], Fast Bi-layer [41], FOMM [29], and One-Shot Talking Heads [38]. For FS-Vid2vid and Fast Bi-layer, we use the pre-trained models provided by the authors. But note that their training sets are much larger than ours (FS-Vid2vid: FaceForensics [25], Fast Bi-layer: Voxceleb2 [5]). For FS Talking Heads and FOMM, we train their models on the same training set as our method. For the latest proposed method One-Shot Talking Heads [38], due to the unavailability of the official source code, we only provide qualitative comparisons using cases reported in their paper.

We provide quantitative evaluation results for video reconstruction on the test set of Voxceleb1 dataset, as listed in Table 1. Our method performs best in all metrics. To visually illustrate the improvement besides the above abstract metrics, we provide qualitative visualizations on motion transfer, where source and driving faces are of different identities, as shown in Figure 4. We intentionally choose image pairs with large pose differences, translational motions and challenging expressions to show our superiority. The qualitative comparison with One-Shot Talking Heads is shown in Figure 5. We generate face images using a similar relative motion transfer strategy as in FOMM, please refer to the supplementary material for the details of the inference process of the motion transfer. Our method shows superiority in generating realistic, pose-preserving and identity preserving face animation results. Moreover, we improves the image quality for the occluded failure cases by [38] with fewer parameters and training data.

We conduct both quantitative and qualitative experiments on video reconstruction to show the contributions of the 3D motion module, the Contextual Attention Module (CAM) and the Geometrically-Adaptive Denormalization layer (GADE). Table 2 shows the quantitative results on the test set of Voxceleb1. With the integration of 3D face prior knowledge, the AKD score decrease significantly, which means the face pose of the reenactment becomes more accurate. The additional contextual attention module further improves the FID scores with context-coherent inpainting. Finally, injection 3D face geometry information into the generation process benefits both image quality and face appearance of the generated image. This shows the effectiveness of 3D geometry embedding in learning the scene structures. We also provide qualitative results as shown in Figure 6. Without 3D modelling, the faces might be improperly distorted. With the contextual attention module, the visual quality of the generated images improves. With the GADE layer, facial details and expressions become more realistic and self-occluded face area is properly recovered. The full model outperforms all the model variations in terms of visual sharpness of both foreground and background.