High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Previous researches [11, 1, 12] project the intermediate landmark representation on the image plane, forming the sketch image as generation condition in an indifferentiable manner. In contrast, our TalkFormer first uses a 1D-convolution embedding module to encode the target full-face landmarks from the landmark completion module into $n$ landmark embeddings $\{e_{i},i=1,2,...,n\}$, where $n$ is the number of landmarks to represent the full face. Then, these landmark embeddings are integrated into cross-attention layers as keys and values, denoted as $K_{1}$ and $V_{1}$, respectively. Simultaneously, the queries $Q_{1}$ are extracted from the hidden features after ResNet[49] blocks through an MLP layer. The output of cross-attention is computed as $Y$ according to the following equation:

where $d_{1}$ is the dimension of queries and keys. Subsequently, the results $Y$ go through a zero-initialized convolution layer and are added to the hidden features of U-Net in a residual manner. In this way, the final generated face is ensured to have talking motion aligned with the motion represented by landmarks, and the diffusion model can be jointly optimized with the landmark completion module for improved lip synchronization.

To enable generalized talking face generation, a single reference face image is typically utilized as condition, ensuring that the synthesized appearance remains consistent with the subject appearance. As illustrated in the pink section of Figure 2, the reference face image is initially encoded to the latent space as $z_{r}$. To retain more fine-grained details from the reference face image, we devise an appearance encoder similar to the U-Net encoder consisting of residual convolution blocks. This appearance encoder converts the latent $z_{r}$ into multi-scale reference features symbolized as $F_{a}=\left\{F_{a}^{i}\mid i=1,2,..,I\right\}$, where $I$ represents the number of scales in U-Net. The dimensions of these features are identical to those of the hidden features in the encoder of U-Net denoiser.

To make the denoiser model aware of more appearance details from the reference image, we align the multi-scale reference appearance features in an implicit warping manner through another cross-attention layer. Specifically, we denote the hidden features after talking motion alignment as $F_{h}^{i}\in\mathbb{R}^{D\times H\times W}$, where scale $i\in\{2,3,\dots,I\}$. To spatially align the reference appearance features $F^{i}_{a}$ with the hidden features $F_{h}^{i}$, the $F^{i}_{h}$ is first transformed to the queries $Q_{2}\in\mathbb{R}^{HW\times d_{2}}$ through an MLP layer while the $F^{i}_{a}$ are projected to the keys $K_{2}\in\mathbb{R}^{HW\times d_{2}}$ and values $V_{2}\in\mathbb{R}^{HW\times D}$, where $d_{2}$ is the dimension of the keys. Then, the correlation matrix between $F^{i}_{a}$ and $F^{i}_{h}$ is computed as follows:

where $S\in\mathbb{R}^{HW\times HW}$, and each element $s_{jk}$ of it indicates the semantic correspondence between the hidden feature in location $j$ and the reference feature in location $k$, with $j,k\in\{1,2,...,HW\}$. Based on this correlation matrix, we can obtain the aligned reference features by referring to the relevant features in the reference appearance features. Specifically, the reference appearance features $F^{i}_{a}$ are warped implicitly via a weighted sum of the values in $V_{2}$ as follows:

where $\bar{F^{i}_{a}}\in\mathbb{R}^{D\times H\times W}$, and its semantic contents are spatially aligned with those of the hidden features $F^{i}_{h}$. Eventually, the aligned reference appearance features $\bar{F^{i}_{a}}$ are passed through a zero-initialized convolution layer and added to the hidden features $F_{h}^{i}$ using a residual way. Consequently, the denoising process can better preserve the subject appearance details from reference images, facilitating high-fidelity talking face video generation.

We conduct experiments on two public audio-visual datasets, VoxCeleb [50] and HDTF [14]. VoxCeleb is a collection of over 100,000 utterances from 1,251 celebrities, all extracted from videos uploaded to YouTube. HDTF is a high-resolution audio-visual dataset consisting of approximately 362 distinct videos, spanning over 15.8 hours, in 720P or 1080P resolutions. Compared to HDTF, the large-scale VoxCeleb dataset is a more standard benchmark commonly used in prior work. To ensure a fair comparison, all comparison methods, including ours, are trained on the VoxCeleb dataset and evaluated using the test sets of both VoxCeleb and HDTF.

We quantitatively evaluate all methods regarding visual quality and lip synchronization. Pixel-level visual quality is assessed through the Peak Signal-to-Noise Ratio (PSNR) and Structured Similarity (SSIM) [51], while feature-level visual quality is evaluated using Learned Perceptual Image Patch Similarity (LPIPS) [52] and Fréchet Inception Distance (FID) [53]. Compared to pixel-level measurements, the feature-level measurements are more in line with human perception [52, 54]. Additionally, we employ the cosine similarity (CSIM) of identity vectors extracted by the ArcFace face recognition network [55] to assess the preservation of subject identity. SyncScore [56] is commonly used by prior work to evaluate the lip-audio synchronization quality, despite some limitations.

We compare our approach against several state-of-the-art person-generic audio-driven talking face video generation methods. DiffTalk (CVPR’23) [9] constructs a Diffusion-based framework for generalized talking face synthesis by conditioning the latent diffusion model on audio signal. PD-FGC (CVPR’23) [10] employs a progressive disentangled representation learning strategy to achieve fine-grained controllable talking face synthesis (e.g., eye, pose control). IP-LAP (CVPR’23) [11] is a two-stage landmark-based method that trains different stages separately and utilizes predicted optical flow to align the reference image with the target pose and expression. PC-AVS (CVPR’21) [20] proposes a GAN-based framework to generate pose-controllable talking face videos by modularizing audio-visual representations. Wav2Lip (MM’20) [8] utilizes a lip sync discriminator to guide the generator in generating lip-synced talking face videos.

Wav2Lip [8], IP-LAP [11], DiffTalk [9], and our method all generate talking face videos by inpainting the lower half of the face. Therefore, during the quantitative comparison, the lower half of the face in the input video is masked. Then, these methods reconstruct the masked area guided by the input audio and reference image. The original input video serves as the ground truth for metric calculation. We train DiffTalk [9] using the official code until convergence, but it generates temporally unstable results. It relies on additional frame interpolation to smooth the results, affecting comparison fairness. Therefore, the frame interpolation post-processing was not employed for fair comparison. PC-AVS [20] utilizes a pose source video, an audio input, and a reference image to generate a talking face video. In our implementation, we substitute its pose source video with the ground-truth video. PD-FGC [10] requires a pose source video, an expression source video, an eye blink source video, an audio input, and a reference image to generate a talking face video. Our version replaces its pose source, expression source, and eye blink source videos with the ground-truth video.

In our framework, the input face images are resized to 256$\times$256, and the latent space of autoencoder $\mathcal{D}(\mathcal{E}(\cdot))$ has a spatial dimension of 64$\times$64. Facial landmarks are extracted from video frames using the mediapipe tool [57]. We represent the lip with $n_{l}=41$ landmarks, the jaw with $n_{j}=16$ landmarks, and the entire face with a total of $n=131$ landmarks. We set $N$ to 5 and the reference full-face landmarks are detected from the randomly selected frames of input videos. The hyper-parameter $\lambda$ is set to 10 and $L$ set to 5. To generate talking face videos, we employ the DDIM[26] diffusion sampler with 200 steps. We set the number of diffusion steps $T$ to 1000. The number of scales $I$ is 4, but we illustrate the case of $I=3$ in Figure 2 for clarity. The reference face image can be any face image from the input video that reflects as many appearance details of the subject as possible. All comparison methods use the same reference face image to ensure a fair evaluation. Our implementation of the proposed method closely follows the code implementation of latent-diffusion [58], while incorporating TalkFormer, Appearance Encoder, and Landmark Completion module[11] as additional components. The Appearance Encoder is designed based on the encoder structure of U-Net denoiser in latent-diffusion [58], excluding self-attention layers.

We train our framework on 2 NVIDIA A100(40GB) GPUs for 500 epochs with Adam optimizer[59]. The batch size is 64, and the learning rate is 4e-5. The pre-trained autoencoder is frozen during training. The Landmark Completion module is jointly trained from scratch with the latent diffusion model. Our method focuses on inpainting the lower half of the face based on the input audio. Hence, to generate talking face videos, we first crop the face area from the input template video as network input. After obtaining the generated face, we employ a post-processing technique following the previous method[11] to seamlessly blend it with the background, producing final talking face videos. For fair comparison, this post-processing technique was not employed during the quantitative and qualitative comparisons. The input template video has no length requirement and can be looped to match the length of the input audio. We will release our code upon acceptance.

Our method outperforms other methods in all visual quality metrics (PSNR, SSIM, LPIPS, FID, CSIM) on both VoxCeleb [50] and HDTF [14] datasets. Specifically, on the perceptual distance metric LPIPS and FID, our method significantly improves over other methods. This verifies that our method can produce high-fidelity talking face videos that align with human perception, preserving more appearance details. Besides, the highest CSIM score achieved by ours also indicates our method can preserve more identity information of the target subject. Although Wav2Lip exhibits a slight lag in terms of PSNR and SSIM metrics compared to our method, its FID and LPIPS values are approximately twice as high as ours, suggesting the presence of artifacts that is not aligned with human perception in their results. The performance of IP-LAP is closely comparable to ours in terms of the PSNR, SSIM, and CSIM metrics, but there remains a certain gap between IP-LAP and ours when assessed on the LPIPS, FID, and SyncScore metrics. While DiffTalk exhibits comparable performance in the FID metric, it still lags behind our approach when assessed on the LPIPS and CSIM metrics.

Due to different speaking styles among individuals, accurate quantitative assessment of lip-audio synchronization remains a persistently challenging task. A common practice is to calculate the SyncScore based on the audio and visual features of SyncNet [56]. Wav2Lip, PC-AVS, and PD-FGC directly model the audio-visual mapping and obtain better SyncScore than ours. However, our approach notably excels in visual quality metrics, particularly in preserving the finer details of subject appearance, an aspect where others have room for improvement. Wav2Lip utilizes SyncNet [56] as a discriminator during training. Hence, it achieves a very high SyncScore, even higher than that of the ground truth. Besides, PD-FGC and PC-AVS adopt audio-visual contrastive learning similar to SyncNet [56], which contributes to a higher SyncScore but compromises visual quality. IP-LAP leverages facial landmarks as intermediate representations, but its lip synchronization is inferior to ours due to the isolated training of different stages. DiffTalk directly models the audio-visual mapping and generates temporally unstable videos with poor lip synchronization. We suspect its issue might stem from the mapping ambiguity magnified by the multi-step iteration of diffusion model.

As shown in Figure 3, we present some representative comparison results on the HDTF [14] and VoxCeleb [50] datasets. It can be observed that our results are visually closer to the ground-truth images than other methods’, with more appearance details (e.g., beard, lip, teeth) preserved. It implies that our TalkFormer module could effectively align the reference image features based on semantic correlation, providing valuable features for the diffusion model. Besides, the lip shapes of ours are also closer to the ground truth. DiffTalk, PD-FGC, PC-AVS, and Wav2Lip directly learn the audio-visual mapping and utilize a misaligned reference image as generation condition. Therefore, their results lose some appearance details of the subject and appear blurry. IP-LAP generates results that are blurrier than ours and exhibit some artifacts, possibly due to the inaccurate optical flow estimation for aligning reference images. For more qualitative comparisons, please refer to the supplementary video as detailed in the following subsection.

We have provided a short video as supplementary material. Please download and watch it. If there are any issues downloading the video from the review system, the same file can also be downloaded through this backup link. The time schedule of the video is as follows:

• •

  00:00 ~ 00:08: Video title.

• •

  00:08 ~ 00:46: Demonstration of Ours. We demonstrate a generated result of our method where the driving audio is sourced from the text-to-speech technique, and the subject is from the HDTF[14] dataset. We also visualize the intermediate landmarks after the landmark completion module.

• •

  00:46 ~ 01:42: Method Comparison. We present the results of all methods as well as ground-truth videos for comparison.

• •

  01:42 ~ 02:19: Ablation Study. We present the qualitative results of the ablation study, which will be further analyzed in the Section IV-D.

• •

  02:19 ~ 03:14: More results of our method. We provide the testing results of our method for more cases.

For comprehensive evaluation, we conduct a user study where 16 volunteers are invited to assess the generated videos of all comparison methods. We randomly sample 10 videos for testing, 5 from HDTF [14] dataset and 5 from VoxCeleb [50] dataset. Volunteers are asked to give their rates (0-5) for each generated video regarding image quality, lip-audio synchronization, and fidelity of appearance details. The videos are presented to participants in a random order and the evaluation criteria is explained in detail to the participants. The mean opinion scores (MOS) of each method are presented in  Table II. Our method receives better evaluations from participants across three dimensions than other approaches.

Our two-stage landmark-based method achieves the joint optimization of landmark completion module and latent diffusion model to improve lip synchronization. To validate the effectiveness of end-to-end optimization, we devise a variant where these two modules are trained separately. Specifically, the landmark completion module is optimized using only $\mathcal{L}_{1}$ loss (Equation 2). In the denoiser U-Net of latent diffusion model, TalkFormer accepts the pre-estimated ground-truth landmarks as condition. The latent diffusion model is then optimized using only $\mathcal{L}_{ldm}$ loss (Equation 1).

The numerical results of this variant are reported in the “Ours w/o End2End” row of Table III. In terms of visual quality metrics, this variant exhibits similar performance to our full model. However, the SyncScore of this variant drops 9.71% compared to the full model’s, verifying the effectiveness of end-to-end training in improving lip-audio synchronization. Besides, as seen in the “Ours w/o End2End” row of Figure 4, without the end-to-end optimization, the synthesized lip shapes are less accurate, while the visual quality remains similar to the full model’s.

Our TalkFormer module aligns the synthesized motion with the motion represented by landmarks via cross-attention layer. To implement a variant without talking motion alignment, we remove the first cross-attention layer in TalkFormer that integrates landmarks embeddings as condition. Following the practice of DiffTalk [9], we redesign the landmark embedding module composed of multiple MLP layers to encode the target full-face landmarks into a single landmark embedding. This landmark embedding is added to all spatial locations of the hidden features in U-Net.

The numerical results of this variant are reported in the “Ours w/o M-Align” row of Table III. It can be seen that the SyncScore drops significantly compared to the full model’s. This is because the synthesized motion of the lip and jaw could not be accurately controlled through a simple addition operation. Besides, the adding operation may introduce artifacts into the generated results, deteriorating the visual quality metrics. As shown in the “Ours w/o M-Align” row of Figure 4, in the absence of TalkFormer’s talking motion alignment, the mouth shape remains predominantly closed. Besides, influenced by the simple addition operation, the generated mouths appear blurry and exhibit some artifacts. In the first and fourth images of the “Ours w/o M-Align” row, the generated mouth shapes are somewhat skewed, which implies loss of the subject identity information.

To develop a variant without reference appearance features alignment, we remove the second cross-attention layer in TalkFormer that incorporates the reference appearance features, and replace it with a self-attention layer akin to DDPM [25]. Besides, the appearance encoder is removed. Following the common practice of previous researches [9, 8], the reference face image is first encoded into the latent space as $z_{r}$. The $z_{r}$ is then concatenated with the noisy latent $z_{t}$ along the channel dimension and fed into the U-Net denoiser network.

The numerical results of this variant are reported in the “Ours w/o R-Align” row of Table III. It can be seen that all the visual quality metrics deteriorates compared to the full model’s. Although the pixel-level metrics (PSNR, SSIM) do not change significantly, the feature-level metrics (LPIPS, FID), which are more in line with human perception, increase by a large margin. The potential reason is that the diffusion model can not extract meaningful features from the misaligned reference features, resulting in the loss of subject appearance details. Besides, the CSIM metric based on identity vectors might not be sensitive to subject appearance details, therefore the CSIM decreases slightly without reference features alignment in TalkFormer. Moreover, the lip shape of the misaligned reference image might have a negative impact on the synthesized lip shape. Therefore, the SyncScore decreases without reference features alignment. Furthermore, as can be seen in the “Ours w/o R-Align” row of Figure 4, in the absence of reference appearance features alignment, the generated results lose some subject appearance details, resulting in some unrealistic contents. Besides, the lip shapes are less accurate influenced by the misaligned reference face image.