Style-Preserving Lip Sync via Audio-Aware Style Reference

The task of audio-driven lip sync is to generate lip-synced videos given a segment of audio, a video, and a style reference video as inputs. In this task, the lower-half face of the input video is masked, and then the masked region is inpainted with lip-synced content by the proposed method.

According to [19], the 3DMM mainly serves as a comprehensive three-dimensional facial model that represents facial shape using a fixed number of points. Specifically, the facial shape can be depicted using a facial mesh $S\in\mathbb{R}^{3N}$ with $N$ vertices, and such a facial mesh $S$ can be parameterized via Principal Component Analysis (PCA) as follows,

Here, $\overline{S}\in\mathbb{R}^{3N}$ denotes the average facial shape, while $B_{id}$ and $B_{exp}$ represent the PCA bases for identity and expression, respectively. The parameters $\alpha\in\mathbb{R}^{80}$ and $\beta\in\mathbb{R}^{64}$ control facial identity and expression, respectively. Head rotation is expressed using another set of parameters, $\gamma\in\mathbb{R}^{3}$.

Typically, off-the-shelf 3D face reconstruction models enable the extraction of the parameters $\alpha,\beta,\gamma$ from any real-world facial image. It is important to note that of the 64 expression parameters, only 13 are significantly related to mouth motions, as illustrated by [6, 5]. Therefore, in this work, we utilize these 13 mouth-related expression parameters to represent lip motion.

An overview of our method is illustrated in Figure 2. In this study, we present a novel two-stage framework called StyleSync, which introduces an innovative audio-aware style reference scheme for style-preserving lip sync. Specifically, we first propose a module named Reference-guided Lip Motion Prediction, aimed at predicting lip shapes from input audio signals guided by a style reference video. This is achieved through an innovative Transformer-based model that aggregates speaking style information from the style reference video. Then, StyleSync introduces another module called Realistic Face Rendering, which utilizes a conditional latent diffusion model to produce realistic talking face videos.

A style reference video of an individual contains the lip shapes produced when that individual pronounces some utterances, reflecting the speaking style. Initially, we employ DeepSpeech [30] to extract the per-frame DeepSpeech features of the style reference video, denoted as $a^{\prime}_{1},a^{\prime}_{2},\cdots,a^{\prime}_{n}$, where $n$ represents the length of the style reference video. These DeepSpeech features are further processed by a 1D temporal convolutional module and fed into a Transformer encoder, yielding reference audio features $k_{1},k_{2},\cdots,k_{n}$.

To obtain the corresponding lip shape representation from the style reference video, we utilize an off-the-shelf 3D face reconstruction model [31] to extract 3DMM expression parameters from video frames. We then utilize the mouth-related expression parameters to represent the reference lip shapes, denoted as $m^{\prime}_{1},m^{\prime}_{2},\cdots,m^{\prime}_{n}$. These mouth-related expression parameters are inputted into another Transformer encoder, generating reference lip features $v_{1},v_{2},\cdots,v_{n}$.

Once obtaining the speaking style information from the style reference video, represented by reference audio and lip features, we utilize a Transformer decoder [32] for generating mouth-related expression parameters corresponding to the input audio. As illustrated in Figure 2, the Transformer decoder consists of $N_{1}$ Transformer blocks, each comprising a stack of self-attention, cross-attention, and feed-forward layers.

To capture temporal coherence, the Transformer decoder takes audio features from a temporal window $t-w,t-w+1,\cdots,t+w$ as input, where $w$ represents the window size and $t$ denotes the current time step. Guided by the reference audio and lip features, it predicts the lip motion at the middle time step $t$. To achieve this goal, we first encode the DeepSpeech features of the temporal window into audio tokens using a 1D convolutional module, denoted as $a_{t-w},a_{t-w+1},\cdots,a_{t+w}$. Subsequently, these audio tokens are inputted into the self-attention layer of the transformer decoder to generate hidden features $h_{t-w},h_{t-w+1},\cdots,h_{t+w}$.

To achieve style aggregation, previous methods, such as [6, 5, 33], utilize a self-attention pooling layer on reference lip features to aggregate the style information. Unlike these previous approaches, this work proposes a novel audio-aware style aggregation scheme using cross-attention layers.

Specifically, we begin by treating the obtained reference audio features as keys $\mathbf{K}={[k_{1},k_{2},\cdots,k_{n}]}^{\top}\in\mathbb{R}^{n\times d}$, and the corresponding reference lip features as values $\mathbf{V}={[v_{1},v_{2},\cdots,v_{n}]}^{\top}\in\mathbb{R}^{n\times d}$, where $d$ represents the dimension of the key. These keys and values are inputted into the cross-attention mechanism. Simultaneously, the hidden features of temporal window act as queries in the cross-attention mechanism, denoted as $\mathbf{Q}={[h_{t-w},h_{t-w+1},\cdots,h_{t+w}]}^{\top}\in\mathbb{R}^{(2w+1)\times d}$. Therefore, the style reference information can be aggregated through the cross-attention layer, formulated as follows,

where $\mathbf{s}\in\mathbb{R}^{(2w+1)\times d}$ denotes the aggregated styles for the temporal window at time step $t$.

It is noteworthy that, when predicting lip motions for different audio inputs, the attention weights $\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}\in\mathbb{R}^{(2w+1)\times n}$ vary based on the similarity between the reference audios and the audios of the temporal window. The closer the reference audio is to the audio within the temporal window, the more its corresponding reference lip shape contributes to aggregating the style. This differs from prior works [6, 5, 33, 11], which aggregate a video-level style remaining static for predicting lip motion for arbitrary audio.

The aggregated styles are then added to the hidden features in a residual manner, which are further fed into the feed-forward layer. As a result, the Transformer decoder outputs $2w+1$ vectors, among which the middle vector, corresponding to the ($w+1$)-th position, is inputted into an MLP layer to generate mouth-related expression parameters, denoted by $\hat{m}_{t}$, for the time step $t$.

During the stage of lip motion prediction, the optimization involves L1 reconstruction for both mouth-related expression parameters and lip vertices, as detailed below.

As the proposed StyleSync predicts lip motion for each frame individually, ensuring temporal consistency in Lip Motion Prediction is crucial. To achieve this, we follow the common practices of previous methods[34, 6, 5] by first generating the lip motions of successive $L=64$ frames for each video during training. Subsequently, we utilize the following L1 reconstruction loss for mouth-related expression parameters:

where $m_{t+i}$ represents the ground truth mouth-related expression parameters for the $(t+i)$-th frame.

Additionally, since mouth-related expression parameters may contain some information unrelated to lip shape, we also impose an L1 reconstruction loss on lip vertices. Specifically, the predicted mouth-related expression parameters $\hat{m}_{t+i}$ can be combined with other parameters and then converted into the face mesh vertices using Equation (1). We denote the lip vertices in the facial mesh as $\hat{l}_{t+i}$, and the reconstruction loss on lip vertices can be expressed as:

where $l_{t+i}$ represents the ground truth lip vertices detected by face alignment tools [35].

Overall, the optimization objective of this stage can be summarized as follows,

where $\lambda$ is a scalar for loss item balance, empirically set to 300.

In the lip motion prediction stage, we have predicted the mouth-related expression parameters representing the lip motion synchronized to the input audio. To ensure that the synthesized facial image has lip shape aligned with the one reflected by the mouth-related expression parameters, we integrate the lip motion condition into the latent diffusion model via modulated convolutional layers, drawing inspiration from StyleGAN2 [37].

Specifically, as shown in the Figure 2, the mouth-related expression parameters are first combined with the identity and rotation parameters extracted from the input video frame, aiming to preserve the subject identity and head rotation. The combined parameters are then encoded to a latent code by a 1D convolutional module. The latent code is employed to modulate the convolution weights in the convolutional layers of denoising U-Net. Formally, we denote the original convolution weights as $\omega$. The latent code is mapped to a scale set $\Phi=\{\phi_{i}\}$ by an MLP layer, where $\phi_{i}$ is the scale coefficient corresponding to the $i$-th input feature map. Then, we compute the weight of modulated convolution $\omega_{ijk}^{\prime}$ as follow,

where $j$ indicates the convolution weight for the $j$-th output features map, $k$ indicates the spatial footprint of the convolution, and ${\epsilon}^{\prime}$ is a small constant for avoiding numerical issues.

Finally, we perform the convolution operation with the modulated convolution weight to produce the output features map.

To ensure the generated face has high-fidelity appearance of the subject, a reference facial image should be provided as a condition for the denoising network. As shown in the Figure 2, the reference image is initially encoded into the latent space as $z_{r}$. To preserve more appearance details from the reference image, we devise an appearance encoder consisting of residual convolution blocks. The appearance encoder converts the reference latent $z_{r}$ into the multi-scale reference features. Given the efficacy of preserving appearance details from the reference image proven by [38, 39], we then integrate the resulting reference features into the denoising network via spatial cross-attention. Specifically, the reference features serve as the keys and values for spatial cross-attention layers, while the hidden features after modulated convolutional layer in the denoising network serve as the queries. Finally, the output of the spatial cross-attention layer are added to the hidden features in a residual manner.

For optimization, we train the diffusion rendering model using the objective as defined in  Equation 6. The latent diffusion model generates the facial image of each frame individually, thus enabling the generation of videos with arbitrary length. As well, for achieving temporal consistency, we adopt the batched sequential training strategy [34] to generate five sequential images for each video during training. For model inference during testing, the denoising processes for all frames share the same initial random noise. Note that we randomly select the reference facial image from the same video during training, but alternatively, choose any facial image reflecting the subject appearance as reference during inference.

In this study, we conduct experiments on two widely utilized audio-visual datasets: VoxCeleb [40] and HDTF [14]. The VoxCeleb dataset comprises over 100,000 utterances from 1,251 celebrities, sourced from videos uploaded to YouTube. It showcases a broad diversity in terms of ethnicities, accents, and ages, and is a standard benchmark commonly used in prior work. On the other hand, the HDTF dataset is a high-resolution audio-visual collection consisting of approximately 362 unique videos spanning over 15.8 hours, with resolutions of 720P or 1080P. To facilitate a fair comparison, all comparison methods, along with the proposed StyleSync, are trained on VoxCeleb and tested on both datasets.

The videos are resampled to 25 fps, and the facial images cropped from the video frames are resized to 256$\times$256 resolution. The proposed method is composed of two stages, each trained separately. For Reference-guided Lip Motion Prediction, the style reference video is extracted from the talking clip of the same individual at different time steps. Importantly, there is no overlap between the style reference video and the ground-truth video during testing. The length $n$ of the style reference video is set to 256 following previous methods [6]. The transformer decoder consists of $N_{1}=3$ blocks, with the temporal window size $w$ set to 5. The query dimension $d$ is 256, and we utilize the multi-head attention layer with 8 heads. In the stage of Realistic Face Rendering, the number of time steps for diffusion, denoted as $T^{\prime}$, is 1000. The latent space of the pre-trained autoencoder has a spatial resolution of 64$\times$64. During inference, we employ the DDIM [29] sampler with 200 steps to accelerate the generation process. The reference facial image is randomly selected from the input video, and all comparison methods use the same reference facial image for a fair evaluation. After obtaining the generated face, we blend it with the background following the previous method[8], producing final talking face videos.

We train all modules using the Adam optimizer [41]. The lip motion prediction network is trained with a batch size of 256 and a learning rate of 2e-4 on 4 NVIDIA 3090 GPUs. In our latent diffusion framework, the autoencoder is initialized with the pre-trained weight from latent-diffusion [42] and is frozen during training. The latent diffusion model is trained with a batch size of 12 and a learning rate of 4e-5 on 4 NVIDIA 3090 GPUs.

In the task of audio-driven lip sync, the evaluation mainly focuses on the accuracy of lip sync and the visual quality of the generated facial video. For the assessment of speaking style preservation in lip sync, this study utilizes the widely adopted metric called Lip Landmarks Distance (LipLMD) [43], which quantifies the discrepancies between the generated lip shapes and the ground-truth lip shapes corresponding to the input audio. A lower score of LipLMD indicates that the generated lip shapes are closer to the ground-truth lip shapes reflecting the individual’s speaking style, signifying a more accurate preservation of the speaking style in the generated lip sync. Furthermore, SyncScore [44] is another commonly used metric for evaluating the synchronization of lip shape with the input audio. The SyncScore is computed based on the distance between the audio feature and visual feature in SyncNet [44]. However, since SyncNet does not model the speaking styles of the individuals, SyncScore primarily assesses whether the generated lip shape matches the lip shape conforming to the general speaking style learned from a large-scale audio-visual dataset. As previously mentioned, there are still discrepancies between the lip shapes conforming to the general speaking styles and those customized to the individual’s speaking style. Therefore, it is worth noting that the SyncScore is insufficient to assess the speaking style preservation. A better SyncScore only indicates that the distance between audio and visual features in SyncNet is closer, which does not necessarily mean that the generated lip shapes are more accurate and preserve the speaking style better. Despite reporting results for this metric, we recommend LipLMD as the preferred indicator for evaluating lip sync quality since our study focuses on the task of style-preserving lip sync.

Additionally, in assessing the visual quality of the generated talking face videos, this study employs Peak Signal-to-Noise Ratio (PSNR) and Structured Similarity (SSIM) [45] for evaluating pixel-level visual quality. Moreover, we assess feature-level visual quality using the Learned Perceptual Image Patch Similarity (LPIPS) [46] and Fréchet Inception Distance (FID) [47]. Compared to pixel-level measurements, the feature-level evaluations are more aligned with the human perception, as observed in [46, 48].

In this work, we compare the proposed method with several representative state-of-the-art (SOTA) subject-generic approaches in audio-driven talking face video generation, including StyleTalk [6], DiffTalk [10], PD-FGC [16], IP-LAP [8], PC-AVS [9], and Wav2Lip [7]. StyleTalk [6] is a representative subject-generic method capable of modeling speaking style. However, the code released for StyleTalk is incomplete and unable to generate results, lacking the phoneme label extraction and training code. Consequently, we made some minor modifications to its code for comparison purposes. The modified version utilizes DeepSpeech [30] audio features instead of phoneme labels and is trained using the same objectives as ours. Additionally, StyleTalk’s image rendering network can only accept a reference facial image to provide subject appearance information, whereas other comparison methods input more conditions, such as the upper-half face at the current time step. For a fair comparison, we replaced StyleTalk’s image renderer with our diffusion rendering model to generate results. We only compare with StyleTalk in terms of lip sync quality. The methods including DiffTalk[10], IP-LAP [8], Wav2Lip[7], and Ours all synthesize talking face videos through inpainting the lower half of the face influenced by the input audio and reference facial image. Therefore, for quantitative comparison, the lower half of the face in the input video is masked and then these methods reconstruct the masked area. The input video before masking serves as the ground truth for metric computation. We train DiffTalk [10] using the released official code until convergence, but it generates temporally unstable lip motion. It relies on additional frame-interpolation techniques to smooth the results, affecting the comparison fairness. Therefore, the frame-interpolation post-processing was not utilized for a fair comparison. PC-AVS [9] utilizes a pose source video, an audio input, and a reference facial image to synthesize a talking face video. For a fair comparison, we replace its pose source video with the ground-truth video. PD-FGC  [16] inputs a pose source video, an expression source video, an eye blink source video, audio, and a reference facial image to synthesize a talking face video. For a fair comparison, we replace its pose source, expression source, and eye blink source videos with the ground-truth video.

As detailed by the LipLMD metric, the proposed method outperforms all of the comparison baselines on both VoxCeleb and HDTF datasets. This demonstrates its generation of more accurate lip sync that better preserves the speaking styles of the individuals. Specifically, Wav2Lip, PC-AVS, DiffTalk, IP-LAP, and PD-FGC lag significantly behind our proposed method in LipLMD due to their incapability to model the speaking styles of individuals. StyleTalk incorporates the use of style reference videos to model speaking styles. However, it still lags behind our StyleSync because of its inaccuracies in style aggregation. Specifically, StyleSync’s LipLMD obtains lower scores than StyleTalk by 8.4% on VoxCeleb and 13.6% on HDTF.

In SyncScore, Wav2Lip and PC-AVS achieve higher scores than the proposed method. This is because Wav2Lip utilizes SyncNet as the lip sync expert to guide the training of its generator, and PC-AVS adopts an audio-visual contrastive learning mechanism similar to SyncNet’s. However, this only indicates that their generated lip shapes are closer to the lip shapes matching the general speaking style rather than the individuals’ speaking styles, since the SyncNet for computing SyncScore overlooks modeling the speaking styles of individuals.

As shown in the Table I, the proposed StyleSync consistently outperforms other algorithms in terms of visual quality metrics, including PSNR, SSIM, LPIPS, and FID. Particularly noteworthy are the improvements observed in the feature-level metrics LPIPS and FID, where our method demonstrates significant enhancements over other approaches. This suggests that our results align more closely with human perception, demonstrating the efficacy of our conditional latent diffusion model in rendering high-fidelity and realistic faces. As well, DiffTalk devises a latent diffusion framework to generate high-fidelity talking face videos but still lags behind ours. This may be because it simply concatenates the misaligned reference image latent as condition while our method integrates the multi-scale reference image features through spacial cross-attention layers. Furthermore, IP-LAP employs the predicted optical flow to align the reference image and features but still achieves inferior performance than our algorithm in terms of visual quality. This may be due to the training instability and mode collapse of its GAN-based framework.

As shown in Figure 3, compared to other algorithms, the proposed method can produce lip shapes that are closer to the ground truth. This indicates our superiority in achieving accurate lip sync conforming to the speaking styles of individuals. StyleTalk sometimes generates inaccurate lip shapes due to its inaccuracy of the style aggregation scheme. Moreover, other approaches, such as DiffTalk, PD-FGC, IP-LAP, PC-AVS, and Wav2Lip, produce sub-optimal lip shapes because they overlook modeling the speaking styles of individuals.

From Figure 3, it can be seen that the proposed method can generate more realistic faces with high-fidelity facial details and fewer artifacts. Compared to other methods, our results visually resemble real faces more closely. This verifies the effectiveness of the proposed conditional latent diffusion model in rendering lip motion into realistic faces. During implementation, StyleTalk uses the same rendering model as ours, resulting in visual quality similar to ours. Besides, DiffTalk exhibits more artifacts than our approach and loses some facial details of the subject. Moreover, PD-FGC and PC-AVS yield results that compromise the subject’s identity and display significant flaws. Meanwhile, IP-LAP generates facial details that are blurrier than ours and exhibits artifacts particularly under extreme head poses. Similarly, Wav2Lip produces blurry results with unrealistic mouth details.

To further validate the effectiveness of our method, we conduct a user study from 12 volunteers following the common practice of previous methods. We randomly generate 6 videos of the HDTF [14] and VoxCeleb [40] datasets for each method. The volunteers are asked to give their scores (0-5) for each generated video of all comparison methods regarding lip sync quality and visual quality. We present the mean opinion scores (MOS) of each method in  Table II. Our method receives better scores from participants across two dimensions than other methods, indicating the superiority of our method.

We have submitted a supplementary video in the review system. The supplementary video demonstrates the comparison results of all methods in several subjects from VoxCeleb [40] and HDTF [14] datasets. Please download and watch it. If there are any issues downloading the video from the review system, the same file can also be downloaded via this single-blind backup link.

The main contribution of the proposed method is the introduction of the novel audio-aware style reference scheme, which aggregates speaking style information by exploring the relationship between the input audio and the reference audios. Actually, the effectiveness of this has been verified by the fact that the LipLMD score of our method is significantly lower than that of StyleTalk as shown in Table I.

Furthermore, to verify the role of the reference audio, we devise a model variant where the reference audios from the style reference video are not utilized for style aggregation like previous methods [6, 13]. Specifically, the Transformer encoder for encoding the reference audio is removed from the proposed framework, and the reference lip features encoded from the reference lip shapes act as both keys and values for the cross-attention layers in transformer decoder. We report the result of this variant in the row "Ours w/o Reference Audio", and its LipLMD score increases by 6.5% compared to our full model’s. Besides, as shown in Figure 5, without the reference audio, the lip sync quality gets worse. This verifies the necessity to explore the relationship between reference and input audio for accurate and style-preserving lip sync.

As the mouth-related 3DMM expression parameters may involve some information unrelated to lip shape, we apply the reconstruction loss over the lip vertices converted from the predicted expression parameters. To verify the effectiveness of this for enhancing lip sync, we devise a variant where the vertices loss in Equation 4 is not utilized for training the lip motion prediction network. The result of this variant is reported in the row "Ours w/o Vertices Loss" of Table III. The LipLMD increases by 5.9% without using the vertices loss. As well, from Figure 5, the lip sync accuracy becomes worse without the vertices loss for optimization.

To evaluate the effectiveness of incorporating the style reference video for modeling the speaking style of individuals, we introduce a variant that predicts lip motion solely from input audio without utilizing the style reference video. In this variant, the reference branch of our lip motion prediction network is removed, and we replace the cross-attention layers in the transformer decoder with self-attention layers. The performance of this variant is reported by the row labeled “Ours w/o Reference ($n$=0)” in Table III. It is evident that the LipLMD score significantly increases without modeling the speaking style.

Additionally, to assess the impact of the length $n$ of the style reference video, we varied $n$ during inference and reported results for $n=64$ and $n=128$ in Table III. Increasing the length of the style reference video leads to improved LipLMD metrics. Besides, as shown in Figure 5, with the length $n$ of the style reference video decreasing, the lip sync quality gradually deteriorates.