Exploring Phonetic Context-Aware Lip-Sync
For Talking Face Generation

Audio-to-Lip module $f_{\theta}$ translates a given input audio into lip representation by considering the phonetic context. Motivated by the great success of masked prediction [21, 22, 23, 24, 25, 26, 27] in context modeling, we bring its concept into our learning problem to capture the phonetic context from the input audio when synthesizing the lip movements. Specifically, the audio input is processed into a sequence of mel-spectrograms which we denote as audio units $\mathbf{A}_{T}=\{a_{t}\}_{t=0}^{T}$, where $a_{t}$ is a frame-level mel-spectrogram at $t$-th frame. Then, we corrupt the audio units as follows:

$$\mathcal{\tilde{\mathbf{A}}}_{T}=r(\mathbf{A}_{T},M),$$

(1)

in which $M\subset[0,T]$ denotes the set of indices to be masked for the audio units $\mathbf{A}_{T}$, and $r$ is a masking function that zero-out the audio unit $a_{t}$ where $t\in M$. As shown in Fig.1, we mask out a continuous sequence of $t_{m}$ audio units with probability $p$, without overlap between each segment. Audio-to-Lip module $f_{\theta}$ translates the corrupted audio units $\tilde{\mathbf{A}}_{T}$ into contextualized lip motion units $\tilde{\mathbf{e}}_{T}=\{\tilde{e}_{t}\}_{t=0}^{T}$,

$$\tilde{\mathbf{e}}_{T}=f_{\theta}(\tilde{\mathbf{A}}_{T}),$$

(2)

where the Audio-to-Lip module $f_{\theta}$ is designed with transformer [20] architectures so the short-term and long-term context can be modeled. Then, it is guided to predict the corresponding ground truth lip motion units $\mathbf{Z}^{v}_{T}=\{z^{v}_{t}\}_{t=0}^{T}$ of the masked timestep $t\in M$ with the following loss function:

$$\mathcal{L}_{a2l}=\sum_{t\in M}(\mathbf{z}^{v}_{t}-\tilde{\mathbf{e}}_{t})^{2}.$$

(3)

As our objective is producing synchronized lip motion from input audio by fully utilizing the phonetic context, the prediction target for our masked prediction should be carefully addressed. To handle this, we leverage a large-scale pre-trained visual encoder [28, 29, 30], whose feature is set to our target for the masked prediction. The visual encoder is pre-trained using contrastive objectives between video frames and the corresponding audio frames. Thus, by predicting its visual features, the Audio-to-Lip module can focus on modeling synchronized lip movements from the input audio while minimizing the influence of speaker characteristics in the speech. The visual encoder extracts lip motion units from lip frames $\mathbf{V}_{T}=\{v_{t}\}_{t=0}^{T}$ in a sliding window of 5 frames, where the timestep $t$ is positioned at the center of the window. Hence, the lip motion units are discretized to exactly match the audio units. In order to predict the lip motion from the masked audio inputs, the Audio-to-Lip module has to first consider relations between the phones to predict the masked phones, and then map the phones to the synchronized visual lip features. As a result, the Audio-to-Lip module translates audio to visual lip intermediate representations, and associates phonetic context while establishing the audio-lip correlation.

Lip-to-Face module $f_{\phi}$ synthesizes face frames with the context-aware lip motion units drawn from the Audio-to-Lip module. Since it leverages the audio-lip correlation established in the Audio-to-Lip module, the Lip-to-Face module only has to focus on the synthesis part, and impose the dynamics of the lips onto the target identity. The Lip-to-Face module generates corresponding face frames $\mathbf{\hat{I}}_{T}$ within the context in parallel as follows:

$$\mathbf{\hat{I}}_{T}=f_{\phi}({\mathbf{e}}_{T},\mathbf{f}^{I}_{T}).$$

(4)

$\mathbf{f}^{I}_{T}$ are corresponding identity features encoded by an identity encoder $E_{I}$. $E_{I}$ takes a random reference frame concatenated with a pose-prior along the channel dimension as in [15, 16]. The pose-prior is a target face with lower-half masked, which guides the model to focus on generating the lower-half mouth region that fits the upper-half pose. The lip motion units from the Audio-to-Lip module guide the Lip-to-Face module to generate lip shapes that are more distinctive to its phone in context. Moreover, as the lip motion units have attended to every other phones in the context, the dynamics are more temporally stable and consistent.

We perform two experiments to analyze to what extent context contributes to lip synchronization. First, we mask out 0.14s (7 frames) of the source audio and generate frames of the masked time steps. As shown in Fig.2, our work can still generate well-synchronized lips even with the absence of audio because it is able to attend to the surrounding phones and predict lip motion that fits into the masked region in context. In contrast, the previous works cannot generate correct lip movements because they do not consider surrounding phones. Such results verify that our model effectively incorporates context information in modeling lip movement of the talking face. In addition, we generate varying sizes of the audio window. We take a frame in the middle as a target frame and increase the size of the input audio window at the frame-level. The maximum audio window for the LRW is $\pm$15 and LRS2 is $\pm$78. The audio frames that lie outside the window are zero-padded and we measure the lip-sync quality of the generated target frame using LMD. As shown in Fig.3, on the LRW dataset, taking the entire audio window yields the best lip-sync performance. It reaches 1.162 in LMD at $\pm$15 frames. But when we further experiment on the LRS2 using wider audio windows, we find that the effect of temporal audio information reaches the optimum 1.059 at around $\pm$13 frames. It demonstrates that the audio context of around 1.2 seconds assists in resolving ambiguities in the lip shapes of phones, improving spatio-temporal alignment.

We conduct an ablation to analyze the effect of each component of CALS in Table 1. We have set the baseline as Wav2Lip and implemented Audio-to-Lip module (A2L), $d_{av}$ and $d_{vv}$ one by one. When the phone-level audio encoder of the Wav2Lip is replaced by the A2L, the visual quality and sync quality are enhanced, where the SSIM improves by 0.029 and LSE-C by 0.349. Adding $d_{av}$ improves LSE-D and LSE-C because it acts on the audio-visual alignment which the two metrics measure. $d_{vv}$ further enhances PSNR, SSIM and LMD because it aligns in the visual domain. The three components altogether yield the highest performance overall. The result indicates that exploiting the phonetic context in the A2L scheme alone has the largest effect in improving the generation overall.

We quantitatively compare the generation results with 4 state-of-the-art methods: Audio2Head [36], PC-AVS [37], Wav2Lip [15], and SyncTalkFace [16] on LRW, LRS2, and HDTF datasets in Table 3. On LRW and HDTF, our method achieves the best on all the metrics, especially outperforming on the lip-sync. Compared to other methods that involve feature disentanglement (PC-AVS), assistant module (SyncTalkFace, Wav2lip), and intermediate structural representations (Audio2Head), we validate that exploiting phonetic context in modeling lip motion is a more powerful scheme to achieve accurate lip synchronization.

We qualitatively compare the generation results in Fig.4. Our method is most precisely aligned in spatio-temporal dimension and generates the most temporally stable lips, distinct to each phone in context. For example in (a), when pronouncing `k@m' in `combating', our method wide opens the mouth and gradually closes with smooth transitioning. On the other hand, other methods are temporally misaligned and discontinuous: Audio2Head fails to completely close the mouth at `m', Wav2Lip closes its mouth but with slightly projected lips, and SyncTalkFace fails to clearly open the mouth at phone `k'. The man in (b) is pronouncing `æskt’ in `asked for’. Our method is the only work that successfully captures the slightly projected lips at `t’ transitioning into `f’. Such results demonstrate that our method fully makes use of the phonetic context for temporally aligned and consistent lip synchronization.

We conducted a user study to compare the generation quality in Table 2. We generated 21 videos using each of the methods and asked 15 participants to rate the videos in terms of visual quality, lip-sync quality, and realness in the range of 1 to 5. Our method has the highest visual quality, lip-sync quality, and realness score, especially outperforming in the lip-sync quality, which is also reflected in the quantitative and qualitative results in Table 3 and Figure 4.