Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis

The talking face synthesis is an eagerly anticipated technique for several applications, such as filmmaking, teleconference, virtual/mix reality, and human-computer interaction. One of the key essences behind the talking face synthesis is the stylization of facial and head pose movements. Different from the talking emotion which reflects in the short-term facial motions, the talking style is a crucial factor which affects long-term facial and head pose movements. People usually talk with diversified talking styles such as ”excited”, ”solemn”, ”communicational”, ”storytelling”, et al. Given one piece of speech, different talking styles exhibit significant differences in the facial and head pose movements. For example, as shown in Figure 1, people with the ”excited” style usually talk aloud, and thus the facial movement of the mouth wide open occurs frequently. Meanwhile, the ”solemn” talking style usually appears in formal occasions, and thus the exaggerated motion seldom occurs. Considering such huge differences between different styles, in order to synthesize diversified and realistic talking faces with respect to one piece of speech, it is necessary to incorporate talking style into the audio-driven talking face synthesis framework.

Previous efforts (Cudeiro et al., 2019; Yi et al., 2020) have shown the rationality of synthesizing stylized talking faces. Yi et al. (Yi et al., 2020) proposed Memory Augmented GAN model to synthesize stylized talking face with the wild training data. Cudeiro et al. (Cudeiro et al., 2019) proposed Voice Operated Character Animation (VOCA) model to learn the identity-wise talking style. The VOCA model captures the talking style of each identity by injecting the one-hot identity vector into the audio-motion predicting network. Overall, these methods have the following two disadvantages which constrain the expressiveness of the synthesized talking styles: (1) These methods assume that each identity has only one talking style. However, in the real scenario, the talking style is only relatively stable inside one video clip, one person can talk with significantly different styles among different video clips. (2) These methods require substantial labour to collect adequate synchronized audio-visual data for each identity, which is unavailable for the wild scenarios.

To address the aforementioned problems, we propose to synthesize stylized talking faces by imitating styles from arbitrary clips of videos. With such motivation, the talking videos with stable and diversified talking styles are in need. Therefore, we collect the Ted-HD dataset which has 834 clips of videos with 60 identities. Each video clip has an average length of 23.5 seconds. The talking styles of the Ted-HD dataset are diversified, and the talking style inside each video clip is stable.

Based on the constructed dataset, we devise a two stage talking face synthesis framework as shown in Figure 2. The first stage imitates talking styles from arbitrary videos and synthesizes 3D talking faces according to the driven speech. Afterwards, in the second stage, we render the 3D face model photo-realistically from one static portrait of the speaker. Overall, the key idea of our framework is to construct style codes from video clips in the wild, and then synthesize talking faces by imitating the talking styles from the constructed style codes. Specifically, for the style codes construction, we conduct exhaustive observations on the Ted-HD dataset, which on one hand verify that even one identity has multiple talking styles, on the other hand find that the talking style is closely related to the variance of facial and head pose movements inside each video. With our observations, we define the style codes as several interpretable statistics of 3D morphable model (3DMM) (Blanz and Vetter, 1999) parameters. Having obtained the style codes of each talking video, we devise a latent-style-fusion (LSF) model to synthesize stylized 3D talking faces by imitating talking styles from the style codes. Detailedly, the LSF model first dropouts (Srivastava et al., 2014) information from the audio stream to prevent the audio from dominating the synthesis process. Further, the LSF model fuses the style codes with the latent audio representation frame-by-frame to synthesize 3D talking face with corresponding talking style. The overall implementation of the LSF model is simple but effective. Our model not only circumvents the annotation for talking styles and avoids collecting substantial training data for each identity, but also enables new talking style generation.

With our proposed framework, talking faces with diversified styles can thus be synthesized. We perform experiments on the Ted-HD dataset for evaluation. Compared with baseline methods, our framework synthesizes more expressive and diversified talking styles. We conduct extensive user studies to investigate the facial motion naturalness and audio-visual synchronization. With the mean opinion score (MOS) of 20 participants, our framework outperforms baseline methods by 0.67 on average in terms of facial motion naturalness and 0.11 on average in terms of audio-visual synchronization.

To summarize, our contributions are summarized as three-fold:

• •

  We propose to synthesize stylized talking faces by imitating talking styles of arbitrary videos. The incorporation of style imitation leads to more diversified talking styles.

• •

  We formalize the style codes of each talking video and devise the latent-style-fusion (LSF) model to synthesize stylized 3D talking faces from style codes and driven audio. Our framework do not require any annotation of talking styles for stylized talking face synthesis.

• •

  We collect Ted-HD dataset, which contains 834 clips of wild talking videos with stable and diversified talking styles. Based on the Ted-HD dataset, we conduct extensive style observations and synthesize expressive talking styles. Code and dataset are publicly available ¹¹1https://github.com/wuhaozhe/style_avatar.

Talking face synthesis has received significant attention in previous literatures. Related work in this area can be grouped into two categories: the unimodal talking face synthesis (Thies et al., 2020; Zhou et al., 2019; Chen et al., 2018; Wiles et al., 2018; Prajwal et al., 2020; Jamaludin et al., 2019; Zhou et al., 2021) and the multimodal talking face synthesis (Wang et al., 2020a; Emre Eskimez et al., 2020; Cudeiro et al., 2019; Yi et al., 2020; Zeng et al., 2020; Ji et al., 2021). For one piece of driven speech, the unimodal talking face synthesis generates unique motion, while the multimodal talking face synthesis generates diversified facial and head pose movements.

Most of the prior works focused on the unimodal talking face synthesis. Karras et al. (Karras et al., 2017) proposed to synthesize 3D talking face with driven audio and emotion state. Suwajanakorn et al. (Suwajanakorn et al., 2017) synthesized high quality videos of talking Obama through hours of Obama’s weekly address footage. Since that Suwajanakorn’s method requires hours of data for each identity, several methods (Thies et al., 2020; Zhou et al., 2020; Zhou et al., 2019; Prajwal et al., 2020; Chen et al., 2019, 2018; Yu et al., 2020; Chen et al., 2020) proposed to simultaneously reduce the required duration of training data and guarantee the photorealism of the synthesized video. The ATVG framework proposed by Chen et al. (Chen et al., 2019) and the DAVS framework proposed by Zhou et al. (Zhou et al., 2019) synthesize talking face with only one image. Despite that these unimodal talking face synthesis methods can synthesize photo-realistic videos, the lack of style results in the stiffness of the synthesized results.

To synthesize diversified facial and head pose movements, some recent literatures addressed the multimodal talking face synthesis. Wang et al. (Wang et al., 2020a) and Eskimez et al. (Emre Eskimez et al., 2020) achieved multimodal synthesis through incorporating emotion condition vector, which enables the generation of diversified facial expressions. However, the synthesized results of these methods still lack in personality because of the overlook of talking style. To address this issue, some methods (Yi et al., 2020; Cudeiro et al., 2019) proposed to incorporate talking style into the synthesis framework. Yi et al. (Yi et al., 2020) proposed Memory-Augmented GAN model to synthesize stylized talking face with wild training data. However, only the head pose is synthesized with multiple styles, while the facial movements still lack personality. Further, Cudeiro et al. (Cudeiro et al., 2019) proposed the Voice Operated Character Animation (VOCA) model to learn the identity-wise talking style. The VOCA model injects one-hot identity vector into the audio-motion predicting network, leading to discriminative styles of facial and head pose movements. However, the VOCA model on one hand requires substantial data for each identity, on the other hand forces one identity to have only one talking style, limiting its capacity on synthesizing diversified styles. To resolve this problem, in this work, we propose to imitate talking styles from arbitrary wild talking videos.

In this paper, we propose a two-stage talking face synthesis framework which synthesizes stylized talking videos with the following three inputs: one static portrait of the speaker, the driven audio and the style reference video. We formalize the first stage of the framework as the 3D talking face synthesis stage and the second stage as the photorealistic render stage. Between two stages, we apply the 3DMM face model (Blanz and Vetter, 1999) as a crucial bridge. Therefore, before formally defining the two stages, we firstly give a brief introduction of the face model we use.

We leverage the 3DMM face model to represent each 3D face. With 3DMM, the face shape $\mathbf{S}$ is represented as an affine model of facial expression and facial identity:

where $\mathbf{\bar{S}}\in\mathbf{R}^{N\times 3}$ is the average face shape; N is the number of the vertexes in the face model; $\mathbf{B}_{id}$ and $\mathbf{B}_{exp}$ are the PCA bases of identity and expression; $\alpha$ and $\beta$ are the identity parameters and the expression parameters. Following deng et al. (Deng et al., 2019), we adopt the 2009 Basel Face Model (Paysan et al., 2009) for $\mathbf{\bar{S}}$, $\mathbf{B}_{id}$, and use expression bases $\mathbf{B}_{exp}$ of Guo et al. (Guo et al., 2018) built from Facewarehouse (Cao et al., 2013), resulting in $\alpha\in\mathbb{R}^{80}$, $\beta\in\mathbb{R}^{64}$. Afterwards, the 3D face shape is projected on the 2D plane according to the head pose and translation $p\in\mathbb{R}^{7}$, where 4 elements represent the pose quaternion and 3 elements represent the translation. Overall, the parameter set $(\alpha,\beta,p)$ controls the appearance of each face. In our framework, the facial movements are the time series of parameter $\beta$, which we denote as $\beta(t)$, while the head pose movements are the time series of parameter $p$, which we denote as $p(t)$. With $\beta(t)$ and $p(t)$, we formalize the two stages of our framework as follows.

3D Talking Face Synthesis Stage. In this stage, given driven audio $\mathbf{X}_{a}$, the facial and head pose movements of style reference video $\beta_{sty}(t),p_{sty}(t)$, we aim to generate corresponding facial and head pose movements $\beta_{pred}(t)$, $p_{pred}(t)$.

Photorealistic Render Stage. In this stage, given the predicted movements $\beta_{pred}(t)$, $p_{pred}(t)$ and the input portrait $\mathbf{X}_{p}$, we aim to generate photorealistic videos $\mathbf{Y}$.

In this section, we systematically investigate how different talking styles reflect in the facial and head pose movements $\beta(t),p(t)$. Afterwards, we formally define interpretable style codes for each video based on our observation.

In order to observe talking styles of each video, we should firstly collect a suitable dataset for observation. The video for the style observation requires the following characteristics: (1) It should be of high resolution, (2) it should contain natural and expressive facial and head pose movements, (3) each clip of video cannot be too short, or the talking style can hardly be observed, (4) the talking style should both be stable inside the clip and be diversified across different clips, (5) the camera pose and location should be static inside each clip, otherwise the head pose parameters will be influenced by the camera movements. Considering the characteristics above, current publicly available wild datasets like VoxCeleb2 (Chung et al., 2018) and LRS3 (Afouras et al., 2018) are much too noisy and thus do not meet the requirements. Meanwhile, several in-lab datasets like MEAD (Wang et al., 2020b) and GRID (Cooke et al., 2006) do not have natural facial expressions and head poses, which dissatisfy the requirements either.

To address this issue, we manually collect a wild dataset Ted-HD which is suitable for style observation and further style synthesis. The Ted-HD dataset selects several speech videos from Ted website. Each video in the dataset has one person giving a speech, which focuses on the facial part of each person and is of high resolution. We cut each video into several clips according to the scene change. Totally, Ted-HD dataset contains 834 clips of videos with 60 identities. The average length of each clip is 23.5 seconds, and the total duration of the dataset is 6 hours. The talking style of these videos are diversified across different clips. Even for the same identity, there could be different talking styles.

Having obtained the dataset, for each video, we reconstruct the facial and head pose movements $\beta(t),p(t)$. We conduct exhaustive data observations on the correlation between talking style and $\beta(t),p(t)$, with the aim of answering the following questions:

• •

  Q1: Whether one identity has multiple talking styles.

• •

  Q2: How the talking style is reflected in time series $\beta(t),p(t)$.

To answer the Q1, we verify the talking style diversity of each identity through user study with A/B test. Specifically: we first randomly construct 100 triplets, each triplet contains two talking videos $v_{1},v_{2}$ from the same identity and one talking video $v_{3}$ from the other identity. Next, we reconstruct $\beta(t),p(t)$ of $v_{1},v_{2},v_{3}$, retarget $\beta(t),p(t)$ to the same identity and render the retargeted faces to videos. The retargeted videos are signified as $v_{1}^{{}^{\prime}},v_{2}^{{}^{\prime}},v_{3}^{{}^{\prime}}$. Afterwards, we show $v_{1}^{{}^{\prime}},v_{2}^{{}^{\prime}},v_{3}^{{}^{\prime}}$ and their transcripts to users, asking the following question: which pair of $(v_{1}^{{}^{\prime}},v_{2}^{{}^{\prime}})$ and $(v_{1}^{{}^{\prime}},v_{3}^{{}^{\prime}})$ has more similar talking styles. Statistics demonstrate that among 100 triplets, $(v_{1}^{{}^{\prime}},v_{3}^{{}^{\prime}})$ is more similar in 30 triplets, while $(v_{1}^{{}^{\prime}},v_{2}^{{}^{\prime}})$ is more similar in 70 triplets. With the statistics that $30\%$ of videos inside each identity have dissimilar talking styles, we conclude that one identity has multiple talking styles.

Since that the talking style is not consistent for each identity, formulating style codes is necessary. Therefore, in Q2, we conduct experiments to find how talking style is reflected in time series $\beta(t),p(t)$. Specifically, we first randomly select 300 talking videos and annotate the talking style of these videos to three categories: tedious, solemn, and excited. Afterwards, for the motion series $\beta(t),p(t)$ of each video, we calculate its derivative series with respect to time $t$: $(\frac{\partial\beta(t)}{\partial t},\frac{\partial p(t)}{\partial t})$. Next, we calculate the mean value $\mu(\cdot)$ and the standard deviation $\sigma(\cdot)$ of $(\beta(t),p(t),\frac{\partial\beta(t)}{\partial t},\frac{\partial p(t)}{\partial t})$ along time, yielding 8 feature vectors (4 for mean, 4 for standard deviation). To observe the relationship between these feature vectors and talking styles, we leverage t-SNE algorithm (Van der Maaten and Hinton, 2008) to visualize each feature vector and plot points from different style categories with different colors, as shown in Figure 3. Figure 3 demonstrates that the talking style is closely related with $\sigma(\beta(t))$, $\sigma(\frac{\partial\beta(t)}{\partial t})$, $\sigma(\frac{\partial p(t)}{\partial t})$, especially $\sigma(\frac{\partial p(t)}{\partial t})$. Meanwhile the talking style is less related with $\mu(\cdot)$, which denotes that the talking style is mostly reflected in the fluctuation of movements, rather than the idle state of movements.

Based on such observation, we define the style codes as the standard deviation of facial and head pose movements. Formally, given arbitrary video with the reconstructed parameter series $\beta(t),p(t)$, the style codes $\mathbf{sty}$ are defined as:

where $\oplus$ signifies the vector concatenation.

To summarize, we make the following two conclusions: (1) one identity have multiple talking styles, (2) the talking style is closely related to the variance of facial and head pose movements inside each video, following which we define the style codes in Equation 2. The style codes are utilized to synthesize diversified talking styles, details will be illustrated in the section 5.

Following the style codes defined in section 4, we propose a two-stage talking face synthesis framework to imitate arbitrary talking styles, as shown in Figure 2. Our framework synthesizes stylized talking videos with the following three inputs: one static portrait of the speaker, the driven audio and the style reference video. In the first stage of our framework, we devise a latent-style-fusion (LSF) model to synthesize stylized 3D talking faces through imitating arbitrary talking styles. Based on the synthesized 3D talking faces, in the second stage, we leverage the deferred neural render (Thies et al., 2019) and few-shot neural texture generation model to generate video frames photo-realistically. In the next two subsections, we will introduce the two stages respectively.

In this section, we conduct extensive experiments to demonstrate the effectiveness of our framework. We evaluate our framework on the collected Ted-HD dataset. Our method has acquired better synthesis results both qualitatively and quantitatively.

In this paper, we propose the concept of style imitation for audio-driven talking face synthesis. To imitate arbitrary talking styles, we firstly formulate the style codes of each talking video as several interpretable statistics of 3DMM parameters. Afterwards, we devise a latent-style-fusion (LSF) model to synthesize stylized talking faces according to the style codes and driven audio. The incorporation of style imitation not only circumvents the annotation for talking style during the training phase, but also endows the capacity of arbitrary style synthesis and new talking style generation. Additionally, to synthesize expressive talking styles, we collect Ted-HD dataset with 834 clips of talking videos, which contains stable and diversified talking styles. We conduct extensive experiments on the constructed dataset and obtain expressive synthesis results with our Ted-HD dataset and LSF model. The constructed Ted-HD dataset will be made publicly available in the future. We hope that the proposal of talking style imitation and the construction of Ted-HD dataset pave a new way for audio-driven talking face synthesis.

This work is supported by the National Key R&D Program of China under Grant No. 2020AAA0108600, the state key program of the National Natural Science Foundation of China (NSFC) (No.61831022), Beijing Academy of Artificial Intelligence No. BAAI2019QN0302.