Playmate: Flexible Control of Portrait Animation
via 3D-Implicit Space Guided Diffusion

Talking face video generation has been a long-standing challenge in computer vision and graphics. The goal is to synthesize lifelike and synchronized talking videos from driving audio and static reference images. Early GAN/NeRF-based approaches, such as Wav2Lip (Prajwal et al., 2020), Dinet (Zhang et al., 2023b), and VideoReTalking (Cheng et al., 2022), primarily focused on achieving high-quality lip-sync while keeping other facial attributes static. Consequently, these methods fail to capture strong correlations between the audio and other facial attributes, such as facial expression and head movements. To address this limitation, GANimation (Pumarola et al., 2018) introduced an unsupervised method to generate talking videos with a specific expression. EAMM (Ji et al., 2022) synthesizes emotional talking faces with augmented emotional source videos. More recent studies have typically employed intermediate motion representations (e.g., landmark coordinates, 3D facial mesh, and 3DMM) to generate videos from audio (Zhang et al., 2023a; Gan et al., 2023; Peng et al., 2024). However, such approaches often generate inaccurate intermediate representations, which restricts the expressiveness and realism of the resulting videos. In contrast, our framework generates accurate motion representations based on diffusion transformer.

Diffusion models have shown impressive performance across various vision tasks. However, previous (Bigioi et al., 2024; Mukhopadhyay et al., 2024; Shen et al., 2023) attempts to utilize diffusion models for generating talking heads have only yielded neutral-emotion expressions, leading to unsatisfactory results. Some of the latest methods have some optimizations for this purpose, such as EMO (Tian et al., 2025), Hallo (Xu et al., 2024a), Echomimic (Chen et al., 2024), and Loopy (Jiang et al., 2024b). EMO introduces a novel framework that ensures consistency in audio-driven animations across video frames, thereby enhancing the stability and naturalness of synthesized speech animations. Hallo contributes a hierarchical audio-driven method for animating portrait images, tackling the complexities of lip synchronization, expression, and pose alignment. MEMO (Zheng et al., 2024) proposes an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Several of the above methods can generate vivid portrait videos by fine-tuning pre-trained diffusion models. However, they usually use coupled latent spaces to represent facial attributes in relation to the audio. Facial attributes such as expression and head posture are often generated directly from audio cues. This coupling limits the ability to customize control over certain facial attributes, such as pose and expression. In Playmate, we leverage a 3D implicit space that decoupled various facial attributes, enabling diverse and controllable facial animations while maintaining high accuracy in lip synchronization.

Facial representation learning has been extensively studied in previous works. Various methods (Siarohin et al., 2019; Ren et al., 2021; Li et al., 2017) disentangled variables using 3DMM, sparse keypoints, or FLAME to explicitly characterize facial attributes. Furthermore, in the field of audio-driven portrait animation, several studies have introduced facial representation techniques to generate lifelike talking videos. Sadtalker (Zhang et al., 2023a) separates generation targets into different categories, including eye blinks, head poses, and lip-only 3DMM coefficients. Recent works such as VASA-1 (Xu et al., 2024b), Takin-ADA (Lin et al., 2024b), DreamTalk (Ma et al., 2023b), and JoyVASA (Cao et al., 2024) have begun to combine face representations with diffusion models to achieve more naturalistic results. Inspired by these advancements, we similarly introduce face representation techniques to generate more natural and controllable talking videos.

Facial representation aims to construct a latent face space that exhibits high degrees of expressiveness and disentanglement. Typically, this approach separates various aspects of facial data into distinct components, such as appearance features and motion attributes. In the field of audio-driven portrait animation, many studies have leveraged these latent spaces to generate talking heads. For example, VASA-1 based its model on the 3D-aid face reenactment framework from (Wang et al., 2021; Drobyshev et al., 2022), while Takin-ADA and JoyVASA constructed their latent space based on face-vid2vid and LivePortrait, respectively. Similarly, we adopt and enhance the decoupled facial representation proposed by face-vid2vid and LivePortrait.

LivePortrait primarily comprises an appearance feature extractor $\mathcal{F}$, a motion extractor $\mathcal{M}$, a warping module $\mathcal{W}$, and a decoder $\mathcal{G}$. When presented with a source image $I_{s}$ and a driving image $I_{d}$, LivePortrait initially utilizes $\mathcal{F}$ to extract appearance feature $f_{s}$ from $I_{s}$, and separately captures the motion information from both $I_{s}$ and $I_{d}$ using $\mathcal{M}$. Subsequently, it leverages the motion information extracted from $I_{d}$ to animate $I_{s}$ using $\mathcal{W}$ and $\mathcal{G}$, ensuring that the animated result retains the original appearance while adopting the motion cues from the driving image. The motion information is characterized by canonical keypoints $x_{c}\in\mathbb{R}^{K\times 3}$, expression deformations $\delta\in\mathbb{R}^{K\times 3}$, a rotation matrix $R\in\mathbb{R}^{3\times 3}$, a translation vector $t\in\mathbb{R}^{3}$, and a scaling factor $s\in\mathbb{R}$. The transformation in LivePortrait is formalized as:

where $x_{c,s}$ represents the canonical keypoints of the source image $I_{s}$, and $x_{s}$ and $x_{d}$ are the source and driving 3D implicit keypoints after transformation. Additional information about this framework can be found in reference (Guo et al., 2024).

Similar to VASA-1, we found that the original LivePortrait also suffered from poor disentanglement between facial dynamics and head pose. To address this issue, we introduce the pairwise head pose and facial dynamics transfer loss used in VASA-1 to improve its disentanglement. Specifically, let $I_{i}$ and $I_{j}$ be two randomly sampled frames from the same video clip. Then we transfer the head pose of $I_{j}$ onto $I_{i}$, resulting in $\hat{I}_{i,j^{pose}}=\mathcal{G}(\mathcal{W}(f_{i},x_{i},x_{i,j^{pose}}))$, the transformation is formalized as:

Similarly, we can transfer $I_{i}$’s expression onto $I_{j}$, yielding $\hat{I}_{j,i^{exp}}=\mathcal{G}(\mathcal{W}(f_{j},x_{j},x_{j,i^{exp}}))$, and the transformation is formalized as:

Ultimately, a perceptual loss (Johnson et al., 2016) is employed between $\hat{I}_{i,j^{pose}}$ and $\hat{I}_{j,i^{exp}}$ o enhance the disentanglement between facial dynamics and head pose, the loss function as:

where $\mathcal{V}$ denotes the feature extractor of VGG19 (Simonyan, 2014).

After constructing the latent face space, we can extract motions by utilizing the frozen $\mathcal{M}$ and train the motion generator, with audio cues serving as the condition.

Adaptive Normalization. To achieve a more effective decoupling of expression and head pose, we employ adaptive normalization by using different means and standard deviations when training the motion generator. Specifically, for expression, we compute the global mean and standard deviation using the entire training dataset, as:

where $N_{i}$ is the number of frames in the $i$-th video clip, and $M$ is the total number of all training samples. For head pose, we treat each identity independently and compute the private mean and standard deviation for each sample along the time dimension, as:

where $\rho$ denotes the head pose information. By combining adaptive normalization with the transfer loss mentioned in 3.1, we achieve better decoupling of motion. We refer to this approach as the motion-decoupled module.

Speech Representation. Extensive research has demonstrated that pre-trained speech models, such as Wav2Vec2 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021), outperform traditional features like MFCC in performance. Similar to other methods, we utilize Wav2Vec2 as our speech encoder to extract audio features. Additionally, we introduce a self-attention module to align audio features with motion features.

Diffusion Transformer. The architecture of the Diffusion Transformer is illustrated in Figure 2. We employ four specialized Proj modules, primarily composed of fully connected layers, to extract semantic information from four unique features, aligning this information across different modalities. Next, we utilize a multilayer Conformer (Gulati et al., 2020), a Pose-MLP, and a Exp-MLP in combination with the diffusion formulation to generate motion sequences. Diffusion models define two Markov chains: the forward chain progressively adds Gaussian noise to the target data, while the reverse chain iteratively refines the raw signal from the noise. During training, we gradually transform clean motion $m$ into Gaussian noise $m_{t}$ following the principles of Denoising Diffusion Probabilistic Models(DDPM) (Ho et al., 2020). The Diffusion Transformer is then trained to reverse this noise-adding process by taking $m_{t}$ and other conditional features as input and predicting the added noise $\epsilon$. The objective function for training can be expressed as:

where $f_{a}$, $f_{id}$ are the audio feature and identity feature, $\hat{\epsilon}_{\theta}$ represents the noise prediction made by the Diffusion Transformer.

After completing the first training stage, we obtain a diffusion transformer that generates motion sequences guided by audio cues. To enhance the controllability of emotions in talking head videos, we propose the emotion-control module. This involves fixing the parameters of the diffusion transformer and training an emotion controller based on the DiT block (Peebles & Xie, 2023).

As shown in Figure 2 and Figure 3, the emotion-control module consists of two DiT blocks. The first block receives inputs from the conformer and the emotion condition, while the second block takes the output of the first block and emotion condition as its inputs. The outputs of the second block are then passed to Exp-MLP, replacing the original conformer’s output. Through the aforementioned operations, we encode emotion control information into the features and pass them to the Exp-MLP module, thereby achieving the ability to generate talking videos with desired emotion.

During the second phase of training, we freeze the parameters of the diffusion transformer and train only the parameters of the emotion-control module. Experimental results demonstrate that this approach effectively maintains lip-sync accuracy while integrating emotion conditions, allowing for precise emotion control in the generated animations.

Classifier-free guidance(CFG). We adopt a classifier-free guidance (Ho & Salimans, 2022) approach following (Brooks et al., 2023), which has been successfully applied to image generation from multiple conditions. During training, a random dropout strategy is applied to each of the input conditions to improve the model’s robustness and generalization. During inference, we apply:

where $w_{a}$ and $w_{e}$ are the guidance scales for audio condition and emotion condition, respectively.

Datasets. We utilize a mixture of datasets, including AVSpeech (Ephrat et al., 2018), CelebV-Text (Yu et al., 2023), Acappella (Montesinos et al., 2021), MEAD (Wang et al., 2020), MAFW (Liu et al., 2022), and a talking video dataset collected by us to train our Playmate. In the first stage, we selected approximately 80,000 video clips from the AVSpeech, CelebV-Text, Acappella, and our own dataset to train the diffusion transformer. For the second phase, we selected approximately 30,000 emotionally labeled video clips from the MEAD, MAFW, and our own dataset to train the emotion control module. The duration of each training video ranges from 3 to 30 seconds. We set aside a portion of video clips from our dataset that were not involved in the training process as our out-of-distribution test set. Besides, we choose HDTF (Zhang et al., 2021) as our out-of-domain dataset , which comprises about 362 different videos with original resolution of 720P or 1080P.

Implementation Details. In our experiments, the videos are initially converted to 25 fps and subsequently cropped to a resolution of 256 × 256 pixels based on face landmarks extracted using InsightFace (Deng et al., 2020; Guo et al., 2021a). The final output resolution is set to 512 × 512 pixels. During preprocessing, the audios were resampled to 16kHz. The first training phase utilized four NVIDIA A100 GPUs over a 3-day period, with models initialized from scratch. In the second phase, we continued training for two days with two NVIDIA A100 GPUs, while freezing the parameters of the diffusion transformer. For all experiments, we employed the Adam optimizer (Kingma, 2014). In the inference phase, multi-condition CFG is performed. The CFG scales of the audio $w_{a}$ and the emotion condition $w_{e}$ are set to 1.5.

Evaluation Metrics. We demonstrate the superiority of our method using multiple widely recognized metrics from previous studies. Specifically, we employ Fréchet Inception Distance (FID) (Heusel et al., 2017) and Fréchet Video Distance (FVD) (Unterthiner et al., 2019) to evaluate the quality of the generated data. For evaluating lip synchronization and motion fluidity, we compute the confidence score (Sync-C) and feature distance (Sync-D) using the pretrained SyncNet (Chung & Zisserman, 2017). Additionally, we compute the cosine similarity (CSIM) of identity vectors extracted using the ArcFace (Deng et al., 2019) face recognition model to evaluate identity preservation. Furthermore, we leverage Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) to measure the feature-level similarity between the generated faces and their ground-truth faces.

Quantitative Results. We benchmark our method against SOTA audio-driven portrait animation methods, including Hallo (Xu et al., 2024a), Hallo2 (Cui et al., 2024), JoyVASA (Cao et al., 2024), MEMO (Zheng et al., 2024), and Sonic (Ji et al., 2024). As shown in Table 1, our Playmate significantly outperforms other methods in terms of FID, FVD, CSIM, and LPIPS on two test datasets, while also exhibiting strong competitiveness in lip synchronization. Regarding video quality, our method achieves the lowest FID and FVD scores on both test sets. Specifically, on the HDTF dataset, our FID and FVD scores are 30% and 20% lower than those of the second-best method, respectively, indicating superior video quality compared to other methods. For the CSIM and LPIPS metrics, we also achieve the best results, indicating superior performance in identity preservation and image quality. Additionally, our method achieves good results in Sync-C and Sync-D, both of which are second-best, exhibiting strong competitiveness.

Qualitative Results. Figure 4 provides a visualization comparison using open datasets. Upon analysis, previous methods were prone to generate artifacts in tooth rendering(e.g., (a)-row 6, column 3; (b)-row 3, column 1) and lip synchronization(e.g., (a)-row 4, column 7; (b)-row 2, column 7). Conversely, our approach boasts a superior decoupling capability, which allows it to create more lifelike talking head videos. For more comparison details, please see the Appendix. Since images cannot adequately reflect important aspects such as synchronization, naturalness, and stability, the full video comparison will be included in the supplementary materials or on our project URL.

Visualization Results in Different Style Images. We further investigate the generation performance of our method in different style images. Given that our Playmate can directly generate motion sequences from audio, it is capable of driving a wide range of portraits, including real humans, animations, artistic portraits, and even animals. Although this versatility may lead to characters having similar expressions, by integrating the emotion-control module and adjusting the CFG weight, we can easily achieve diverse expressions and movements. As illustrated in Figure 5, our method successfully generates a wide range of expressions and diverse movements across various style images, underscoring its robust performance.

Emotion Control. Figure 6 shows the generation results of our method under different emotional conditions using the same audio clip. These examples clearly illustrate our model’s proficiency in encoding emotional signals into latent space and producing talking face animations with the desired emotion. This highlights Playmate’s effectiveness in achieving precise emotional control while maintaining natural and lifelike results.

Adaptive Normalization. We analyze the effectiveness of our proposed Adaptive Normalization in decoupling expressions and head pose. As shown in Figure 7, without adaptive normalization, the generated results are prone to pose jittering, inaccurate lip-sync, and strange expressions, resulting in temporal discontinuities and artifacts within the final video. After introducing adaptive normalization, where head pose feature and expression feature are normalized using different mean and variance values, the model can independently identify and distinguish pose feature and expression feature. Consequently, the final generated videos maintain better continuity in both pose and expression.

CFG scales. By adjusting the CFG scales, we can strike a balance between the quality and diversity of the generated results. In Table 2, we evaluate the selection of CFG scales for the audio and emotion conditions (represented by $w_{a}$ and $w_{e}$ in Equation 8) within our model. Firstly, we conduct ablation experiments solely on $w_{a}$, as shown in the upper part of Table 2. When $w_{a}$ is set to 1.5 or 2.0, Playmate achieves better evaluation metric scores. Upon comprehensive consideration, we set $w_{a}$ to 1.5 and conduct an additional ablation experiments on $w_{e}$. In order to better measure the impact of $w_{e}$, we introduce an additional emotion classifier from (Savchenko, 2023) to calculate the emotion accuracy of the generated videos, so that we can better select an appropriate value for $w_{e}$. As shown in the bottom part of Table 2, when $w_{a}$ is set to 1.5, setting $w_{e}$ to 1.5 as well can achieve more balanced results across various testing metrics. Therefore, we also set $w_{e}$ to 1.5.