ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance

Our method requires the correct preservation of features from different motion datasets, necessitating the unification of various data representations. First, we extract the Euler angles from motion capture data (BVH format) and convert them into rot6D representations corresponding to the 55 joints in SMPL-X [17]. For 3D locations, we randomly extract a skeleton as a reference and align other data to this skeleton to unify the 3D coordinates. We then scale the root node’s displacement based on the extracted skeleton, adjusting the orientation to ensure consistency across all data. Finally, we combine the extracted rot6D rotation representation ($J\times 6$), 3D location ($J\times 3$), linear velocity ($J\times 3$), angular velocity ($J\times 6$), and ground contact signals (4) into a kinematic feature representation, where $J$ represents the number of joints. Each frame of the unified motion sequence has a 994-dimensional feature representation, denoted as $x_{0}\in\mathbb{R}^{N\times 994}$, where $N$ is the number of motion sequence frames. Due to the lack of mixed-modality data, we artificially synthesized some data. Specifically, we considered the lower body (including the root) below the third spine as the locomotion-involved part, and the remaining upper body as the gesture-involved part. We concatenated the two parts, resulting in 20K artificially generated text-audio-motion matching pairs. We used a ratio of (0.6, 0.4) to mix gesture data and artificial data as the training dataset.

ExpGest builds upon the diffusion model [18] to generate co-speech gestures. As illustrated in Figure 2, our approach involves learning how to progressively denoise from pure Gaussian noise to obtain expressive full-body gestures. This process encompasses a forward diffusion step, where noise is added to the original gestures, and a subsequent reverse denoising procedure.
Denoising Diffusion Probabilistic Model. We sample clean gestures $x_{0}$ from the distribution of real gestures $x_{0}\sim q(x_{0})$, where $q(x_{0})$ represents the distribution of the original data. Subsequently, Gaussian noise is incrementally added to $x_{0}$. When the number of noise steps $T$ is sufficiently large, the final noisy gesture $x_{T}$ converges to pure Gaussian noise. Each step is modeled as a forward process denoted by $q$. The forward process diffuses the data samples through Gaussian transitions parameterized with a Markov process:

where $\{\beta_{t}\}_{t=1}^{T}$ is the variance schedule and $\alpha_{t}=\prod_{s=1}^{t}(1-\beta_{s})$.
Then the reverse process becomes $p_{\theta}(x_{0:T}):=p(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t})$, starting from $x_{T}\sim\mathcal{N}(0,I)$ with noise predictor $\epsilon_{t}^{\theta}$:

where $z_{t}\sim\mathcal{N}(0,I)$ and $\sigma_{t}^{2}=\beta_{t}$ means the variance schedule stays constant.
Expressive Gesture Diffusion Generator. The objective of ExpGest is to generate motion-capable talking avatars of arbitrary lengths, accompanied by semantically relevant and expressively rich synchronized gestures. However, unlike the original DDPM [18], we take into account the extreme physical constraints inherent in 3D human bodies and replace the predicted noise with the original human representation, deviating from image generation. As a result, in each step of our denoising process, we reconstruct the original representation from pure Gaussian noise, ultimately producing the final generated result through a cyclic process of noise addition and denoising:

where $c$ is the conditions. As shown in the upper part of Figure 2, our conditions include noise step $t$, seed posture, motion text, audio information, and semantic latent code. The noise step $t$ and seed posture are projected to the same dimensions through MLP and Linear layers, respectively, and then added together. The audio is encoded using WavLM, and to match the number of gesture frames, it is interpolated in the time dimension. The text description is directly encoded into the CLIP space, while the semantic features are obtained through our semantic alignment module. Finally, we perform Huber loss on $\hat{x}_{0}$ reconstructed at each step and $x_{0}$ sampled in the real distribution to optimize ExpGest:

It is worth noting that during the denoising process, if there is only a single modality input, the other modality is directly masked and added to the conditions in the form of $\varnothing$.

In previous gesture generation methods, the crucial aspect of semantic alignment was often overlooked, posing a significant challenge for accurately generating semantically aligned actions due to the many-to-many mapping relationship between audio content and gesture sequences. To tackle this issue, we learn a joint embedding space for gestures and audio transcriptions, allowing their alignment in latent space and uncovering semantic associations between both modalities. As depicted in Figure 3, we initially train gesture and transcription encoders using a motion-VAE structure and a BERT tokenizer [19], respectively. We parameterize transcriptions into tokenized word embedding sequences and linearly map them to a space with the same dimensions as the gesture latent code. We employ global average pooling to extract global information from both modalities and utilize CLIP-style contrastive learning to fine-tune the encoders. The NT-Xent [20] Loss serves as the objective to maximize the similarity of transcription-gesture matching pairs in latent space while minimizing the similarity of non-matching pairs. Formally, the loss function is as follows:

where, $z_{s}$ and $z_{g}$ are the latent space representations of a matching transcription-gesture pair. $sim$ is similarity score between two latent codes, $K$ is a set containing one positive sample transcription and a group of negative sample gestures, and $\tau$ is the temperature parameter used to adjust the sensitivity of the function. Finally, we freeze the trained semantic alignment module and deploy only the transcription encoder into the GDM, ensuring that the final generated results accurately capture the semantic content.

Previous methods transformed emotions into a set of one-hot encodings embedded directly into the conditions, controlling emotions by appending identity. However, this approach fails to capture the underlying relationships and continuity between emotions, making it difficult for the model to learn smooth transitions between different emotions. To address this issue, we introduce a noise emotion classifier that optimizes the denoised noise gesture at each sampling step, moving it towards a human-specified emotional direction. This method is decoupled from the diffusion computation graph, preserving the original semantic content and making it applicable to different structures in the same representation. Specifically, we randomly select emotion-gesture matching pairs and add Gaussian noise of random steps to the gestures, turning them into noisy gestures. We then train the noise emotion classifier based on the noisy gesture-emotion matching pair data. After training, we graft the classifier onto the GDM, input the denoised $x_{t}$ into the GDM, compute the gradient with the specified emotion, and then backpropagate it to $x_{t}$ for optimization. The formula is as follows:

where $\alpha$ is gradient update weight and $y$ is the real emotion label.

Datasets. Our audio data comes from the large-scale, high-quality BEAT dataset [8], featuring 76 hours of multimodal speech data from 30 speakers in 8 emotions and 4 languages. We selected the English data from both presentations and conversations for training. We also used AMASS[21] and 100-STYLE[22] for locomotion training. From AMASS, we selected motion pairs with text descriptions, containing only the body’s SMPL representation and zero-filled hand representations. 100-STYLE, a multi-style motion dataset, includes various actions each with a BVH file and short phrase description. We processed these data into the motion representation described in Sec 2.1 for unified training.
Evaluation Metrics. We use three metrics to evaluate the quality, emotion alignment, and semantic matching of gestures generated by ExpGest. Gesture quality is assessed using Fréchet Gesture Distance (FGD) [23], which calculates the distance between latent feature distributions of generated and real gestures, evaluating gesture quality. Lower FGD values indicate higher motion quality. Additionally, we propose a Semantic Alignment (SA) metric to assess the semantic consistency between generated gestures and audio. We use the trained VAE encoder to encode transcribed audio text and gestures into a shared latent space and evaluate their pre-established similarity as the assessment score.The calculation formula is as follows:

where G represents the gestures generated by the model, and S denotes the hidden states encoded by the BERT [8] model after tokenizing the transcribed text, serving as a representation of the semantics. We also introduce Emotion Alignment (EA) and Emotion Control Success Rate (EC) to evaluate the model’s stylization capabilities. Unlike in Section 2.4, we pre-train a gesture emotion classifier, which doesn’t require adding noise to the gesture motion but directly classifies emotions in the original gestures. EA represents consistency with the true emotions of the audio, while EC denotes the success rate of style transfer for specified emotions. Higher values for both EA and EC indicate more accurate emotion guidance.

We compare our method with three cutting-edge approaches: Trimodal [23], CaMN [8], and DiffStyleGesture [6]. We remove the text guidance describing motion and use an audio-only mode for comparison. Quantitative results in Table I show our method consistently outperforms all other methods across all evaluations. Specifically, in terms of Fréchet Gesture Distance, we achieve state-of-the-art performance in the audio-only mode. Our method improves the feature space by 57.3 (42.7%) and the raw space by 22 (65.2%) in evaluations with only upper-body limbs compared to our baseline. Although there is a slight decrease in the mixed mode, our method still outperforms the baseline. Thanks to the semantic-gesture joint embedding space and using aligned semantic features as guidance conditions, our model achieves improvements in SA. Furthermore, by optimizing the noise gradient and guiding generation towards specified emotion styles, our method significantly outperforms existing methods in EA and EC metrics.

To evaluate our emotion classifier’s guidance capability, we randomly select two to four emotions from the eight available in the BEAT [8] dataset and assign them to each audio segment in the validation set. We feed the generated results into a pre-trained emotion classifier to determine the emotion category and compare the output with the ground truth emotion labels. As shown in Table II, our baseline method significantly outperforms DiffStyleGesture [6] in emotion-alignment and emotion-controllability metrics. We also provide an emotion evaluation metric for fingers, observing noticeable changes in some emotions. For instance, under the angry emotion, the virtual character often displays pointing gestures while speaking. To substantiate our emotion classifier’s effectiveness, we incorporate it into the DiffStyleGesture method’s inference stage for evaluation. Using the same data representation ensures a fair comparison with consistent evaluation logic and classifier model across both methods. Table II shows that, with the emotion classifier, DiffStyleGesture achieves higher emotion alignment compared to the original approach (relying on one-hot encoding for emotion representation).

User Study. To evaluate ExpGest’s real-world visual performance, we conducted a user study comparing generated gesture sequences under various guidance modes, DiffStyleGesture, and ground truth data. We curated 40 audio clips from the BEAT test set, featuring diverse voices, and divided them into short (8-16 seconds) and long (60-90 seconds) segments. About 120 participants evaluated the gesture sequences rendered into dynamic virtual speakers using Blender. We employed four evaluation metrics: “Human-likeness”, “Gesture-appropriateness”, “Emotion-compatibility”, and “Global-coherence”, each scored on a 1-5 scale. Table III presents the average scores, showing ExpGest outperforms previous methods in visual performance and gains higher acceptance from participants. The mixed guidance approach demonstrates remarkable synergy in generated gestures.
Results. Figure 4 (top) demonstrates a comparison of ExpGest with prior state-of-the-art methods under single audio guidance. We uniformly selected serene recordings from Wayne and Solomon, two speakers in the BEAT dataset, as test audio. The generation results of DiffStyleGesture, which relies solely on melody features, are relatively static with little variation. In contrast, ExpGest, benefiting from semantic alignment in the latent space, reveals highly expressive gesture results. Furthermore, our proposed noise emotion guide can naturally and smoothly perform emotional transitions, thereby enhancing the expressiveness of the generated results for reference). Figure 4 (bottom) showcases the superior performance of ExpGest under mixed control. By integrating audio with action, ExpGest can generate a variety of motion sequences, such as “dragging left leg, running forward”. Simultaneously, using existing large language models to edit the action text description for a speech (“he looks for a place to sit while talking”), combined with the audio itself, a vivid speaking character can be generated.

we designed the following ablation experiments. The results are detailed in Table IV. Our baseline method has already achieved significant improvements compared to DiffStyleGesture. Subsequently, we incorporated an Emotion-Guided Classifier (EG) during the reverse diffusion process, optimizing noise using gradient backpropagation. Although the performance in FGD has decreased, it still outperforms previous methods and significantly improves emotion alignment. Then, we introduced the Semantic Alignment (SA) module. Compared to the baseline, FGD has increased significantly, indicating the importance of semantic alignment in gesture generation, and since there are no additional optimization operations in reverse diffusion, the generation speed is also close to DiffStyleGesture, taking approximately 20 seconds to generate a 180-frame gesture slice. In this experiment, we conducted ablation studies only on the proposed baseline, without involving mixed guidance. All evaluation results were generated using pure audio guidance.