A Comprehensive Review of
 Data-Driven Co-Speech Gesture Generation

Hasegawa et al. [HKS^∗18] proposed an autoregressive approach to generate gesture from audio utterances using a bi-directional LSTM [HS97]. The bi-directional LSTM learned audio-gesture relationships with both backward and forward consistencies over a long period of time. The model was trained with a then novel audio-gesture dataset, collected using a headset and marker-based motion capture [TKS^∗17]. The model predicted a full skeletal human pose from the utterance features input at every LSTM timestep. Temporal filtering was then used to smooth out discontinuities in the generated pose sequences.

Kucherenko et al. [KHH^∗19] extended the work of Hasegawa et al. [HKS^∗18], removing the need for temporal smoothing through representation learning of an autoencoder. The proposed model transformed audio input into a gesture sequence in the form of 3D joint coordinates. They achieved this by (i) learning a lower dimensional representation of human motion using a denoising autoencoder consisting of a motion encoder (called MotionE) and a motion decoder (called MotionD) and (ii) training a novel speech encoder (called SpeechE) to transform speech to the corresponding motion representation with reduced dimensionality. During inference, the SpeechE predicted the motion representations, based on a given speech signal, and the MotionD decoded the motion representations into gesture sequences. However, their approach was deterministic and thus unable to capture the commonly observed phenomena where a person gesticulates differently at different points of the same utterance.

Deterministic generative approaches usually learn their parameters using a regression objective, e.g. L1 (Mean Absolute Error) or L2 (Mean Squared Error). Optimizing with either of those objectives typically forces the model toward learning to generate the mean representation of the data, producing averaged motion for different inputs, and resulting in undesirable results; usually called regression to the mean. Several approaches avoided this by incorporating probabilistic components into their objectives. Probabilistic components can increase the range of gesture motion in multiple ways, namely: (i) greater range of motion for different inputs, or (ii) stochastic motion for the same input. The most prominent are implicit log-likelihood evaluation via adversarial learning with Generative Adversarial Networks (GAN) [GPAM^∗14], explicit log-likelihood evaluation via variational inference with Variational Autoencoders (VAE)[KW13], and exact log-likelihood evaluation via invertible transformations with Normalizing Flows [KPB20, PNR^∗21].

GANs aim to do implicit density estimation of the underlying distribution through the interplay of a generator that tries to produce samples that are representative of the data, and a discriminator that strengthens the generator by classifying samples as real (from the distribution) or fake (not from the distribution). Multiple gesture generation approaches added an adversarial objective as a term in a composite loss function, which increased the range of gesture motion although still deterministic for a given audio input [SB18, GBK^∗19, FNM20, YCL^∗20, ALNM20, RGP21, WLII21b, WLII21a, ZRMOL22, HES^∗22]. We discuss some notable examples below.

Ferstl et al. [FNM19, FNM20] added multiple adversarial objectives to a recurrent neural network, known to be susceptible to regression to the mean for long sequences. The adversaries accounted for gesture phase structure, motion realism, displacement and diversity of the minibatch. Ahuja et al. [ALNM20] learned to implicitly estimate a mixture of densities, each representing a speaker to enable the transfer of one speaker’s style onto the speech input of another. The mixture of densities was estimated by instantiating a generator (per speaker) that is responsible for generating gestures that are representative of that speaker’s underlying gesture distribution. All the generators were trained in unison using the adversarial objective. Most recently, Habibie et al. [HES^∗22] used an adversarial objective in the form of a conditional GAN to refine gesture clips that were selected using a k-Nearest Neighbour (kNN) search that is conditioned on speech and a control signal.

Normalizing flows learn to describe highly complex distributions by applying invertible sub-transformations to a simple initial distribution, where invertibility allows one to optimize the exact likelihood of the deep generative model via gradient descent [KPB20, PNR^∗21], unlike in GANs or VAEs. In the context of gesture generation, Alexanderson et al. [AHKB20] extended a normalizing flow-based model for locomotion [HAB20] to apply to speech audio-driven gesture synthesis with style control. Their normalizing flows learned invertible transformations from simple Gaussian distributions to the distribution of upper- or full-body motion capture data. These transformations were conditioned on speech acoustics and, optionally, arbitrary style parameters. Specific style parameters considered included properties such as the average hand height, gesture speed, and gesture radius in a 4-second interval. The resulting model produced gestures that scored among the best in terms of naturalness and appropriateness in the GENEA Challenge 2020 [KJY^∗20].

VAEs aim to do explicit density estimation of the underlying data by optimizing a combination of a reconstruction loss for an autoencoder (usually an L1 or L2 loss) and the Kullback-Leibler (KL) divergence for distribution matching between a prior distribution and approximate posterior distribution of the data. The prior distribution is usually instantiated as a Gaussian for simple parameterization. The learned stochastic variables can then be sampled and decoded into diverse outputs. Recently, Li et al. [LKP^∗21] used a conditional VAE to translate speech into diverse gestures. They explicitly modeled a one-to-many speech-to-gesture mapping by splitting the cross-modal latent code into a shared code (audio + gesture) and motion-specific code (gesture only). The shared code modeled the correlation between speech and gesture e.g. synchronized speech and rhythmic gestures. The motion-specific code attempted to capture the diversity in gesticulation, independent of audio information.

In a similar vein, Ghorbani et al. [GFC22, GFH^∗23], used a VAE-based framework for style controllable co-speech gesture generation conditioned by a zero-shot motion example i.e., an instance of a motion style unseen during training. Given an audio input and a motion example, they generated an encoding of the audio and a style embedding from the motion, and the two latent codes were used to guide the generation of stylized gestures. The variational nature of the style embedding enabled them to easily modify style through latent space manipulation or blending and scaling of style embeddings. Moreover, the probabilistic nature of the model enabled the generation of varied gestures for any audio and exemplar motion input. The resulting model performed favorably against state-of-the-art probabilistic techniques [AHKB20] in terms of naturalness of motion, appropriateness for speech, and style portrayal.

Taylor et al. [TWGM21] adapted the conditional Flow-VAE framework [BHF^∗19], combining the advantages of the VAE and normalizing flow architectures,to generate spontaneous gesture movement for speaker and listener roles in a dyadic interaction. They used the Flow-VAE framework for modeling expressive gesture because of its ability to improve generative capacity of the VAE by estimating the latent space with a highly complex distribution using a normalizing flow, instead of the standard Gaussian. Their autoregressive framework was trained on a set of previously generated gestures, an audio input window and the dyadic role, i.e. speaker or listener, as input. The preceding gestures were encoded into a latent variable then transformed into a complex distribution using a normalizing flow, conditioned by the audio window and role in the dyad. Their decoder then generated the next gesture based on the latent variable sampled from the complex distribution. The resulting model could generate expressive co-verbal gestures in a dyadic setting based.

Hybrid systems that combine deep learning and database matching components can also help tackle the regression to the mean problem [KNN^∗22, ZBC22, FNM21, HES^∗22]. Indeed, this approach has been used effectively in motion synthesis problems, e.g. game animation where high fidelity motion is crucial [HKPP20]. In the context of conversational gesture, the intuition is that modeling the association between high dimension audio input and gestures, represented by exact joint positions or angles, using standard regression objectives (L1 or L2 loss) discourages the model from producing otherwise plausible gestures that do not exactly match the ground truth, thus greatly reducing the variety of generated gestures. Alternatively, the audio-gesture association can be modeled by predicting higher-level parameters for gesture motion. Ferstl et al. [FNM21] realized this idea by learning to map audio to gesture via higher-level expressive parameters, specifically gesture velocity, acceleration, size, arm swivel angle, and extent of hand opening. First they pre-trained a model to associate audio prosodic features to the expressive parameters. Then they predicted gesture timing by extracting pitch peaks in the audio signal. At inference time, the prosodic features were used to estimate the expressive parameters that were in turn used to search for a matching gesture in the database, and the pitch peaks were used to temporally position the matching gesture. Finally, synthetic preparation and retraction phases were added to connect the gestures in the sequence.

Another interesting approach for preserving gesture form is through audio-based search in a video gesture database where the gestures are representated by video frames. Zhou et al. [ZYL^∗22] explored this idea in a gesture reenactment task by generating a gesture video for an unseen audio input, using gesture frames from a reference video. They first encoded the reference gesture video as a “video motion graph” - a directed graph where each node represented a video frame and corresponding audio features, and the edges represented transitions. The graph encoded how the reference video can be split and re-assembled in difference graph paths. In order to increase graph connectivity, i.e, diversity of plausible path, they added synthetic edges based on a frame pose similarity threshold computed using the SMPL pose parameters [LMR^∗15]. Given unseen audio input as a guide, they traversed the graph using a beam search algorithm [RR77] to find the most optimal path or order of gesture frames that best matches the speech audio. For graph paths that contain temporally disjoint frames, they trained pose-aware video blending network to synthesize smooth transitions between the frames.

Approaches that used audio as the primary modality produced well-timed hand movements that tend to be highly associated with acoustics, largely corresponding to beat gestures. However, the lack of text transcript means they were not informed by the structure and context inherent in the text, for example, semantic meaning and punctuation. Such structure can help produce more meaningful and communicative gestures. Therefore, next, we describe some approaches that used text as the primary input modality.

Ishi et al. [IMMI18] proposed a text-based gesture generation approach for controlling a humanoid robot. They modeled the text-to-gesture motion translation by associating words to concepts, concepts to gesture categories (i.e. iconic, metaphoric, deictic, beat, emblem and adapter), and gesture categories to gesture motions. Further, they estimated conditional probabilities to model the association between word concepts and gesture categories, and between gesture categories and gesture motion clusters that were pre-computed with the k-means clustering algorithm.

Yoon et al. [YKJ^∗19] proposed an encoder-decoder approach that transformed speech text, from a dataset based on TED talks, into a sequence of gestures. They created the TED video dataset by picking video segments with the speaker’s upper body and hands. Then they performed pose estimation using OpenPose [CHS^∗19] and removed segments containing noisy or no estimations. Speech text was converted into a sequence of 300-dimensional word vectors using pre-trained GloVe embeddings [PSM14]. Similarly, poses estimations were converted to 10-dimensional vectors using principal component analysis (PCA). Therefore, co-speech gesture generation became a sequence-to-sequence translation problem from word embeddings to human poses encodings. The encoder part of the network was a bi-directional GRU [CVMG^∗14] taking in speech text one word (vector) at a time and capturing bi-directional context. The last hidden state of the encoder was passed into the decoder, also a bi-directional GRU. The decoder also took previous pose estimations to condition the prediction of the next pose, in addition to using soft-attention [BCB15] to focus on specific words when predicting the next pose. Finally, the generated 2D poses were mapped to 3D and executed on the NAO humanoid robot.

Recently, Bhattacharya et al. [BRB^∗21] used text transcripts to produce expressive emotive gestures for virtual agents in narration and conversation settings, using MPI-EBEDB, a dataset of actors performing multiple emotion categories (amusement, anger, disgust, fear, joy, neutral, pride, relief, sadness, shame, surprise) [VDLRBM14]. Their approach consisted of Transformer-based encoders and decoders [VSP^∗17], where the encoder took in the text transcript sentences (encoded as GloVe embeddings [PSM14]) to produce an encoding which was concatenated with the agent attributes such as narration/conversation, intended emotion, gender and handedness. The previous pose’s encoded concatenation and 3D joint positions were passed as input to the Transformer decoder to generate the next pose’s joint positions. The process was repeated in a recurrent manner until the full pose sequence was generated.

An interesting trade-off exists between audio-based and text-based gesture generation systems. audio-based generators have access to intonation and prosody which helps generate rhythmic or kinematic gestures (e.g. beats) but lack semantic context. Conversely, text-based generators have access to semantic context which helps generate meaning-carrying gestures (e.g. iconic or metaphoric), but lack intonation and prosodic information. Therefore, combining the audio and text modalities enables a gesture generator to learn to produce semantically relevant and rhythmic co-speech gestures.

Although generating meaning-carrying gestures using audio only is theoretically possible, it is unlikely since prosody is suitable for kinematics, but not sufficient to infer shape which is associated with meaning [LWH^∗12]. As far as we know, meaningful gestures from speech audio alone have not been empirically demonstrated. Instead, combining audio with text appears to be the most promising approach to generating meaningful gestures to date. We, therefore, focus on approaches that combine these two modalities for generating meaning-carrying, communicative gestures.

Chiu et al. [CMM15] proposed an approach that combined the text and prosody of the speech to generate co-verbal gestures. Their model, called the Deep Conditional Neural Field (DCNF), was a combination of a fully-connected network, for representation learning, and a Conditional Random Field (CRF), for temporal modeling. For the gesture prediction task, the model took in a text transcript, part-of-speech tags and prosody features as input, and predicted a sequence of gesture signs which were a set of predefined hand motions.

Leveraging the representation power of deep learning models for multimodal input (i.e. audio and text) for co-speech gesture generation was the next logical step. In fact, three groups of researchers independently proposed the first deep-learning-based gesture generators that used both audio and text to generate continuous gestures, namely Yoon et al. [YCL^∗20], Ahuja et al. [ALIM20] and Kucherenko et al. [KJvW^∗20]. We discuss their pioneering work combining audio and text next, followed by subsequent efforts in the area.

Yoon et al. [YCL^∗20] proposed a gesture generation approach that combines the tri-modal context of speech, text and speaker identity to produce gestures that were human-like, and matched the content and rhythm of the speech. The model processed input speech and text with a speech and text encoder, respectively. The speaker identity was used to sample the intended speaker from a learned style embedding space. Together the three features (i.e. speech encoding, text encoding, and style) were passed to a gesture generator to produce the sequence of poses. Closely related was Liang et al. [LFZ^∗22] whose framework utilized audio and text information in order to generate meaningful gestures by disentangling semantic and beat gestures. Their system consisted of two encoders, one that took in audio and text to encode semantics, and another that took in audio volume and beat to encode non-semantic information. The encoded information from both encoders ensured the disentanglement of semantic and beat gestures, while decoder took this information and was trained to encourage generation of meaningful semantic gestures.

Ahuja et al. [ALIM20] identified two key challenges in an attempt to learn the latent relationship between speech and co-speech gestures. First, the underlying distributions for text and gesture are inherently skewed and therefore necessitated the need to learn their respective long tails, accounting for rarely occurring text or gestures. Second, gesture predictions are made at the sub-word level, which necessitated the need to learn the relationship between language and acoustic cues that may give rise to, or be accompanied by, a particular gesticulation. So motivated, they proposed the Adversarial Importance Sampled Learning (AISLe) framework, that combined adversarial learning with importance sampling to balance precision and coverage. The model took in speech and text transcripts and performed encoding and alignment between sub-words and acoustics, using a multi-scale Transformer [VSP^∗17]. The resulting alignment was passed to the model’s generator to predict the pose sequence and an adversarial discriminator was used to determine if the pose was real or fake. For optimizing the adversarial objective, the AISLe framework scaled the loss function such that rarely occurring gesture samples, the long tail of the distribution, were weighted more than those that are more likely to occur.

Kucherenko et al. [KJvW^∗20] proposed an autogressive generative model that combined speech acoustics and semantics to produce arbitrary acoustically-linked or semantically-linked gestures. The key insight of their approach was to envision a gesticulation system that encompasses so called “representational” gesture types (i.e. iconic, metaphoric and deictic) that convey semantics, and beats that are synchronized with acoustics. Their approach took a concatenation of semantic features that were extracted using BERT [DCLT18] and acoustic features represented as log-power mel-spectograms as input into an encoder. Then they integrated past and future context for each gesture pose frame via a sliding window operation over the encoded speech features. The model generated each pose autoregressively where each was conditioned on the information of three preceding frames to ensure motion continuity. Their extensive evaluation indicated that autoregression for continuous motion and combining audio and text had the most significant positive impact on the quality of the generated gesticulations.

Equally inspired by autoregressive generative models, Korzun et al. [KDZ21, KDZ20] reimplemented the text-only recurrent framework by [YKJ^∗19] to accommodate both text and audio input. The proposed model was a combination of a recurrent context encoder, inspired by [KHH^∗19], that generated hidden states for 3-second audio and text context windows and a recurrent encoder-decoder that took in the concatenated results of the context encoder and used an attention mechanism to condition the generation of the final gesture motion. Similar to Yoon et al. [YKJ^∗19], they trained the model using the continuity and variance objectives to ensure fluid and natural-looking gestures. The resulting model produced gestures that were deemed natural and appropriate as part of the GENEA Challenge 2020 [KJY^∗20].

Designing generation systems that produce meaningful gestures is one of the major goals in non-verbal behavior research. Spurred on by this question, Kucherenko et al. [KNN^∗22] investigated whether contemporary deep learning-based systems could predict gesture properties, namely phase, type and semantic features, as a way to determine if such systems can consistently generate gestures that convey meaning. Their model used both audio and text for predicting gesture properties, through two distinct components that predicted the probability for gesticulation and probabilities for the aforementioned set of gesture properties. They conducted their experiments on a direction-giving dataset with a high number of representational gestures [LBH^∗13]. Their experiments showed that gesture properties related to meaning such as semantic properties and gesture type could be predicted from text features (encoded as FastText embeddings [BGJM17]), but not from prosodic audio features. Conversely, they found that rhythm-related gesture properties (e.g. phase) could be better predicted from audio features.

In order to mimic the communicative intent of co-speech gestures, it is crucial to understand and model the complex relationship between speech acoustics, text, and hand movements. An interesting approach is to group gestures with distinct movement properties in order to find emergent rhetorical categories. Saund et al. [SBM21] investigated this approach by modeling the rhetorical, semantic, affective and acoustic relationships between gestures and co-occurring speech audio and text, for a hypothetical gesture generation system. They first used k-means clustering to cluster speech-gesture pairs into functional domain clusterings (i.e. rhetorical, affective and semantic) based on functional tags generated from third-party natural language parsers. The speech-gesture pairs were refined into sub-clusters based on gesture motion. Therefore each speech-gesture pair belonged to at least one sub-cluster (based on motion), within one functional cluster (based on its assigned functional tags). At run-time, a hypothesized virtual agent would leverage the same pre-trained parsers and clusters to analyze an input speech and text transcription, select a functional cluster and from that a motion sub-cluster. The agent could then either choose an appropriate gesture from a pre-recorded library or the centroid gesture in the motion sub-cluster.

Motion graphs, commonplace in conventional animation systems (e.g. [KGP02, AF02, LCR^∗02]), can be effective at producing realistic non-verbal behavior because they rely on databases of high-quality motion capture or RGB video. As we discussed before, they were effectively employed for audio-driven gesture reenactment using video-based motion graphs [ZYL^∗22]. Zhou et al. continued this trend for audio and text by adapting a motion-graph-based music-to-dance system [CTL^∗21] for co-speech gesture generation  [ZBC22]. They first built a database of audio, text and gesture clips from 3-tuples of (audio, text transcript, gesture), using a splitting algorithm. For each audio clip, they generated a style signature using StyleGestures [AHKB20], and a rhythm signature using a binary encoding scheme that denotes the presence of words by leveraging the word-level timing information in the text transcript. For the corresponding gesture motion, they generated a style signature, parameterized by the same attributes as StyleGestures [AHKB20] (e.g. wrist speed, radius and height), and a rhythm signature using a similar binary scheme that denoted the presence of pausing, sharp turning or a stroke phase of the gesture. During synthesis, they computed the rhythm and style signatures for input audio and text and used a graph-optimization algorithm to find gesture clips that closely matched the generated style and rhythm in terms of Hamming distance, and minimized the motion transition in the graph. This model performed on par or better than motion capture data in terms of Naturalness in the GENEA Challenge 2022 [YWK^∗22].

Style transfer is a widely adopted optimization technique in deep learning for blending visual content and style, e.g. given a content image and a reference image that specifies the style, adjust the content image to match the style [GEB16]. In the context of co-speech gesture generation, it might be desirable to transfer the speaking style of one speaker to the predicted gestures of another. Ahuja et al. [ALNM20] learned unique style embeddings for multiple speakers that enabled either generation of gestures consistent with the original speaker in the audio input, or style transfer by combining the audio input of one speaker with the style embedding of a different speaker. Although they proposed the PATS data where multiple modalities such as audio, gesture pose and text have style and content, they focused on gesture pose style to learn unimodal speaker-specific style embeddings. Fares et al. [FGPO22] leveraged the multiple modalities in the PATS dataset to learn multimodal style embeddings based on audio, text and gesture pose input. Their framework consisted of a speaker-style encoder that used speaker audio, text and gesture pose to learn a multimodal style embedding, and a sequence-to-sequence decoder that generated gestures based on audio and text, and conditioned on the desired speaker’s style embedding. Furthermore, unlike the work of Ahuja et al. [ALNM20] that required the entire speaker’s gesture data to learn the speaker’s style embedding, their trained speaker-style encoder could generate style embeddings in a zero-shot manner i.e., for speaker styles not seen in the training set.

A key tenet of semantically meaningful gestures is that they are appropriate for the given utterance. To achieve this, there needs to be a greater emphasis on generating precise gestures using audio and text as grounding (i.e. the appropriateness of the gesture to the utterance), versus generating diverse gestures. Lee et al. [LAM21] investigated this approach and made an interesting observation about human gesticulation, that multiple semantically different utterances are often accompanied by the same gesture. They thus proposed a contrastive-learning framework that constrained the mapping of semantically different utterances to a smaller subset of relevant high-quality gestures. They introduced a novel contrastive learning objective that preserved similarities and dissimilarities of gestures in the latent representation. The objective ensured that latent language representations of two semantically different utterances were close together if they were accompanied by the same gesture. They first clustered gestures based on similarity or dissimilarity, then created positive (similar gesture poses) and negative (dissimilar gesture poses) required for the standard contrastive learning objective. Finally, they learned gesture-aware embeddings via a contrastive and adversarial objective. The resulting embedding space was used to generate gestures that were semantically relevant and closer to the ground truth.

When designing 3D avatars, it may be desirable to have a holistic animation system that includes facial and full-body movement. Combining audio and text modalities can be effective at achieving this goal because of their rhythmic and semantic properties. Zhuang et al. [ZQZ^∗22] investigated this approach by proposing a hybrid system consisting of Transformer-based encoder and decoder modules, and motion-graph retrieval module to generate facial motion and full-body motion that included gestures. Their encoder used both audio and text, in the form of phoneme labels and Mel Frequency Cepstral Coefficients (MFCC) and Mel Filter Bank (MFB) features, to generate 3D facial parameters for synchronous lip movement. Simultaneously, the decoder used speech features, previous expression motion and semantic tags to generate 3D facial parameter for expression. The motion-graph retrieval sub-system used speech audio and text to find the most appropriate body motion segments, including gesture, that correspond to the text semantics and rhythm in the audio. Finally the facial and body motion were used to drive a skinned polygonal model.

Several deep learning-based systems complemented input audio or text with additional information that could reasonably be deemed relevant to co-speech gestures. This included speech context, speaker style, discourse or an interlocutor’s movements. Sadoughi and Busso [SB19] proposed a system that bridges rule-based and learning-based techniques in order to select gestures that are communicative and well synchronized with speech. They proposed a Dynamic Bayesian Network (DBN) which took in speech and two constraints to condition the generation. The constraints were: 1) discourse function, which restricts the model to behaviors that are characteristic of that discourse class (e.g. questions); 2) prototypical behaviors, which restricted the model to certain target gesticulations (e.g. head nods). Given constraints on prototypical behaviors, the approach could be embedded in a rule-based system as a behavior realizer creating head and hand trajectories that are temporally synchronized with speech.

In a dyadic conversation between interlocutors, there can be a lot of spontaneous non-verbal behavior that is influenced by the nature and tone of the interaction. Leveraging the co-adaptation of non-verbal behavior between interlocutors present in human-to-human interactions, cf. [BK12, CBK14, OB16], can enable virtual agents to be naturally conversational and collaborative. Ahuja et al. [AMMS19] proposed the Dyadic Residual-Attention Model (DRAM), a framework that could interactively generate an avatar’s gesticulation conditioned on its speech and also the speech and gesticulation of a human interlocutor in a telepresence setting. In order to generate natural behavior, the avatar had to consider its own speech as well as the speech and gesticulation of the human. The DRAM model generated natural dyadic behavior by taking in the speech and pose history of the avatar as well as the speech and pose history of the human to adapt the avatar’s gesticulation accordingly.

The idea of conditioning the motion of a deep-learning-based agent on interlocutor speech and motion has subsequently been used in several other works. Jonell et al. [JKHB20] used a model based on normalizing flows for generating head motion and facial expression, while Nguyen and Celiktutan [NC22] used conditional adversarial learning to drive full-body skeletons. Both of these works found that statistically significant improvements in generated behaviors were achieved by being interlocutor-aware.

A similarly interesting dyadic scenario is human-robot interaction where one of the interlocutors is a social robot. In this case, the robot must exhibit natural non-verbal behavior in order to be engaging and interesting. Therefore, it is desirable for the robot to mimic human non-verbal motion with gestures that are natural and communicative. Deichler et al. [DWAB22] investigated this idea by proposing a combination of a data-driven and physically-based reinforcement learning (RL) framework to generate pointing gestures learned from motion capture data. Given a diverse motion capture dataset of pointing gestures and corresponding targets, they trained RL control policies adapted from [PALvdP18, PMA^∗21] to imitate human-like pointing motion while maximizing the reward based on pointing precision.

Automatic synthesis and animation of gestures that accompany affective verbal communication can endow virtual agents with emotional impetus. Bozkurt et al. [BYE20] directly mapped emotional cues in speech prosody into affect-expressive gestures. They investigated the use of three continuous affect attributes (i.e. activation, valence and dominance) for the speech-driven synthesis of affective gesticulation. They proposed a statistical model based on hidden semi-Markov models (HSMM) where states were gestures, and observations were speech prosody and continuous affect attributes. They first estimated the affective state from speech prosody and then used the state and speech prosody to predict gesture clusters. The gesture segments were animated using a unit selection algorithm [BEY15], and discontinuities were smoothed using an exponential smoothing function. Finally, the smoothed sequence was animated in Autodesk MotionBuilder.

Text encodes important semantic information, potentially useful for conveying meaningful emotion through gesture, although it encodes fewer cues about emotional state compared to audio e.g., intonation and speech pauses. An interesting approach is to combine text with an intended emotion for affective gesture generation. Bhattacharya et al. [BRB^∗21] pursued this approach by combining text transcripts associated with narrative or conversational acting and emotion labels, to produce expressive emotive gestures for virtual agents. The emotions represented were amusement, anger, disgust, fear, joy, neutral, pride, relief, sadness, shame and surprise[VDLRBM14]. Their approach consisted of a Transformer [VSP^∗17] encoder and decoder, where the encoder took in the text transcript sentences, intended emotional state and agent attributes (e.g. narration/conversation, intended emotion, gender, handedness). The previous pose’s encoded concatenation and 3D joint positions were passed as input to the Transformer decoder to generate the next pose’s joint positions. The process was repeated in a recurrent manner until the full affective pose sequence was generated.

A speaker’s identity or style can affect how they gesticulate, as some speakers gesture a lot while others rarely do. Moreover, they may also prefer particular gesture forms, and use different hands or gesture sizes. Modeling such variation in non-verbal behaviour can help make virtual agents seem unique and have a personality. To this end, Yoon et al [YCL^∗20] used speaker identity to guide gesture generation that matched the speaker’s style. Their adversarial approach combined the tri-modal context of audio, text and speaker identity to produce gestures that were human-like, and matched the content and rhythm of the speech. The model processed input audio and text with an audio and text encoder, respectively. The speaker identity was used to sample the intended speaker from a learned style embedding space. Together the three features (i.e. audio encoding, text encoding, and style) were passed to a gesture generator to produce the sequence of poses. Similarly, Ahuja et al. [ALNM20] learned a mixture of adversarial generators, representing diverse gesticulation styles of speakers from talk-show hosts, lecturers and televangelists. Learning speaker-specific generators enabled one speaker’s style to be aligned with, or transferred to, the audio of another speaker.

Developing robust deep-learning-based gesture generators requires large amounts of diverse gesture data from real world scenarios, captured either via motion capture or pose estimation from videos. However, capturing or estimating hand gestures is very challenging because of the intricate finger motion, relatively small size of hands with respect to the whole body and frequent self-occlusions [HLW^∗18, LDM^∗19, JYN^∗20]. In contrast, capturing body motion (up to and including the arms) is less error prone because the joints are further apart and the articulations are relatively simpler. Therefore, the “upper body” motion as a modality can be an informative prior for generating conversational hand gestures. Ng et al. [NGDJ21] investigated this idea while making the observation that body motion is highly correlated with hand gestures. Their proposed approach took in 3D upper-body motion (up to the wrist) and predicted 3D hand poses. In addition to upper-body motion, the model could take in 2D images of hands and produce the corresponding 3D hand pose estimations. Similar to [GBK^∗19], they used a combination of a L1 regression loss for the model training signal and an adversarial loss to ensure realistic motion. The learned body-motion-to-hands correlation was versatile enough for several use-cases, namely conversational hand gesture synthesis, single-view 3D hand-pose estimation and synthesizing missing hands in motion capture data and image-based pose estimation data.

Although control can take either linguistic or non-linguistic forms, it is distinct because it can convey the explicit design and execution intent of an animator. Multiple works in motion synthesis use control as an additional input either during the training phase or the inference phase of learning-based models (e.g. [HKS17, LZCvdP20]). Typically, during training, the control signal is used to train the system to generate animations with certain biomechanical constraints such as posture, gait, etc. During inference, control may be introduced to impose style-related constraints [SCNW19] or user input [HKS17, HAB20, LZCvdP20].

In the context of conversational gesture, Alexanderson et al. [AHKB20] trained a probabilistic model that generated spontaneous co-verbal gesture that was conditioned on control constraints such as wrist height, radial extent and handedness. However, the constraints were introduced at training time, meaning modeling a new constraint required re-training the entire model. Habibie et al. [HES^∗22] provided a more flexible approach. They first learn a speech-to-gesture motion search through a kNN algorithm, and then refine the motion using conditional GAN. Style control can be exerted at runtime by dynamically restricting the portion of the database that the kNN algorithm is run on, allowing style variation even within an extended utterance without the need to retrain.

Control can also be imposed by implicitly specifying the desired gestures by learning emergent prototypes of gesture shape or form. Qian et al. [QTZ^∗21] explored this idea by learning conditional vectors, so-called “template vectors”, that could determine the general appearance and thus narrow the potential range of plausible gestures. Their framework took in audio and a zero initialized condition vector, through a 1D UNet-based autoencoder, in order to generate the corresponding gestures as 2D joint positions. During training, they periodically updated the condition vector, through back-propagation, using the gradients computed on the L1 regression loss between the generated and ground-truth gestures. They regularized the template vector space through the KL-divergence between the vectors and a normal distribution. They also separately pre-trained a VAE to reconstruct ground truth gestures and used the resulting latent space to encode gestures into template vectors. At test time, they sampled arbitrary template vectors, either learned through back-propagation or extracted by the pre-trained VAE, to generate diverse gestures.

Animators typically want to specify high-level style parameters to convey design intent e.g. energetic oratory gesticulations or subdued gestures to convey sadness. Additionally, it is desirable to specify the style once in the workflow and for the animation system to generate arbitrarily many motions for that specification. However, there is a gap between desired abstract design intent and existing deep-learning-based style control systems that tend to rely on biomechanical constraints such as wrist speed, radius or height [AHKB20, HES^∗22]. Style specification is also not data efficient, requiring as many samples as the size of the training set for the model to learn a style [AHKB20, ALNM20]. We conclude this section by discussing several works that proposed approaches for data-efficient style specification [GFC22, GFH^∗23, FGPO22, ALM22].

Ghorbani et al. [GFC22, GFH^∗23] proposed a framework that improves on high-level style portrayal by using exemplar motion sequences that demonstrate the intended stylistic expression of gesture motion. Their framework was able to efficiently extract style parameters in a zero-shot manner, only requiring a single example motion and was able to generalize to example motions (and therefore styles) unseen during training. Fares et al. [FGPO22] used an adversarial framework to learn a speaker-style encoder that could generate speaker-specific style embeddings from novel multimodal inputs – audio, text and gesture pose – not seen during the training phase. The framework generated co-speech gestures in a style that is either consistent with the original speaker in the audio or a different speaker, depending on the chosen style embedding. Ahuja et. al [ALM22] proposed an adversarial domain-adaptation approach for personalizing the gestures of a source speaker with plenty of data, with the style of a target speaker with limited data, using only 2 minutes of target training data. Given a model pretrained on a large co-speech gesture dataset, their framework could adapt the model’s parameters using a smaller target dataset by modeling the cross-modal grounding shift, i.e., the change in distribution of speech-gesture associations, and the distribution shift in the target gesture space. The approach’s ability to identify distributions shifts between the source and target domain for parameter updates, enabled the model to extrapolate to gestures in the target distribution without having seen them in the source distribution during pretraining.

One important aspect to evaluate for gesture-generation systems is the human-likeness of the generated gestures, which is measured and compared through human perceptual studies, often with comparable stimuli presented side by side as in e.g. [JYW^∗21, KJY^∗21, WGKB21]. On the other hand, evaluating the other aspects such as the appropriateness and/or specificity of generated gestures in the context of speech and other multimodal grounding information (see Section 6.4) is quite challenging, especially since differences in the human-likeness of the motions being compared tends to interfere with perceived gesture appropriateness (cf. the results in [KJY^∗21]). To alleviate this challenge for appropriateness, a new evaluation paradigm of matched vs. mismatched gesture motion has recently been proposed [JKHB20, RGP21, YWK^∗22]. In this setup, human participants are asked to choose between two motion clips that both were generated by the same system, and therefore have similar appearance and human-likeness, but where one clip is intended to be appropriate to the situation (e.g., the motion in it corresponds to the actual speech audio in the video) whereas the other is chosen at random (e.g., it was generated by feeding unrelated speech audio into the same system instead, and does not match the actual audio track). The extent to which humans are able to identify the video that matches the situation can be used both to probe the strength of grounding in different modalities, and to assess gesture appropriateness for speech, rhythm, interlocutor behavior, etc., while controlling for human-likeness. We expect this methodology to gain wider adoption and advance the state of the art in subjective assessment of different aspects of co-speech gestures.

Another compelling area for future work is to evaluate gesture generation in actual interactions, since the ultimate goal of embodied conversational agents is to enhance human-computer communication and interaction. Initial studies [NKM^∗21, HPK22] have found that embodied agents that perform gestures generated by data driven models as opposed to performing no gestures, attract more attention from the audience. A larger attention span on a gesticulating agent is indicative of a more engaging communicative quality of gestures and opens doors to evaluating gesture generation in a more natural setting. Although the situated and time-demanding nature of such interactions, coupled with their reliance on many non-gesture components necessary to create interactivity (e.g. human wizards or automatic speech recognition, dialogue systems, and text-to-speech), make proper interactive evaluation challenging and seldom done, it is an important long-term goal for evaluations in the field. Given the difficulties in comparing different research papers in the field, we think that controlled, large-scale comparisons [KJY^∗21, YWK^∗22] with open data and materials are going to play an important role to develop the co-speech gesture field and its evaluation practices in the shorter term. This is similar to the role challenges have played in the development of text to speech [Kin14] and the wide use of leaderboards and benchmarks across deep learning today.

While subjective metrics from appropriately designed user-studies are the gold standard in co-speech gesture evaluation [WRB22], they are expensive and time consuming, and thus lack scalability. There is therefore interest in objective metrics to automatically assess synthetic motion, for example its its human-likeness. Objective metrics are useful to measure progress during model development in a heavy compute, data-driven learning setup. A natural metric is accuracy of prediction (i.e., how often the predicted position of a joint is within some tolerance of the joint position in a human motion capture clip), which is often called the Probability of Correct Keypoints (PCK). However, this quantity is often not indicative of performance due to the one-to-many nature of the gesture-generation problem. Two examples of human motion for the same speech might involve very different joint positions, and thus have low mutual agreement. Measuring the mean squared error (MSE) between generated motion and human motion capture suffers from the same issue.

Statistics of motion properties such as acceleration and jerk have been used as an alternative for quantifying and comparing generated gesture distributions [KHK^∗21], but there is no compelling evidence that these metrics correlate with subjective assessments of motion human-likeness. To improve the measurement of distributional similarity of gestures, new objective quality metrics based on innovations from image processing, namely the Fréchet Inception Distance (FID) [HRU^∗17] and the Inception Score [SGZ^∗16], were proposed in [ALIM20, YCL^∗20] and [ALNM20] respectively. Among these proposals, only [YCL^∗20] computes the Fréchet distance in a learned space. There has also been work in learning to estimate the human-likeness of gestures from databases of gesture motion and associated subjective ratings data [He22]. However, learning to predict human preference can be difficult even from relatively large training databases, as seen in similar research into predicting the subjective ratings of synthetic speech [HCT^∗22]. A recent preprint [KWY^∗23] used the data from the GENEA Challenge 2022 to compute correlations between subjective ratings and a number of objective metrics. They found that almost none of the objective metrics displayed a statistically significant correlation with the human-likeness scores from the large user study, with the Fréchet Gesture Distance being the one exception.

Since the above approaches depend on motion data only, they can only give an indication of whether or not generated motion is statistically similar to the human motion capture in the database, but not how appropriate the motion is for the context in which it occurs (whether it is grounded in that context). The methods can therefore not assess whether or not the motion is synchronized with the co-occurring speech, whether the motion is semantically relevant, etc. In general, unlike human-likeness, not many techniques have been proposed for objectively quantifying properties like gesture diversity or different kinds of motion appropriateness. One exception is the recent Semantic Relevance Gesture Recall (SRGR) metric from [LZI^∗22], which proposes to quantify the semantic relevance of gesture by using semantic scores, annotated in the speech text data, to weight the probability of correct keypoints between the predicted and ground-truth gestures higher when the ground-truth gesture has a high semantic score. This is a step in the right direction for evaluating semantic appropriateness, but may suffer from the same issues as regular PCK due to the idiosyncratic, one-to-many nature of gesticulation. Given the impact that the Inception Score and the Fréchet Inception Distance have had in driving progress in image generation, reliable metrics that estimate gesture human-likeness and especially appropriateness for e.g. the rhythm and semantics of co-occurring speech are an important continuing challenge, where recent and future innovations are likely to have significant impact on the field.

Gestures are temporal, which is a result of their correlation with a heavily temporal acoustic modality, along with the fact that they might depict occurrences or trace out paths or shapes over time. The rhythmic nature of the gestures (i.e. beat gestures) in context of acoustic prosody has been studied heavily since the era of rule based gesture synthesis  [CMM99, MXL^∗13]. Fast forward to approaches with data-driven synthesis, some explicitly rely on extracted prosodic features [FNM21], while others  [GBK^∗19, ALNM20] learn implicit embeddings from acoustics which prosody is one of the key components. It seems clear that gesture production must be grounded in the rhythm of audio data, and appropriate beat gestures will be challenging to achieve from text transcriptions alone, without timing information [KNN^∗22]. Alternatively, both audio and gesture must be synthesized to have comparable rhythmic structure.

Beyond the rhythmic nature of gestures, there is often a semantic meaning associated with the performed gesture. The small size of gesture-generation databases, and the complicated relationship and weak correlation between speech semantics and gesture form (see Section 6.4.6), mean that it is unrealistic to expect systems to learn to generate semantically appropriate gestures driven by speech acoustics alone. Text, on the other hand, is a compact way to represent much of the semantic content behind co-speech gestures, and has been heavily studied since the era of rule-based gesture synthesis [CVB01] as well as in data-driven synthesis [SB19, LAM21, ALIM20, KJvW^∗20, YCL^∗20, ZYL^∗22, LFZ^∗22]. Current data-driven approaches typically attempt to gain semantic awareness by relying on deep-learning based language models trained on large amounts of text, such as [MCCD13, DCLT18]. Recent large language models based on large amounts of text [BMR^∗20] have indeed been capable of generating text with surprisingly coherent semantics, suggesting that they can capture lexical meaning to a significant extent. While the inclusion of text has improved human perception of automatically generated gestures [KJvW^∗20, ALIM20, YCL^∗20, AGL^∗22], it is still not trivial to measure the semantic content of gestures (see the discussion in Section 6.1). Hence, it is unclear how much (if any) of the improved human perception can be attributed to the semantic awareness created due to the use of language models, nor how much of the bottlenecks that exist may be removed with continuing progress in neural language models. More broadly, there is a need for gesture synthesis models to perform better with regard to semantics. Gesture is most powerful when it conveys information, and doing this effectively has been a challenge for most deep learning systems; cf. Figure 3.

Co-speech gestures are idiosyncratic. The manifold of gestures performed by a speaker are not just a function of the content of the speech, but are also dependent on the identity, emotional state and the context of the speaker. Generating personalized gestures based on speaker identity became possible with the influx of large scale multi-speaker datasets [YCL^∗20, ALNM20]. Several GENEA Challenge 2022 [YWK^∗22] systems also make use of speaker identity. A deeper analysis of the impact of speaker identity input [KNN^∗22] shows that different speakers have different gesture-property prediction certainty, evoking even more interest in the idiosyncrasies of co-speech gestures. More recently, it was also shown that a short motion clip can be used for style control in “zero-shot style adaptation” [FGPO22, GFC22, GFH^∗23]. For many applications, it is desirable for the designer to be able to control the nature of the motion. This goes beyond replicating idiosyncratic motion recorded of an individual to being able to specify novel characters. We are far from having ways to author a character with an imagined personality for a particular application.

Apart from the speaker identity, the emotional or affective state of a speaker also impacts the gestures performed by them. A striking example of this is the large range of expressive motion variation with the same lexical message explored in the Mimebot data [AONB17]. Building emotionally aware embodied agents is a common research direction  [CBFV16, SZGK18]. More recently, data-driven models have been explored where affective cues were learned using a dedicated encoder in an adversarial setup [BCRM21] to imitate these patterns of affective behavior. It is important to be able to drive these emotions in a way that is consistent with a character’s personality and to be able to shift mood and emotion over time. One way forward might be to leverage findings from the literature of gesture and motion perception, which has identified many useful properties of gesture motion that correlate with the perception of personality [Lip98, KG10, SN17] and emotion [NLK^∗13, CN19]. By changing these properties in synthesized gestures, we may exert some control over the perceived speaker personality and emotion  [AHKB20, HES^∗22]. Again, speech synthesis provides an analogy, where it was recently shown that simple and easy additions of filler words and pausing can meaningfully and reliably be used to alter listeners’ perception of speaker certainty  [KLSG22].

While non-verbal behavior is impacted by internal state of the speaker, the external context also guides the types of gestures a speaker might perform. In a dyadic conversation, the model must be aware of the behavior of the interlocutor while generating the relevant gestures [AMMS19, JKHB20, NC22, YYH20]. This includes modeling appropriate listener behavior as well as speaker behavior. Characters must modify their behavior to react to the content, mood and timing of interlocutors. Characters must be able to be surprised, angered, pleased, etc. based on what their interlocutor may say. Given the increasing availability of dyadic datasets with motion capture for both conversational parties, we expect to see more research in this direction in the next few years.

Even more generally, the context could also include spatial understanding of the environment. For example, the correctness of deictic gestures relies on the information about objects and directions in a scene. To carry communicative value, most of these gestures will therefore require access to visual and/or spatial information beyond what may be contained in the speech – think about a phrase such as “You need to go that way”, which completely lacks information about which direction the system should point. People also use spatial configurations in complex ways while gesturing, for example, placing ideas in a referential space in front of them and then referring to ideas by referring to the space they have been located in. While studies that involve external contexts are quite common for downstream tasks like navigation [SKM^∗19], non-verbal behavior generation in multiple external contexts is up and coming [DWAB22, KNN^∗22] which makes it a promising research direction, if relevant data can be obtained.

Let’s imagine that we have access to all the variables discussed thus far that impact the dynamics of co-speech gestures, such as acoustics, text, speaker identity, emotional state, and external contexts. Further imagine that we are able to gather large-scale datasets with all these variables, which is unlikely to ever happen due to the combinatorical explosion of possible combinations of different factors. Would having this rich input information and broad data coverage be sufficient to confidently predict the specific co-speech gestures that a given speaker will perform? The best we can likely say is “Maybe!” While large scale datasets may enable us to minimize our epistemic uncertainty about gesticulation, it is unclear how significant the stochasticity is, i.e. aleatoric uncertainty, of these gestures will be. The situation is analogous to the problem of prosody in text-to-speech, where there can be many possible acoustic realizations and intonation contours for the same lexical input [WWK15, LTHY17, WSZ^∗18]. Significant variation persists even when a speaker is asked to read the same text several times under exactly the same circumstances [HMS^∗14]. To handle ambiguity in gesture realization, it is compelling to consider probabilistic models, since they can “hallucinate” the missing information and stochastic components of non-verbal behavior, as a way to resolve the one-to-many problem for motion and gesture generation [HAB20, AHKB20].

Gesture authoring enables an animator or system creator to design and edit motion, e.g. making a character appear less nervous or stressed, thus grounding the animation within the designer’s creative intent. Typically, animation design intent is captured through key-framing or motion capture. However, these approaches are difficult to scale for nonverbal behavior because the former requires specialized animation skills, while the latter requires expensive camera setups and laborious post-processing. Automatic gesture generation approaches in part solve the scalability issue by the ability to generate abundant motion data, but they struggle with high-level control. For instance, attempts at handling control either bake in mechanistic, low-level control signals like wrist height, wrist velocity, and radial extent [AHKB20], or they generate gestures that deviate from the intended control specifications [HES^∗22]. Moreover, in multi-speaker scenarios, they are unable to capture the variability of different speakers’ gesticulation, and cannot distinguish between gesture types used in a certain scenario (e.g. deictic gestures for a lecturer in front of display) from gesture style differences between speakers [ALNM20]. Yoon et al. [YPJ^∗21] recently proposed an innovative approach to this challenge: an authoring toolkit that balances gesture quality and authoring effort. The toolkit combines automatic gesture generation using a GAN-based generative model [YCL^∗20] and manual controls. The generative model first produces a gesture sequence from speech input, and animator can interactively edit the motion through low-level pose control and coarse-level style parameters. We think similar gesture authoring approaches that maximize design intent and gesture quality, while minimizing authoring effort will be important for grounding nonverbal behavior within the animator’s creative intent.