ACT2G: Attention-based Contrastive Learning
for Text-to-Gesture Generation

As Li et al. mentioned (Li et al., 2021), recent data driven methods take an approach where the input (text, audio, speaker ID, etc.) and the output (gesture) are mapped in a one-to-one fashion. However gestures are diverse, and the same gesture does not always appear in the same text. Inconsistencies in training data can hinder learning. Therefore, in order to achieve a one-to-many correspondences between gesture and text, we apply clustering on gestures in advance.

We use intermediate features of VAE (Kin, 2014) for the clustering of gestures. The network structure is shown in Fig. 3. One of the main problems when clustering time-series data such as gestures is the difference in sequence length. Especially in networks using RNN as encoders, problems such as vanishing gradient make feature extraction of too long data difficult. Therefore, we train the network using the key poses of the gestures as input. Considering key-poses such as labanotation plays an important role in the analysis of human motion, as shown in previous researches (Manoj et al., 2009; Ikeuchi et al., 2018). We extracted the key-poses $\textbf{p}=\{\textbf{p}_{1},...,\textbf{p}_{n}\}$ using off-the-shelf algorithm (Ikeuchi et al., 2018) in advance and input them into VAE. The key poses are input to the bi-directional LSTM, where the mean $\mu$ and standard deviation $\sigma$ are estimated and the latent feature $\textbf{z}\in\mathbb{R}^{32}$ is randomly sampled from the normal distribution corresponding to that parameter. The number of dimensions of z, 32, was determined empirically by plotting ellipses from the mean $\mu$ and standard deviation $\sigma$ and observing the overlap. Then the gesture is reconstructed by decoder $D$ from z. The number of key poses $n$ ranged from 5 to 12 frames, and data were selected from the representational gestures in the TED Gesture-Type Dataset. Each key pose $\textbf{p}_{t}$ is represented as relative positions of the 8 upper body joints. The loss function for Gesture-VAE is as follows:

where $D_{KL}[.]$ denotes the Kulback-Leibler divergence, $\mathcal{L}$ refers to the likelihood of the parameters of encoder and decoder (i.e., $\theta$ and $\phi$) and $\textbf{p}^{\prime}$ denotes the result of reconstructing the key-poses. Latent feature z are then used for gesture clustering. Gestures were clustered into 40 clusters by K-Means algorithm.

The results of clustering the gestures are shown in Fig. 4. By extracting gesture features in VAE, each gesture was mapped and clustered in continuous space. For example, cluster 39 and cluster 12 are in opposite positions in Fig. 4, with the gesture of cluster 39 facing left, but the gesture of cluster 12 facing right. In addition, the gesture in cluster 21 has the left hand down, while the gestures in clusters 8 and 14 have the arm up gesture.

Gesture and text in the same cluster were labeled Positive, while data in different clusters were labeled Negative. These positive and negative labels are used during contrastive learning in Sec. 3.3.

Designing gestures manually requires specialized knowledge and is very labor intensive. Gesture generation using AI simplifies this task. However these approaches cannot represent the gesture as the user would like to design it. Therefore, we propose a method to generate gestures by focusing on words that are likely to be expressed in gestures. Representational gestures in the TED Gesture-Type Dataset are annotated with the corresponding words in the utterance correspond to that gesture. We use this data to estimate attention weight A that represents the weight of the word to focus on.

Our proposed network structure is shown in Fig. 5, which is the $E_{t}$ portion of Fig. 2. First, each word is converted to a word embedding set $\textbf{w}=\{\textbf{w}_{1},...,\textbf{w}_{T}\mid\textbf{w}_{i}\in\mathbb{R}^{768}\}$ by using pre-trained BERT (Devlin et al., 2019). $T$ is the number of input words, set to 32. Then, the attention weight $\textbf{A}=\{A_{1},...A_{T}\}$ is estimated by encoder $E_{t1}$ from w, which consists of a fully connected layer:

where $f$ is the following normalization function to ensure that text features are not affected by word count:

Another word feature $\textbf{w}^{\prime}$ is estimated from w by using encoder $E_{t2}$ and by multiplying with A. During training, w is always the same value for the same word because BERT is frozen, but $\textbf{w}^{\prime}$ is fine-tuned by $E_{t2}$.

Each word feature is then concatenated and the encoder $E_{t3}$ outputs a text feature $\textbf{f}_{t}$ as in the following equations:

where $\|_{i=1}^{T}$ represents the vector concatenation from 1 to T and $\odot$ represents the Hadamard product. Attention weight A is regularized by binary cross entropy loss:

where $\hat{A}_{i}$ is a ground truth with label 1 for the word corresponding to the gesture and 0 for the others. The part on the right side of Fig. 5, illustrate estimation process of A. This part was pre-trained with data from all representational gestures in the TED Gesture-Type Dataset. When training contrastive learning, which is discussed in Sec. 3.3, A is fine-tuned by using text features $\textbf{f}_{t}$ to reconstruct the gesture. During inference, gestures are generated from text features $\textbf{f}_{t}$, where A can be explicitly given. In Sec. 4.5, we describe an experiment in which A is also input to generate a gesture.

Many recent gesture generation methods output sequences of poses directly from the network, but they are often overly slow or jerky. We assume that the slow movement problem is due to the generator’s RNN and autoregressive model. Therefore, we propose a method to create a gesture library and search for the appropriate gesture from the library. ACT2G generates gestures using contrastive learning, which is often used to improve text embedding (Kiros et al., 2014) or image and video retrieval (Miech et al., 2018; Bain et al., 2021).

The overview of the network structure is shown at Fig. 2. Key-poses are extracted from the gesture and input to encoder $E_{g}$, which consists of a bi-directional LSTM or FCN. $E_{g}$ outputs the gesture features $\textbf{f}_{g}\in\mathbb{R}^{32}$, and the gesture is reconstructed through the decoder similar to Gesture-VAE, described in Sec. 3.1. Contrastive loss is as follows:

where $B$ is the batch size, and $\textbf{P}\in\mathbb{R}^{B\times B}$ is the positive matrix between each data defined by the gesture clustering described in Sec. 3.1, and is a square matrix of 1 if each data is positive and 0 if negative. $\textbf{D}\in\mathbb{R}^{B\times B}$ is the L2 distance matrix between text feature $\textbf{f}_{t}$ and gesture feature $\textbf{f}_{g}$. And $\textbf{m}\in\mathbb{R}^{B\times B}$ is the margin, which is a square matrix with all elements $m$. We set $m=20$ in practice. This contrastive loss means, in the multimodal space, the distance between gesture-text pairs, defined as positive in Sec. 3.1, is small, while the distance between pairs, defined as negative, is large. The loss to reconstruct the key-poses is:

The loss function for the entire framework is as follows:

Two parameters $\alpha$ and $\beta$ controls the weights of the loss terms, and they were empirically determined to 10 and 2, respectively. The multimodal space constructed by the contrastive loss is shown in Fig. 6. The blue dots, orange dots, and red lines represent text features $\textbf{f}_{t}$, gesture features $\textbf{f}_{g}$, and pairs of mutually Positive gestures, respectively. Even though all positive data are connected with each other by a red line, fewer of them are visible as a line compared to the number of dots, indicating that the data that are positive with each other are clustered together.

During inference, using reconstructed key-poses is difficult to humanize the gesture by simply interpolating between the key-poses. Therefore, we propose a method that uses the multimodal space to retrieve appropriate gestures from a gesture library. The gesture library contains the gestures in the training data and their corresponding positions in the multimodal space as shown in Fig. 6. When text is input, the text feature is extracted by the encoder introduced in Sec. 3.2, and the gesture is randomly sampled from nearby that text feature in the multimodal space. Long input text is empirically divided into 8 word and entered into the network. Gesture speed is adjusted to match the length of the human voice, if present, or the synthesized voice, if absent. The gestures generated for each segmented text were combined by spline interpolation.

We first conducted an ablation study to gain more insights into our framework. We used Diversity score (Li et al., 2021) and FGD score (Yoon et al., 2020) as metrics for quantitative evaluation and perceptual scores from the user study for qualitative evaluation. The Diversity proposed by Li et al. is not suitable for computation with gesture data of different length, since the distance between gestures is computed with L1 distance of each joint in the same frame. We therefore used the distance between latent features trained in the Gesture-VAE introduced in Section 3.1 as the distance between gestures:

where $\mu$ refers to the latent vector of the VAE and $N$ is the number of motion clips.

The FGD proposed by Yoon et al. (Yoon et al., 2020) could only handle short gestures of 34 frames. Therefore we extracted key poses from the gestures and used them as input to the Gesture-VAE (Fig. 3), allowing us to evaluate gestures with an average length of 233 frames and a maximum length of 1795 frames. Key poses were extracted using the method of Ikeuchi et al. (Ikeuchi et al., 2018) method, which summarizes the entire motion. The gestures, the input to the VAE, were padded to 64 frames and the dimension of the latent space was set to 256 dimensions empirically. During the ablation study, we trained the VAE on the Trinity Gesture Dataset, and tested with 972 motion clips from TED Gesture Dataset.

In the user study, 50 participants in the Amazon Mechanical Turk rated 3 kind of gestures on a scale of 1-100 for the question ”How appropriate are the gestures for the speech?”. The requirements for participants in Amazon Mechanical Turk and the gestures used in the evaluation are the same as those described in Sec. 4.3 Comparison with Previous Methods below.

Table 1 shows the results of the ablation study. W/o contrastive is the case of no contrast learning, i.e., multimodal space is not created. After text features were extracted by BERT, gestures were selected by nearest neighbor method with the features from TED Gesture-Type Dataset. W/o attention means the way the gesture is generated without encoder $E_{t1}$ in Fig. 5. The result between w/o contrastive and full model in Diversity shows that it is possible to generate a variety of gestures by a multimodal space of text and gestures, rather than a simple nearest neighbor method with only text. The user study results show a statistically significant difference for the full model over the other two ways. The results with w/o contrastive were particularly significant, showing that the multimodal space created by contrastive learning allows for better gesture selection. Although the result of the full model was inferior to the other two methods in terms of the FGD, since FGD just means how similar the gesture is to the original gesture and it is not necessarily an appropriate gesture for speech, we prefer the user score and use the full model in subsequent evaluation.

We conducted two user studies to compare gestures generated by ACT2G with existing methods and original gestures. The existing methods are compared with (1) Trimodal (Yoon et al., 2020), (2) HA2G (Liu et al., 2022b), (3) Deep Gesture Generation (Teshima et al., 2022) that serve as a baseline, since they are also considering semantic information for gesture generation. (1) Trimodal is a method that takes text, audio, and speaker identity as input and generates gestures using bidirectional GRU model. This model is trained with TED Gesture Dataset (Yoon et al., 2019). (2) HA2G is also a method that takes text, audio, and speaker identity as input and generates gestures using decoders that are hierarchically divided into body parts. When inferring, gestures are generated by only the audio and speaker identity as input without text. In both methods Trimodal and HA2G, speaker identities were chosen randomly from the training data, TED Gesture Dataset (Yoon et al., 2019). (3) Baseline takes text as input and uses a gesture library to generate gestures. This model generates gestures by each gesture-type generator after predicting the probability of gesture type (beat, representational, non-gesture) from each word. These methods were evaluated using test data in the TED Gesture Dataset.

We built an evaluation environment similar to the GENEA Challenge (Kucherenko et al., 2021b). As the user interface for evaluating gesture videos, we used HEMVIP (Jonell et al., 2021), which displays multiple videos in parallel and is intuitive and easy to use. Also, the BVH Visualizer (Kucherenko et al., 2021b) was used to visualize the gestures. Since gestures generated by ACT2G, Baseline and HA2G had to be converted to BVH format to use this Visualizer, each joint position was converted to Euler angle. The two questions we prepared for the gesture evaluation items are also the same as in GENEA Challenge, as follows:

• $(a)$

  Appropriateness How appropriate are the gestures for the speech?

• $(b)$

  Human-likeness How human-like does the gesture motion appear?

When evaluating human-likeness, gestures were evaluated only on the basis of movement, with no audio. Study participants were recruited through the crowdsourcing platform Amazon Mechanical Turk instead of Prolific used in GENEA Challenge. The participants were selected if they satisfied the following three requirements: 1) they had completed more than 500 tasks, 2) their approval rate of task is over 90%, and 3) they had passed the attention check. The attention checks are a way to check the quality of the worker, such as by displaying text such as ”Attention! Please rate this video 35” in the some video or replacing the audio. 108 participants met the requirements in the Appropriateness and 115 in the Human-likeness. Participants rated each gesture on a 100-point scale labeled (from best to worst) ”Excellent,” ”Good,” ”Fair” ”Poor” and ”Bad” in 20-point intervals. We randomly selected 28 gestures from those annotated as representational gestures in the TED Gesture-Type Dataset with an average original gesture’s arm speed above a threshold.

Fig. 7 (a) shows the results of the ”Appropriateness”, Fig. 7 (b) shows the results of the ”Human-likeness”. The orange line represents the median value, and the green triangle represents the mean value. The results of ANOVA showed ACT2G was significantly higher than Baseline, HA2G and Trimodal with $p<0.01$ in both Apprppriateness and Human-likeness indicators. Examples of a generated gesture are shown in Fig. 9. In this figure, the parts where representational gestures might appear are indicated by colored texts and boxes. We can find that Ours has a higher frequency of representational gestures than the other methods. In the example in the upper row, original shows a gesture of spreading his right hand twice, which may represent ”chair” or ”turn on”. On the other hand, our method makes a gesture corresponding to ”tie me on the chair” by rotating the left hand, and a gesture corresponding to ”loud” by spreading the arms wide. Baseline shows a movement like putting something down, while HA2G and Trimodal show a movement like spreading the right hand and raising both hands to shoulder level when saying ”turn on,” respectively. In the example in the middle row, original shows the movement of opening both hands when saying ”most”. Our gesture shows the movement of both arms spread wide when saying ”a lot of” and the movement of the right hand to the center when saying ”very little”. Baseline generates a non-gesture, almost no hand movement. HA2G is gently tightening and opening the elbows, while Trimodal is spreading the left hand down at the ”little” part and then opening both arms at the ”most” part. In the example in the bottom row, the original gesture is a beat-like gesture with ”here”, ”us”, and ”all” each with the arms swinging down. In ours, the right hand spreads when saying ”right here” and the right elbow rises when saying ”all”. Baseline and HA2G are both very slow beat gestures. Trimodal is also slow motion and raises the right hand at the ”all”. The slow movement of previous methods is considered that GRUs and autoregressive models may have caused excessively smooth movements. As for HA2G and Trimodal, there may be some influence of randomly selected speaker identities.

One major issue in gesture generation research is how to establish evaluation criteria. We focused on user scores to evaluate gestures according to the GENEA Challenge (Kucherenko et al., 2021b, c; Yoon et al., 2022), but it is very labor intensive to evaluate each generated gesture with a user study each time. Therefore, many previous studies have quantitatively evaluated gestures, and sought better evaluation metrics. In this study, we report on the better metric in this experiment by comparing how close the previously proposed metrics are to the distribution of user scores. We investigated three metrics used in previous studies: L1 norm (Ginosar et al., 2019; Li et al., 2021), FGD (Yoon et al., 2020; Liang et al., 2022; Liu et al., 2022b), and Jerk (Kucherenko et al., 2019, 2021b). When calculating L1 norm, the speed of the generated gestures were normalized to make the duration as same as the original gestures. Fig. 8 shows the relationship between user score and each metric, where the user score means the result of the Appropriateness described in Sec. 4.3. X-axes of 8(a) and (b) are the L1 norm and the FGD, respectively, and the smaller the better. The correlation coefficients between L1 norm, FGD and user score are -0.0305 and 0.0055, respectively, indicating little correlation. On the other hand, the correlation coefficient for jerk is 0.4459, indicating the certain correlation between them. We thought that the reason for the trend toward higher ratings with larger jerks might be explained by simple fact that meaningful texts are usually spoken by large movement of human.

In Section 4.3 we evaluated four different models trained on the TED Gesture dataset on test data from the same dataset. While the TED Gesture Dataset contains data from a variety of speakers speaking on a variety of topics, the situation in which they are speaking in front of an audience is the same for all data. In addition, the test data did not include anything other than representational gestures such as beat gesture and non-gesture. Therefore, we conducted an additional user study to confirm the generalization performance.

For the test data, we used the Trinity Speech-Gesture Dataset (Ferstl and McDonnell, 2018), which is mo-cap data from a single speaker speaking on a variety of topics in the experimental room. The comparison methods include ours, baseline, and HA2G trained with the TED Gesture Dataset, plus Gesticulator (Kucherenko et al., 2020) trained with the Trinity Speech-Gesture Dataset. The experimental environment, including the interface for evaluation and the gesture visualization method, was the same as in the user study in Sec. 4.3, and only human-likeness was used as a question item for participants. We randomly sampled 30 gestures from test data, and 115 participants rate the gestures.

Fig. 10 shows the results for the user study with Trinity Speech-Gesture Dataset. Statistical tests showed Original gesture was significantly best for any other gesture at $p<0.01$. Ours was significantly superior to Baseline at $p<0.05$, and to HA2G and Gesticulator at $p<0.01$. As can be seen from Fig. 8, users tend to prefer the fast-moving (larger movement) gestures when they see five different gestures in parallel. Our method tends to generate fast-moving representational gestures, which is why it was rated higher than the baseline and other methods. Gesticulator outperformed HA2G by p¡0.01, possibly because HA2G is a model that is trained on the TED Gesture Dataset, while Gesticulator is a model trained on the Trinity Speech-Gesture Dataset.

The attention-based text encoder described in Sec. 3.2 predicts attention, the likelihood that a gesture will appear, for each word. We assumed that by pre-defining the attention corresponding to each word and inputting it into the network, the gesture corresponding to the word with the highest attention weight would appear. Attention weights were set to 0.5 for specified words and 0.1 for other words.

Fig. 11 shows the results with the text ”there are a lot of little children there. Fig. 11 (a) shows the result of attention weight A estimated by the network with only text input, and (b), (c), and (d) are the results of generated gestures with higher weights for A corresponding to ”a lot of”, ”little children”, and second ”there”, respectively. The each value of A is normalized by Equation 3, including the CLS token and padding tokens. In (a) gesture, the right hand is first rotated in a small motion, then both hands are brought forward. This appears to represent a ”little” or a second ”there”. Also, (b), (c), and (d) gestures appear to represent ”a lot of” with both arms outstretched, ”little children” with arms down, and ”there” with right arm extended, respectively. Because of random sampling from the gesture library, it may be generated slightly different gestures even for the same text and attention weight. Although a variety of gestures can be generated by entering text alone, even more diverse gestures can be generated by changing the attention weight.