Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model

Each 3D human pose is represented by the position of $J$ keypoints. Each keypoint has three coordinates in 3D. Therefore, a 3D human pose can be denoted by $\mathbf{p}\in\mathbb{R}^{J\times 3}$. 3D human motion is a sequence of poses and hence can be written as $\mathbf{x}\in\mathbb{R}^{L\times J\times 3}$ where $L$ represents the number of time steps. For simplicity, we flatten the pose representation into a one-dimensional vector. Then the motion can be represented as $\mathbf{x}\in\mathbb{R}^{L\times C}$ where $C=J\times 3$.

The denoising diffusion probabilistic model consists of a forward process and a reverse process. The forward diffusion process converts the original highly structured and semantically relevant keypoints distribution into a Gaussian noise distribution. We denote motion with increasing levels of noise as $\mathbf{{x}_{0}},\mathbf{{x}_{1}},...,\mathbf{{x}_{T}}$. The forward diffusion process is modeled as a Markov chain:

where $q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}\right)$ is denoted as the forward process, adding noise to motions at the previous time step and generate the distribution at the next time step. Given a set of pre-defined hyper-parameters $\beta_{1},\beta_{2},...,\beta_{T}$, each transition step can be written as

where $\beta_{t}$ controls the diffusion rate of the process and usually ranges from 0 to 1.

The reverse diffusion process is a generation process where keypoints sampled from a Gaussian noise distribution $p\left(\mathbf{x}_{t}\right)$ are given as the initial input. Then the initial input is progressively denoised on an inverse Markov chain following the given text description. The text description is encoded into a latent variable $\mathbf{z}$ using the BERT [16] model. $\theta$ are the parameters of the denoising model. The reverse diffusion process can be written as follows:

where $\boldsymbol{\mu}_{\boldsymbol{\theta}}$ is the target we want to estimate by a neural network. $t$ is the timestep indicating where the denoising process has conducted, which is encoded as a vector based on the cosine schedule [17]. $p\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{z}\right)$ represents the reverse transitional probability of keypoints from one step to the previous step.

In the previous subsection, we established the forward and reverse diffusion process for 3D human motion generation from textual descriptions. In this subsection, we will derive the mathematical expressions for our training objective of the reverse diffusion process.

In order to approximate the intractable marginal likelihood $p_{\boldsymbol{\theta}}\left(\mathbf{x}\right)$, we maximize a variational lower bound to make the objective optimal:

It is the same objective as in DDPM [9] except for involving the text embedding $\mathbf{z}$. We add text embedding $\mathbf{z}$ and time embedding $\mathbf{t}$ into a new embedding during our specific training process. For simplicity, we still represent the fused embedding with $\mathbf{t}$, which makes the objective consistent with DDPM:

The same as DDPM simplifies the objective, our final training objective is

where $\epsilon$ is the noise sampled from standard Gaussian distribution, $\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{\mathbf{t}},t\right)$ is the output of the noise prediction model.

In principle, the training target is to minimize Eq. (7). The simplified training and sampling algorithm is as follows:

1. 1.

   Recognition Precision. We use a pre-trained Motion encoder and Text encoder to compute the generated samples’ embedding and text embedding, and further compare the similarity of those.

2. 2.

   Frechet Inception Distance (FID). Unlike agreeing to use the inception network for feature extraction for image FID, there is no agreed network for 3D human motion. Meanwhile, to better compare with Temporal VAE’s experimental results, we use the same encoder as in their work to measure FID. Their model is both trained and evaluated on this encoder so it puts our method at a disadvantage. We only show this as a reference point.

3. 3.

   Diversity. We propose this new metric which evaluates diversity without an encoder. Specifically, diversity measures how much the generated motions diversify with the same text input. Given a set of generated 3D motions with $T$ text inputs. For $t$-th motion, we randomly sample two subsets with the same size $S$, and we use $m$ to represent the generated motion. The diversity can be formalized as:

   $$\text{ Diversity }=\frac{1}{T\times S}\sum_{t=1}^{T}\sum_{i=1}^{S}\left\|\mathbf{m}_{t,i}-\mathbf{m}_{t,i}^{\prime}\right\|_{2}$$

   (8)

   Method	R Precision(T-1)$\uparrow$	R Precision(T-2)$\uparrow$	R Precision(T-3)$\uparrow$	FID$\downarrow$	Diversity$\uparrow$	Variance$\rightarrow$
   Real motions	0.511	0.703	0.707	0.002	0	9.503
   Seq2Seq [19]	0.180	0.300	0.396	11.75	0	6.223
   Language2Pose [6]	0.246	0.387	0.486	11.02	0	7.626
   Text2Gesture [20]	0.165	0.267	0.345	7.664	0	6.409
   MoCoGAN [3]	0.037	0.072	0.106	94.41	9.421	0.462
   Dance2Music [21]	0.033	0.065	0.097	66.98	7.235	0.725
   Temporal VAE [8]	0.457	0.639	0.740	1.067	18.529	9.188
   Ours	0.406	0.612	0.735	10.21	23.692	7.660

4. 4.

   Variance. This metric measures the variance of the generated samples without considering the text input. For any two subsets of the dataset with the same size $S$, suppose after encoding with the Temporal VAE method, their motion feature vectors are $\left\{\mathbf{v}_{1},\ldots,\mathbf{v}_{S}\right\}$ and $\left\{\mathbf{v}_{1}^{\prime},\ldots,\mathbf{v}_{S}^{\prime}\right\}$. The variance of motions is calculated as:

   $$\text{ Variance }=\frac{1}{S}\sum_{i=1}^{S}\left\|\mathbf{v}_{i}-\mathbf{v}_{i}^{\prime}\right\|_{2}$$

   (9)

Table 1 shows the quantitative evaluation results for the HumanML3D dataset. For Recognition Precision, our method and Temporal VAE are far more superior than other methods. This indicates that the generated keypoint sequences are highly correlated with the textual descriptions. For the FID and Diversity metrics, Seq2Seq [19], Language2Pose [6] and Text2Gesture [20] can generate natural results, but due to their deterministic model design, they have no diversity for this task. MoCoGAN [3] and Dance2Music [21] have unfavorable FID scores but achieve diversity. Clearly, there is a trade-off between the naturality and diversity of the generated results. Overall, Our method and Temporal VAE are far better than other methods in terms of naturality and diversity.

Fig 2 illustrates the visualized results of our method vs. Temporal VAE. For the input “A person walks steadily along a path while holding onto rails to keep balance”, our method generates motions with different directions, velocities and distances. This shows that the diversity of our method is beyond numerical variances, reaching semantic levels. We do not observe this level of diversity in other models. For instance, the generated results from Temporal VAE contain only minor differences. Our model even interprets ‘path’ to be ‘stairs’, which is shown in row 4 and row 5.

Thanks to the pre-trained model BERT [16] and classifier-free guidance technique [12], our diffusion model has very impressive zero-shot capability. As shown in Fig 3, for unseen actions and combinations, our method can generate natural and semantically relevant results.