Generating Smooth Pose Sequences for Diverse Human Motion Prediction

In this section, we first briefly review the use of deep generative models in the context of human motion prediction and then introduce our solution for diverse prediction.

To overcome this, we propose to explicitly generate $K$ future motions $\{\hat{\mathbf{Y}}_{j}\}_{j=1}^{K}$ for each sample during training, and explicitly promote their diversity. Because we aim for diversity, we cannot encourage all generated motions to match the ground truth. Therefore, we redefine the reconstruction error so that at least one of them is close to the ground truth. This yields the loss

$$\mathcal{L}_{r}=\min_{j}\|\hat{\mathbf{Y}}_{j}-\mathbf{Y}\|^{2}\;,\vspace{-0.2cm}$$

(3)

where $j\in\{1,2,\cdots,K\}$ indexes over the generated motions for one sample.

This loss, however, imposes constraints on only one of the $K$ generated motions. To better constrain the others, we leverage the intuition that, while the dataset contains only one ground-truth future motion for each past motion, several sequences have similar past motions. For each past motion, we therefore search for the training samples with similar past motions, based on a distance threshold, and take their future motion as pseudo ground truth. Let $\{\mathbf{Y}_{p}\}_{p=1}^{P}$ denote the resulting pseudo ground truths. Then, we define the multi-modal reconstruction error

$$\mathcal{L}_{mm}=\frac{1}{P}\sum_{p=1}^{P}\min_{j}\|\hat{\mathbf{Y}}_{j}-\mathbf{Y}_{p}\|^{2}\;,\vspace{-0.2cm}$$

(4)

where $j\in\{1,2,\cdots,K\}$. It encourages our generator to cover each pseudo ground truth with at least one sampled motion.

To further explicitly encourage diversity across the generated motions, following [57], we use the diversity-promoting loss

$$\mathcal{L}_{d}=\frac{2}{K(K-1)}\sum_{j=1}^{K}\sum_{k=j+1}^{K}e^{-\frac{\|\hat{\mathbf{Y}}_{j}-\hat{\mathbf{Y}}_{k}\|_{1}}{\alpha}}\;,$$

(5)

where $\alpha$ is a normalizing factor.

One drawback of this diversity loss, however, is that it may lead the model to produce unrealistic and physically-invalid motions, particularly if $P<K$ above, which leaves some motions unregularized. The most straightforward way to overcome this would be to learn a motion prior. This, however, would require an impractically large amount of training data. Instead, we therefore leverage the observation that natural motions are composed of valid human poses and thus introduce the pose prior and angle losses discussed below.

Following the normalizing flow literature [47, 15], this allows us to compute the likelihood of a pose $\mathbf{x}$ as

$$p(\mathbf{x})=g(\mathbf{h})\left|\det\left(\frac{\partial f}{\partial\mathbf{x}}\right)\right|\;,\vspace{-0.2cm}$$

(6)

where $g(\mathbf{h})=\mathcal{N}(\mathbf{h}|0,\mathbf{I})$ and $\det(\frac{\partial f}{\partial\mathbf{x}})$ is the determinant of the Jacobian matrix of $f(\cdot)$.

In practice, the function $f$ is modeled via a deep network. Considering model size and inference efficiency, we choose a simple network with only 3 fully-connected layers, which is in stark contrast with the much larger architectures in the recent normalizing flow literature [15, 29, 43, 25]. To ensure that $f(\cdot)$ is invertible, we compute the weights of each layer via a QR decomposition and use monotonic activation functions. More details about our normalizing flow architecture are provided in the supplementary material.

Given a dataset $\mathcal{D}$ of valid human poses, we learn the function $f$ by maximizing the log-likelihood of the samples in $\mathcal{D}$. This can be written as

	$\displaystyle f^{*}$	$\displaystyle={\arg\max}_{f}\sum_{\mathbf{x}\in\mathcal{D}}{\log{p(\mathbf{x})}}$		(7)
		$\displaystyle={\arg\max}_{f}\sum_{\mathbf{x}\in\mathcal{D}}{\log{g(\mathbf{h}})+\log{\left|\det\left(\frac{\partial f}{\partial\mathbf{x}}\right)\right|}}\;.\vspace{-0.2cm}$		(8)

Given the trained function $f$, we then define a loss function to encourage our generator to produce valid poses. Specifically, we write it as the negative log-likelihood of a generated human pose $\hat{\mathbf{x}}$, which yields

	$\displaystyle\mathcal{L}_{nf}$	$\displaystyle=-\log p(\hat{\mathbf{x}})$		(9)
		$\displaystyle=-\log g(\hat{\mathbf{h}})-\log\left|\det\left(\frac{\partial f^{*}}{\partial\hat{\mathbf{x}}}\right)\right|\;,\vspace{-0.2cm}$		(10)

where $\hat{\mathbf{h}}=f(\hat{\mathbf{x}})$ and $g(\hat{\mathbf{h}})=\mathcal{N}(\hat{\mathbf{h}}|0,\mathbf{I})$.

To this end, as shown in Fig. 4, we first compute unit length vectors, which represent either the orientations of body parts or the directions of limbs. A body part orientation is the normal of the plane defined by 3 joints. For example, the torso plane, shown in blue in Fig. 4 (a), is defined by the left and right shoulders and the pelvis. A limb direction is defined by 2 joints of the limb. We then compute the angle between those vectors for every pose in $\mathcal{D}$ and define the valid range based on the minimal and maximal such angle. We list all angles and their valid ranges in the supplementary material.

Let $\{a_{j}\}_{j=1}^{L}$ denote the $L$ pre-defined angles, and $l_{a_{j}}$ and $u_{a_{j}}$ the lower bound and upper bound of angle $a_{j}$, respectively. Given a human pose $\hat{\mathbf{x}}$, we write our joint angle loss for $a_{j}$ as

$\displaystyle\mathcal{L}_{a_{j}}=\begin{cases}(a_{j}(\hat{\mathbf{x}})-l_{a_{j}})^{2},\,\text{if }a_{j}<l_{a_{j}}\\ (a_{j}(\hat{\mathbf{x}})-u_{a_{j}})^{2},\,\text{if }a_{j}>u_{a_{j}}\\ 0,\,\text{otherwise}\\ \end{cases}\vspace{-0.2cm}$

(11)

where $a_{j}(\hat{\mathbf{x}})$ is the angle value calculated from pose $\hat{\mathbf{x}}$. We then combine the different angles in our final angle loss $\mathcal{L}_{a}=\sum_{j=1}^{L}\mathcal{L}_{a_{j}}$.

Specifically, given the past motion sequence $\mathbf{X}$, we first replicate the last pose $T$ times to generate a temporal sequence $\tilde{\mathbf{X}}$ of length $H+T$, where $T$ is the length of the future sequence to predict. We then compute the DCT coefficients of this sequence $\mathbf{C}$ as

$$\mathbf{C}=\tilde{\mathbf{X}}\mathbf{T}\;,\vspace{-0.2cm}$$

(12)

where $\mathbf{T}\in\mathbb{R}^{(H+T)\times M}$ and each column of $\mathbf{T}$ represents a predefined DCT basis; $\tilde{\mathbf{X}}\in\mathbb{R}^{D\times(H+T)}$ and each row of $\tilde{\mathbf{X}}$ is the trajectory of a joint coordinate; $\mathbf{C}\in\mathbb{R}^{D\times M}$ and each row of $\mathbf{C}$ represents the first $M\leq H+T$ DCT coefficients for a trajectory.

We then make our generator predict the DCT coefficients of the future motion $\hat{\mathbf{C}}$ given those of the past motion $\mathbf{C}$. Given the predicted coefficients, we recover the future motion via inverse DCT as

$$\hat{\mathbf{Y}}=\hat{\mathbf{C}}\mathbf{T}^{T}\;,\vspace{-0.2cm}$$

(13)

where $\hat{\mathbf{C}}\in\mathbb{R}^{D\times M}$ and $\hat{\mathbf{Y}}\in\mathbb{R}^{D\times(H+T)}$. As in [39], our generator also outputs the past motion to encourage the transition from past poses to future ones to be smooth. See the supplementary material for more detail.

The diverse motion prediction framework discussed above does not provide any control over the generated motions. For controllable motion prediction, our goal is to predict future sequences that share the same motion for parts of the body while being diverse for the other parts, e.g., same leg motion but diverse upper-body motion. To this end, instead of directly modeling the joint data distribution as discussed above, we propose to model a product of sequential conditional distributions.

Specifically, let a human motion be split into $N$ different body part motions, that is, $\mathbf{Y}=[\mathbf{Y}^{(1)},\mathbf{Y}^{(2)},\cdots,\mathbf{Y}^{(N)}]$, where $\mathbf{Y}^{(i)}\in\mathbb{R}^{T\times D_{i}}$ defines the motion of the $i^{th}$ body part, e.g., the left leg. Then, the future body motion distribution $p(\mathbf{Y}|\mathbf{X})$ can be expressed as

$p(\mathbf{Y}|\mathbf{X})=p(\mathbf{Y}^{(1)}|\mathbf{X})p(\mathbf{Y}^{(2)}|\mathbf{X},\mathbf{Y}^{(1)})\cdots p(\mathbf{Y}^{(N)}|\mathbf{X},\{\mathbf{Y}^{(i)}\}_{i=1}^{N-1})$ .

(14)

Each of the $N$ conditional distributions describes the motion of a particular body part given the motion of the previous body parts. Similar to the standard deep generative model discussed above, we model each conditional distribution as

$$\mathbf{z}^{(i)}\sim p^{(i)}(\mathbf{z})\,,\vspace{-0.2cm}$$

(15)

$$\mathbf{Y}^{(i)}=\mathcal{G}^{(i)}(\mathbf{z}^{(i)},\mathbf{X},\{\mathbf{Y}^{(j)}\}_{j=1}^{i-1})\,,\vspace{-0.2cm}$$

(16)

where $i\in\{1,2,\cdots,N\}$ and $p^{(i)}(\mathbf{z})$ is a standard Gaussian distribution $\mathcal{N}(0,1)$. In other words, to compute a future motion, we sample different random variables $\{\mathbf{z}^{(i)}\}_{i=1}^{N}$ and decode them sequentially to obtain the future motion of each body part. By fixing some random variables $\{\mathbf{z}^{(i)}\}_{i=1}^{J}$ while varying the others $\{\mathbf{z}^{(i)}\}_{i=J+1}^{N}$, our model can generate multiple times the same motion for body parts $\{\mathbf{Y}^{(i)}\}_{i=1}^{J}$ while producing diverse motions for the other body parts $\{\mathbf{Y}^{(i)}\}_{i=J+1}^{N}$.

Note that our generator design lets us achieve both diverse and controllable motion prediction at the same time. During training, for each past motion of the $i$-th body part, we generate $K$ motions for the $(i+1)$-th body part leading to $K^{N}$ future motions given one past motion. For example, as illustrated in Fig. 3, we sample $K$ different leg motions $\{\mathbf{Y}_{j}^{(1)}\}_{j=1}^{K}$ and for each leg motion $\mathbf{Y}_{j}^{(1)}$, we generate $K$ different upper-body motions $\{\mathbf{Y}_{j,k}^{(2)}\}_{k=1}^{K}$. In this context, we then re-write our diversity-promoting loss as a per-body-part loss, leading to

$\displaystyle\mathcal{L}_{d_{i}}=\frac{2}{K(K-1)}\sum_{j=1}^{K}\sum_{k=j+1}^{K}e^{-\frac{\|\hat{\mathbf{Y}}_{\cdot,j}^{(i)}-\hat{\mathbf{Y}}_{\cdot,k}^{(i)}\|_{1}}{\alpha^{(i)}}}\;,\vspace{-0.2cm}$

(17)

where $i\in\{1,2,\cdots,N\}$ is the index of the body part, and $\mathbf{Y}_{\cdot,k}^{(i)}$ is the $k$-th future motion for the $i$-th body part¹¹1Note that we eliminate the sample index of previous body parts for simplicity. For example, the $k$-th future motion of the $2$nd body part should be represented as $\mathbf{Y}_{j,k}^{(2)}$, where $j\in\{1,2,\cdots,K\}$ is the sample index of the $1$st body part.. Altogether, our final training loss is then expressed as

(18)

where $N$ is the number of body parts.

In practice, inspired by [39], we define each generator $\mathcal{G}^{(i)}$ as a Graph Convolutional Network (GCN) [31] with several graph convolution layers. Given a feature map $\mathbf{F}\in\mathbb{R}^{D\times|\mathbf{F}|}$, each such layer computes a transformed feature map $\mathbf{F}^{{}^{\prime}}\in\mathbf{R}^{D\times|\mathbf{F}^{{}^{\prime}}|}$ as $\mathbf{F}^{{}^{\prime}}=\tanh(\mathbf{A}\mathbf{F}\mathbf{W})$, where $\mathbf{A}\in\mathbb{R}^{D\times D}$ represents a fully connected graph with learnable connectivity and $\mathbf{W}\in\mathbb{R}^{|\mathbf{F}|\times|\mathbf{F}^{{}^{\prime}}|}$ is a matrix of trainable weights. The details of our network architecture are provided in the supplementary material.

Following [57], we evaluate our method on 2 motion capture datasets, Human3.6M [26] and HumanEva-I [49], and adopt the same training and testing settings as in [57] on both datasets.

Our method consistently outperforms all baselines on all metrics. In general, stochastic motion prediction methods outperform deterministic ones in accuracy (ADE, FDE, MMADE, MMFDE). The reason is that, for multi-modal datasets, deterministic prediction models tend to predict an average mode, which leads to higher errors. For stochastic motion prediction, there is a trade-off between sample diversity (APD) and accuracy. One can achieve high diversity while sacrificing some performance in accuracy, e.g., DSF [58], or vice versa, e.g., BoM [7].

Let us now focus on DLow [57], which constitutes the state of the art. Although, its learnable sampling strategy balances diversity and accuracy well, leading to better results than the other baselines, our method outperforms it on all metrics for both datasets. Note that, on Human3.6M, our method achieves $8\%$ lower reconstruction error (ADE) with $25\%$ higher sample diversity (APD). The qualitative comparisons in Fig. 6 further evidence that our predictions are closer to the GT and more diverse. We further provide a detailed comparison in Fig. 7, which shows that DLow [57] still produces some invalid poses, highlighted with red boxes. This can be caused by its lack of a pose-level prior.

In Table 3, we evaluate the influence of our different loss terms. In general, there is a trade-off between the diversity loss $\mathcal{L}_{d}$ and the other losses. Without the diversity loss, the model yields the best ADE at the cost of diversity. By contrast, removing any other loss term leads to higher diversity but sacrifices performance on the corresponding accuracy metrics. Note that, in Table 3, we also report the negative log-likelihood (NLL) of the poses obtained from our pose prior to demonstrate their quality. Although, without the pose prior loss $\mathcal{L}_{nf}$ (first row), we can achieve higher diversity and almost the same accuracy, such diversity gain comes with a dramatic decrease in pose quality (NNL). In other words, while some samples are accurate, many others are not realistic. For qualitative comparison, please refer to the supplementary material.

Our pose prior aims to model the distribution of valid human poses. As the validity of a pose mostly depends on the kinematics of its joint angles, instead of modeling the distribution of 3D joint coordinates, which couple the limb directions with their lengths, we propose to learn the distribution of limb directions only. In particular, given the $i$-th joint coordinate $\mathbf{J}_{i}\in\mathbb{R}^{3}$ and the coordinates of its parent joint $\mathbf{J}_{p_{i}}$, the limb direction can then be computed as

$\displaystyle\mathbf{d}_{i}=\frac{\mathbf{J}_{i}-\mathbf{J}_{p_{i}}}{\|\mathbf{J}_{i}-\mathbf{J}_{p_{i}}\|_{2}}\;.$

(1)

We then represent a human pose as the directions of all limbs $\mathbf{d}=[\mathbf{d}_{1}^{T},\mathbf{d}_{2}^{T},\cdots,\mathbf{d}_{J}^{T}]^{T}\in\mathbb{R}^{3J}$, where $J$ is the number of limbs, which, in our case, is equal to the number of joints.

As mentioned in the main paper, we choose a simple network with 3 fully-connected layers to model our pose prior. To ensure the invertiblity of the network, we formulate each fully-connected layer as

$\displaystyle\mathbf{f}^{{}^{\prime}}=\sigma(\mathbf{f}\mathbf{Q}\mathbf{R}+\mathbf{b})\;,$

(2)

where $\mathbf{f}^{{}^{\prime}}\in\mathbb{R}^{3J}$ and $\mathbf{f}\in\mathbb{R}^{3J}$ are the feature vectors of the output and input, respectively; $\mathbf{Q}\in\mathbb{R}^{3J\times 3J}$ is an orthogonal matrix; $\mathbf{R}\in\mathbb{R}^{3J\times 3J}$ is an upper triangular matrix with positive diagonal elements; $\mathbf{b}\in\mathbb{R}^{3J}$ is the bias; $\sigma(\cdot)$ is the PReLU function.

In Fig. 1, we visualise additional angles used to defined our kinematics constraints. In Table 1, we provide the valid motion range for most angles used.

Recall that the input to the generator is $\tilde{\mathbf{X}}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{H},\mathbf{x}_{H},\cdots,\mathbf{x}_{H}]$ of length $H+T$. The first $H$ frames are the motion history, and the remaining $T$ frames are replications of last observed frame $\mathbf{x}_{H}$. The goal of the generator is to predict the DCT coefficients of the future motion $\hat{\mathbf{Y}}\in\mathbb{R}^{D\times(H+T)}$ given those of the replicated motion sequence, which we translate to learning motion residuals. Note that, to encourage a smooth transition between past poses and future ones, our generator not only predicts the future motion (corresponding to the last $T$ frames of $\hat{\mathbf{Y}}$), but also recovers the past motion (corresponding to the first $H$ frames of $\hat{\mathbf{Y}}$). Therefore, we define a reconstruction loss on the past motion as

$\displaystyle\mathcal{L}_{past}=\|\hat{\mathbf{Y}}[1:H]-\tilde{\mathbf{X}}[1:H]\|_{2}^{2}\;,$

(3)

where $\hat{\mathbf{Y}}[1:H]$ and $\tilde{\mathbf{X}}[1:H]$ are the first $H$ frames of $\hat{\mathbf{Y}}$ and $\tilde{\mathbf{X}}$, respectively. Furthermore, to enforce the limb length of the past poses to be the same as those of the future poses, we include the limb length loss

$\displaystyle\mathcal{L}_{limb}=\sum_{t=1}^{T}\sum_{i=1}^{J}\|\hat{l}_{t,i}-l_{i}\|_{2}^{2}\;,$

(4)

where $\hat{l}_{t,i}$ is the $i$-th limb length of $t$-th future pose, $l_{i}$ is the ground truth of $i$-th limb length obtained from the history poses and $J$ is the number of limbs. The loss weights of $(\mathcal{L}_{past},\mathcal{L}_{limb})$ for Human3.6M and HumanEva-I are both $(100,500)$.

The detailed architecture of our generator is shown in Fig. 2. It consists of 4 residual blocks. Each block comprises 2 graph convolutional layers. It also contains two additional layers, one at the beginning, to bring the input to feature space, and the other at the end, to decode the feature to the residuals of the DCT coefficients.

In the main paper, we divided a human pose into 2 parts ($N=2$): lower body and upper body, and show the results of predicting motions with same lower body motion but diverse upper body motions. Here, we provide qualitative results with $N=5$. In particular, as shown in Fig. 3, we split a pose into: 1. torso, 2. right leg, 3. left leg, 4. right arm and 5. left arm, following this order for prediction. As shown in Fig. 4, our model is able to perform different levels of controllable motion prediction given the detailed body parts segmentation.

We show a qualitative comparison on the results of our model without either the pose prior loss $\mathcal{L}_{nf}$ or the angle loss $\mathcal{L}_{ang}$ in Fig. 5. This clearly demonstrates the dramatic decrease in pose quality in both cases.