DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Prediction

By observing the body pose given by the 3D coordinates of $V$ human joints for $T$ frames, the model needs to predict the pose for $K$ future frames.

Supposed that an observed pose sample can be formulated as $\textbf{X}_{1:T}=\{\textbf{x}_{1},\textbf{x}_{2},\dots,\textbf{x}_{t},\dots,\textbf{x}_{T}\}$ with $1\leqslant t\leqslant T$ where a frame $\textbf{x}_{t}\in\mathbb{R}^{N}$ at time $t$ denotes the $N$-dimensional body pose. $N=V\times C$, where $N$ is the number of joints in the skeleton representation unique in different scales (we use 22, 10, and 5 joints for our three scales in this paper) and $C$ the dimension of each joint. Following the previous works, each joint of the 3D skeleton is represented as 3D coordinates, thus we have $C=3$. Then, we denote ${\hat{\textbf{Y}}}_{T+1:T+K}=\{{\hat{\textbf{y}}}_{T+1},{\hat{\textbf{y}}}_{T+2},\dots,{\hat{\textbf{y}}}_{T+K}\}$ be the prediction frames, and ${\textbf{Y}}_{T+1:T+K}=\{{\textbf{y}}_{T+1},{\textbf{y}}_{T+2},\dots,{\textbf{y}}_{T+K}\}$ be the corresponding ground truth.

For the convenience of calculation, the input motion history data $\mathbf{I}$ was formulated as a $N\times C\times T\times V$ shaped tensor. For spatial graph convolution, we define a graph $\mathcal{G}_{s}=(\mathcal{V}_{s},\mathcal{E}_{s})$ with $V$ nodes $i\in\mathcal{V}_{s}$. Edges $(i,j)\in\mathcal{E}_{s}$ are represented by a masked spatial adjacency $\mathbf{A}_{s}\in\mathbb{R}^{V\times V}$. Similarly for temporal graph convolution, we define another graph $\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t})$ with $T$ nodes $t\in\mathcal{V}_{t}$. Temporal edges $(t,u)\in\mathcal{E}_{t}$ are represented by a temporal adjacency $\mathbf{A}_{t}\in\mathbb{R}^{V\times V}$.

The human pose can be abstract progressively to obtain a set of scaled poses. Intuitively, two ideas motivate us to propose a multiscale architecture to leverage the information from different scales. First, it is much easier to make predictions on a coarse scale than on the fine one. What’s more, the coarse one restricts the modeling of the fine-scale trajectories within local regions, which facilitates the stableness of predicted poses.

As shown in Figure 1, the original scale called “Joint Scale” has twenty-two joints. We abstract the original scale recursively to obtain two scales, the “Bone Scale” with ten joints and the “Part Scale” with five joints. The downsampling methods drawn between scales depict how to combine the joints in the fine-level scale. We take an average 3D coordinate as the place of the new joints. Note that the grouping method we use is the most stable one obtained after several trials.

Dynamic fusing process. After information extraction by GCNs, we obtain features at three levels. It is crucial to fuse the lower level features to a higher level to facilitate the abundance of information for the decoder input. We tried some manually designed ways that do not perform well, mainly because their rigid structures hinder the convergence of the model where the parameters are updated simultaneously. Then we discover that only a Linear layer seems to work well. According to the results, it is plausible to fine tune the fusing way dynamically by networks. The fusing process can be expressed by the following formula:

where $X_{i}$ represents the features in the $i$th level in Figure 2 from top to bottom, $W_{ij}$ the weight matrix upsampling features from $i$th level to $j$th level, $\alpha$ the fusion coefficient. The superscript “+” stands for fused features.

We use GCNs to extract the spatial information of different scales. Intuitively, relying on a predefined adjacent matrix Yan et al. (2018) according to the human kinematic chains neglects the intensity of information interaction between joints. The method Mao et al. (2019) regards the adjacency matrix as a parameter to train the connection strength has too many redundant connections, which lead the network hard to convergence. To overcome those drawbacks, we propose a semi-autonomous-learned GCN network.

Background on GCNs. As described above, we assume that the human body pose as a graph $\mathcal{G}_{s}=(\mathcal{V}_{s},\mathcal{E}_{s})$ with $V$ nodes $i\in\mathcal{V}_{s}$. Edges $(i,j)\in\mathcal{E}_{s}$ are represented by an adjacency $\mathbf{A}_{s}\in\mathbb{R}^{V\times V}$. Then, a pose sequence with $T$ frames is formulated as $\mathcal{M}_{s}=\{\mathcal{G}_{1},\mathcal{G}_{2},\dots,\mathcal{G}_{T}\}$. Take one frame as an example, a graph convolutional layer $p$ then takes as input a matrix $\mathbf{H}_{s}^{(p)}\in\mathbb{R}^{V\times C^{(p)}}$ with $C^{(p)}$ the graph node dimensions of layer $p$. Given this information and a set of trainable weights $\mathbf{W}_{s}^{(p)}\in\mathbb{R}^{C^{(p)}\times C^{(p+1)}}$, we describe a graph convolutional layer as the form

where $\sigma$ is an activation function, such as ReLU or PReLU.

Our improvements. Instead of relying on a fully connective adjacency as in Mao et al. (2019), we decide to restrict the joint within several hops. So we design a fixed mask matrix $\mathbf{M}\in\mathbb{R}^{V\times V}$ to block the connection of some joint pairs with long distances. Besides, we also design a Time-Sharing Weight Table $\mathbf{T}_{s}^{(p)}$ which contains the learned spatio feature to embed the joints. Then the general formula of a GCN layer becomes

where $\mathbf{M}$ changes according to the pose graph, i.e., each scale we use has its mask, as shown in Figure 3. The symbol $\circ$ indicates an element-wise product. Masks prevent the connection of some joints and reduce the updatable parameters of the adjacency matrix, making the adjacency matrix semi-autonomous learned.

Similar to the basic Spatio GCN, we define temporal edges that connect joints across the frame. Then we obtain a temporal graph $\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t})$ with $T$ nodes $t\in\mathcal{V}_{t}$. Temporal edges $(t,u)\in\mathcal{E}_{t}$ are represented by a temporal adjacency $\mathbf{A}_{t}\in\mathbb{R}^{V\times V}$. This time, we only add a Space-Sharing Weight Table $\mathbf{T}_{t}^{(p)}$ contains the learned temporal features:

where $\mathbf{H}_{t}^{(p)}\in\mathbb{R}^{T\times C^{(p)}}$ is the input matrix of layer $p$, and $\mathbf{W}_{t}^{(p)}\in\mathbb{R}^{C^{(p)}\times C^{(p+1)}}$ is the trainable weight matrix of layer $p$.

Once we get the body dynamics features from the encoder, the estimation of 3D coordinates of body joints in the future falls to the decoder that applies to the temporal dimension. The decoder takes the observed frames as the input and the body dynamics features as the rule to generate plausible future poses.

To tackle the training difficulty and mitigate the error accumulation of traditional RNN Fragkiadaki et al. (2015); Jain et al. (2016); Martinez et al. (2017); Pavllo et al. (2020), we adopt Temporal Convolution Networks (TCN) Sofianos et al. (2021) as our decoder.

Human 3.6M. Following previous work, we use the Human3.6M dataset (H3.6M) Ionescu et al. (2013), which is widely used in training for human motion prediction models. H3.6M includes 15 identical actions such as walking, eating, sitting, and phoning, which are performed by 7 actors. For a fair comparison, we adopted the representation with 3D coordinates from Mao et al. (2019); Dang et al. (2021), where each pose contains 32 joints. We report the same average error as Mao et al. (2019); Dang et al. (2021) do on all test sequences.

CMU motion capture. To prove the universality of our model, we also report results on the CMU motion capture dataset (CMU Mocap). CMU Mocap has captured different actions with more joints in comparison with H3.6M. We apply the same preprocessing procedure as on H3.6M. For a fair comparison, we adopted the same data representation, training/test splits, and evaluation sequences as in Mao et al. (2019).

Model configuration. We implement our model with Pytorch 1.9.0 on one NVIDIA Quadro RTX 6000 GPU. In the encoder, we use 7 STGCN Blocks, each of them consisting of a proposed Masked SGCN layer and a TGCN layer, with a dropout rate of 0.1. Besides, we use residual layers between STGCN blocks to transfer information between them, as shown in Figure 2. Then, we use 4 TCN layers to generate the future frames. In training, we set batch size 32; we use an ADAM optimizer and a scheduler to half the learning rate about every ten epochs.

Loss function. To train our model, we consider the $\ell_{1}$ loss. Let the $v$th joints at time $t$ of the prediction sequence be $\hat{\textbf{y}}_{t,v}$ and the corresponding ground truth be ${\textbf{y}}_{t,v}$. The loss function is:

where ${\|\cdot\|}_{1}$ denotes the $\ell_{1}$ norm. $\ell_{1}$ loss gives significant gradients to joints with small losses and stable gradients to joints with large losses Li et al. (2020). We found that $\ell_{1}$ loss leads to more precise predictions than $\ell_{2}$ loss does in our experiment.

Evaluation Criteria. Following the standard evaluation metric in Mao et al. (2019); Dang et al. (2021), we report the results in Mean Per Joint Position Error (MPJPE) in milimeter, which is the most widely used evaluation metric:

where $\hat{\textbf{y}}_{t,v}\in\mathbb{R}^{3}$ is the $v$th joint position at time $t$, and $\textbf{y}_{t,v}$ is the corresponding ground truth. Instead of averaging the MPJPE loss along the time dimension, we also obtain the MPJPE loss at each time by calculating the MPJPE loss on every single frame.

Following previous work, we evaluate our model in 1000ms (i.e., 25 frames) high-precision long-term motion prediction. We compare our model with some competitive models Martinez et al. (2017); Mao et al. (2019); Dang et al. (2021) in qualitative and quantitive ways. It is worth mentioning that most works evaluate their methods on only eight sequences for each action category. We argue that such little test data is not enough to represent the performance of the compared approaches. Dang et al. (2021) also questions on the same point. So we modified their published codes to use the whole test set.

Qualitative results. We first visualized the results of motion prediction sequences generated by different models (Traj-GCN Mao et al. (2019), MSR-GCN Dang et al. (2021), and ours) to make a qualitative comparison, as shown in Figure 4. Prediction sequences generated by other models show a visible difference compared with the ground truth. In this figure, the action we need to predict is to raise the right arm over the head and lower it a little. The MSR-GCN model does not foresee that the arm will be higher than the head, and the Traj-GCN outputs almost unmoved pose sequences. Though there still has a gap with the ground truth, our model (DMS-GCN) predicts the most reliable results.

Quantitative results. We also further evaluate the MPJPE error in 3D coordinates between the ground truth and some models (Residual supMartinez et al. (2017), Traj-GCN Mao et al. (2019), MSR-GCN Dang et al. (2021), and ours). The quantitative comparisons for both short-term and long-term prediction results are presented in Table 1 and Table 2 respectively. GCN-based approaches are better than the RNN-based method Residual sup. Martinez et al. (2017), which validates the effectiveness of GCNs for human motion prediction. Among the methods based on GCNs, our method is superior to others, especially short-term prediction. As for the long-term, though the improvement is not obvious as the short-term, we still have the best results.

Computational efficiency contrast. The total number of training parameters as well as time cost per epoch are shown in Table 3. All experiments were performed using PyTorch on an NVIDIA Quadro RTX 6000 GPU. Our model (DMS-GCN) requires fewer parameters than existing methods, and its computation speed is significantly faster. We have the best prediction results while consuming the least computational resources compared to the two current best models, which shows the superiority of our model in all respects.

Similar to the experiments above, we investigate the same method on the CMU Mocap dataset to prove the generalization capability of our model, as shown in Table 4. The experimental results show that our approach still achieves a significant improvement in the short-term prediction. To be honest, our model in the long-term prediction has not much advantage as usual and even exceeded by Traj-GCN slightly at the final frame. We argue two reasons: first, when predicting long-term poses, our model shares the same parameters with the short-term model, which causes the error accumulation; second, by visualizing the poses predicted by Traj-GCN, we see they tend to be a mean pose. Traj-GCN engages with minimizing the loss but neglects to capture the movement trend. In brief, our model still shows the most reliable results, which proves that our model does not depend on specific actions or a specific number of human joints, i.e., our method is universal in the field of human motion prediction.

Effects of multi-scale architecture. To study the effectiveness of the multi-scale mechanism, we conduct experiments on a different number of scales. The comparison results are shown in Table 5. Please see the rows corresponding to DMS-GCN-1L, DMS-GCN-2L, and DMS-GCN. The experiment with the whole three scales achieves a total victory. These experiments demonstrate the effectiveness of our multi-scale architecture.

Effects of the semi-autonomous-learned GCNs. To verify the effectiveness of the semi-autonomous-learned GCNs of the proposed architecture, we remove the mask for our model and find the results get worse at every sequence, as contrasted in the two rows corresponding to DMS-GCN and DMS-GCN w/o Mask of Table 5. This experiment illustrates the effectiveness of our semi-autonomous-learned GCNs.

Effects of temporal GCN blocks. We remove all TGCN blocks to analyze the effects of the TGCN blocks. Please see the rows corresponding to DMS-GCN and DMS-GCN w/o TGCN of Table 5. The experimental results show that TGCN blocks improve the prediction by a large margin, which strongly validates the importance of TGCN blocks.