Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction

In recent years, GCN-based networks have been widely utilized for the modeling of spatio-temporal dependencies of the structural time series and made inspiring progress, which provides a means of human motion prediction. Specifically, we represent the skeleton-based pose as a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, in which $\mathcal{V}$ is the joint-node set, $\mathcal{E}$ is edge set. The joint node features are 3D coordinates or joint angles and the edge is related to the adjacency matrix $A\in\mathbb{R}^{N\times N}$.

The state-of-the-art works use the trainable adjacency matrix to replace the constant adjacency matrix, which can model not only skeletal connections but also the implicit dependencies of joints without natural connections, making GCN more powerful for learning spatial dependencies. A single layer with the trainable adjacency matrix can be expressed as:

$$H^{l+1}=f(H^{l};A,W)=\sigma(AH^{l}W^{l})$$

(1)

where $A$, $H^{l}\in\mathbb{R}^{N\times F^{l}}$ and $W^{l}\in\mathbb{R}^{F^{l}\times F^{l+1}}$ are the trainable adjacency matrix, input feature and the trainable transformation matrix, respectively.

Most prior works perform decoupled modeling and direct concatenation for spatio-temporal relationships without considering their cross-dependency, which makes it difficult to accurately depict such complex spatio-temporal relationships. Additionally, since the inter-joint and inter-frame relationships will change with the motion variance and action types, a stable adjacency matrix will lead to inherent poor generalization on multi-action motions. Thus, we propose Spatio-Temporal Gating-Adjacency GCN(GAGCN) to cope with these issues.

Gating Adjacency  As shown in prior works[25, 18, 24, 26, 6, 19, 20, 28, 22], a stable trainable adjacency matrix can handle spatio-temporal dependencies to some extent. However, the relationship will be difficult to depict when it comes to the variants and diverse action types in human motion data. Our ”Enhancing Block”(shown in the right part of Fig. 2) is motivated by the observation that the spatio-temporal relationships are changing with the diverse variances and action types. Therefore, we aim to find the adaptive spatio-temporal relationships to cope with multi-action motion prediction.

The Mixture of Experts(MoE) is a classic machine learning method, which is proven to be able to enhance the generalization of human motion models[38, 29, 30, 31, 21]. The gating network is regarded as a motion classifier to automatically calculates the probability of which motion class the input motion belongs to. The results of the relevant experts are blended to obtain the adaptive output. Therefore, the generalization of the human motion model is largely enhanced. Inspired by MoE, we apply the gating network on GCN to learn the blended adjacency matrix, which is adaptive to the diverse motion variances and action types.

Different from the traditional MoE-based methods which apply weighted sum on all of each expert’s network parameters, we only use the gating network to blend the adjacency matrix. This can make our network lightweight and ensure that only the feature learning process is affected while keeping the feature transferring process unaffected. Specifically, given the features from the previous layer, the gating network in our GAGCN output several blending coefficient parameters, which can be denoted as follows:

$\displaystyle\{\omega^{i}\}$

$\displaystyle=Gating(H)=softmax(FC(H))$

(2)

where $FC$ denotes the 3 fully connected layers, $softmax$ is the activation function, $H$ is the input feature, $\{\omega^{i}\}$ is the set of blending coefficients. Then the blending coefficients are used to blend the candidate trainable adjacency matrices to get the adaptive adjacency matrix:

$$\mathcal{A}=\sum_{i}{A}^{i}\cdot\omega^{i}$$

(3)

where $\{{A}^{i}\}$ is the set of trainable adjacency matrices and $A$ is the adaptive adjacency matrix.

Spatio-Temporal Modeling   Previous literature usually learn spatial and temporal dependencies in a decoupled manner and directly concatenate them, therefore the cross-dependency of the spatio-temporal information is still not fully explored. To address this issue, we propose the balancing and fusion strategy shown in the left dotted box in Fig. 2. The key idea is to adjust the number of candidate adjacency matrices to control and balance the weight between the spatial and temporal modeling, and then fuse spatio-temporal features with the Kronecker product.

Specifically, our balancing strategy is designed as follows: At first, we divide the adjacency matrix $A$ into $A_{s}$ and $A_{t}$, as shown in the left of Fig. 2. Following[35], we treat all channels of a joint as a node instead of regarding each channel of a joint as a node, which can significantly reduce the size of the adjacency matrix and maintain the correlation of different channels of the same node. $A_{s}\in\mathbb{R}^{N\times N}$ represents the inter-dependencies between joint and joint, whether they have skeletal connections or not. Meanwhile, $A_{t}\in\mathbb{R}^{T\times T}$ is trained to learn the frame-to-frame dependencies in the historical sequence.

Then, based on Equ. 2 and Equ. 3, we adopt two gating networks to learn the blending coefficients for spatial and temporal respectively, then blend the candidate adjacency matrices to obtain the adaptive adjacency matrix. We further update these two equations into the following form:

$\displaystyle\{\omega^{i}_{k}\}=Gating_{k}(H),\quad\mathcal{A}_{k}=\sum_{i}^{q}{A_{k}^{i}}\cdot\omega^{i}_{k}$

(4)

where $\mathcal{A}_{k}$ is the adaptive adjacency matrix , and the subscript $k\in\{s,t\}$ indicate ”spatial” $s$ or ”temporal” $t$. It is worth noting that the number of candidate adjacency matrices $q\in\{n,m\}$ can be adjusted, which indicates the complexity of spatial and temporal modeling. The proposed GAGCN can be utilized to balance the weight of spatial and temporal modeling like a scale by adjusting the number of candidate matrices. For example, we use $n=4$, $m=3$ on Human 3.6M and $n=6$, $m=4$ on AMASS.

As for the fusing strategy, a single layer of GAGCN can be formulated as follows:

$$H^{l+1}=\sigma((\mathcal{A}_{s}^{l}\otimes\mathcal{A}_{t}^{l})H^{l}W^{l})$$

(5)

where $H^{l}\in\mathbb{R}^{w^{l}\times N\times T}$, $W^{l}\in\mathbb{R}^{w^{l}\times w^{l+1}}$, $\mathcal{A}_{s}^{l}$ and $\mathcal{A}_{t}^{l}$ are the input feature, trainable transformation matrix, adaptive spatial adjacency matrix and adaptive temporal adjacency matrix of layer $l$, respectively. $\otimes$ denote the Kronecker product. The temporal feature and spatial feature are fused by the Kronecker product to ensure that we can find the hidden cross-dependency of spatio-temporal relationships from the historical motion sequence.

The fused features are fed into the next layer for further learning. Through 6 GAGCN layers, we extract flexible and implicit dependencies between joints and frames represented as the spatio-temporal features. Finally, the features are passed into TCN decoders to predict future sequence, which is confirmed to make better performance and less error accumulation than RNN[1].

Our training process is end-to-end and supervised. With the help of the highly expressive spatio-temporal features extracted by the GAGCN encoder, our network uses a relatively simple loss function to get state-of-the-art results.

For 3D joint coordinates representation, we use MPJPE loss:

$$L_{MPJPE}=\frac{1}{N\cdot t}\sum_{i=1}^{t}\sum_{j=1}^{N}{\|\widetilde{p}_{ij}-p_{ij}\|}_{2}\vspace{-5pt}$$

(6)

where ${p}_{ij}$ represents the predicted 3D coordinates of $j_{th}$ joint in $i_{th}$ frame, and $\widetilde{p}_{ij}$ is the corresponding ground truth.

For the angle-based representation, we use MAE loss:

$$L_{MAE}=\frac{1}{N\cdot t}\sum_{i=1}^{t}\sum_{j=1}^{N}{\mid\widetilde{x}_{ij}-x_{ij}\mid}\vspace{-5pt}$$

(7)

where ${x}_{ij}$ represents the predicted joint angle in exponential map of $j_{th}$ joint in $i_{th}$ frame, and $\widetilde{x}_{ij}$ is the corresponding ground truth.

The datasets used in our experiments include Human 3.6M [12], AMASS[23], and 3DPW[33]. We will introduce these 3 datasets as follows:

Human 3.6M Human 3.6M is the most used dataset in the field of motion prediction. Human 3.6M has 3.6 million 3D poses, consisting of 15 motion categories from 7 subjects. We down-sample the frame rate to 25Hz. Following[27, 24], we use subjects 1,6,7,8,9 for training, subject 11 for validation and subject 5 for testing.

AMASS The Archive of Motion Capture as Surface Shapes(AMASS) dataset is a recently published human motion dataset, which gathers 18 existing mocap datasets, such as CMU, KIT, and BMLrub. We down-sample the frame rate to 25Hz as for Human 3.6M. Then, following[24], we select 8 datasets from AMASS for training, 4 datasets for validation and 1 dataset(BMLrub) for testing.

3DPW The 3D Pose in the Wild dataset consists of both indoor and outdoor actions, which contains 51,000 frames captured at 30Hz. We down-sample the frame rate to 25Hz as for Human 3.6M. We only use 3DPW to test the generalization of the models trained on AMASS.

Metrics and Baselines Our model can be trained on both 3D coordinates representation and angle-based representation. Thus, we evaluate the results on both 3D coordinates errors and angle errors. We adopt the MPJPE metrics for 3D coordinates representation and MAE angle error metrics for angle-based representation¹¹1It is worth noting that the evaluation metric we used is following STSGCN[28], that is, the average error over frames (denoted by *), which we only found when we checked their test code on July 9, 2022.. We compare our approach with Res-GRU[27], ConSeq2Seq[17], LTD-10-25[25], HRI[24], and STSGCN[28] on Human 3.6M and LTD-10-25[25], HRI[24], and STSGCN[28] on AMASS and 3DPW. We adapt the code and the pre-trained models released by the authors to evaluate their results. Note that HRI[24] takes the past 50 frames as input to predict the future 25 frames while others takes the past 10 frames as input to predict the future 25 frames.

Human 3.6M   Because of the ambiguity in the angle-based representation, most recent works use 3D coordinates to measure the accuracy of motion prediction, i.e. MPJPE. As in previous work, we predict future motion for 25 frames(1000ms) based on a 10(400ms) frames historical motion sequence. We select 14 action types from Human 3.6M and randomly select 8 sequences for each motion to calculate the average error. As in Table. 1, we show the comparison of the short-term and long-term prediction of our model and the baselines on Human 3.6M.

Our method outperforms STSGCN over almost all time horizons. In particular, thanks to the adaptive adjacency matrix, our model makes larger improvements in action types that are difficult to predict, such as ”Posing” and ”Sitting down”. Moreover, our method has significant advantages in long-term (1000ms) motion prediction. Nonetheless, there are very few time points where our method does not perform best. These time points are in short term with small prediction errors for all methods, thus it is reasonable to have a marginal error. The bottom right of Table. 1 is the average error of all action types, where our method performs better than all the comparative methods for whole time horizons.

Additionally, we demonstrate the average angle errors on Human 3.6M in Table. 2 with the same setting for MPJPE metrics. The results illustrate that our method also outperforms STSGCN in angle-based representation.

AMASS & 3DPW   We demonstrate the short-term and long-term prediction results on AMASS-BMLrub in Table. 3. We train the model on 8 datasets from AMASS and use BMLrub for testing. AMASS has much more subjects and motion sequences than Human 3.6M, which is more suitable to test the generalization of the model. Our method outperforms STSGCN on AMASS, which proves that our model can indeed enhance the generalization of GCN.

The model trained on AMASS is further tested on 3DPW, and the results are shown in Table. 4. The significantly better results compared with other methods provide another strong evidence of our model’s generalization across different datasets.

We perform ablation studies to evaluate the effect of two key component in our method, i.e. enhancing block, balancing block. The effect of fusion block can be found in the supplementary material.

Effect of Enhancing Block   We have shown the generalization of our method across different datasets in 4.2, and then we will further show the generalization of our method across different action types. The results are shown in Table. 5. We explore the generalization of our model by testing on unseen action types(Walking Together). The results in the second row of the table are significantly better than the first row, indicating that GAGCN helps to predict unseen action types. And the results in the second and third rows are very close, indicating that GAGCN performs accurate predictions on unseen action types.

Effect of Balancing Block   To demonstrate the effect of balancing block, we set up three contrast experiments against our method(shown in Table. 6).: 1. Contrast experiment 1 illustrates the necessity of using gating network both spatially and temporally. The better results of our method indicate that applying gating network in time and space simultaneously helps to model spatio-temporal dependencies more effectively. 2. Contrast experiment 2 shows that more candidate matrices are not always better. We empirically classify the motions in human 3.6M into roughly four categories of similar motions, thus four spatial candidate matrices are used. Too many candidate matrices will increase the complexity of the network and cause under-fitting. 3. Contrast experiment 3 says that the weight of spatio-temporal modeling affects the accuracy of the prediction results indeed, and that is why we balance them by adjusting the number of candidate matrices.

Visualization of Predicted Sequence  We visualize the predicted sequence on Human 3.6M and compare them with Ground Truth in Fig. 3. For periodic motions such as ”Walking” and ”Walking Together”, our predictions are almost identical to Ground Truth over the entire time horizons. Meanwhile, for more complex non-periodic motions like ”Discussions” and ”Posing”, our predictions match the Ground Truth well in the short term, and the long term predictions are also quite close to GT despite some acceptable errors on the left leg of ”Discussions” and the right arm of ”Posing”. Non-periodic motion prediction is a more challenging problem, especially when the subjects in the testing dataset perform in different ways compared with subjects in the training dataset. The visualization of predicted sequences on AMASS is shown in the supplementary material.

Visualization of Spatial Blending Coefficients  Moreover, we randomly select 16 sequences from a single action type to compute the average spatial blending coefficients(visualization of temporal blending coefficients can be found in supplementary material). Then we do the same operation on several action types and visualize them(seeing Fig. 4). We can see that there is a clear difference in the blending coefficients distribution for different action types. The blending coefficients for ”Walking Together” are derived from the partial train model in 4.3. Since ”Walking Together” and ”Walking” are similar periodic motions, their blending coefficients are similarly distributed with higher $\omega 3$ and $\omega 4$ values. That is why our model can achieve accurate prediction results for ”Walking Together” without seeing it before. Non-periodic motions like ”Discussion” and ”Sitting Down” have higher $\omega 2$ values, but their coefficient distributions are very different. Given different inputs, GAGCN can generate the corresponding blending coefficients, which helps to learn the adaptive adjacency matrix for diverse action types. The visualization of adaptive adjacency matrix can be found in the supplementary material.