Existence Is Chaos: Enhancing 3D Human Motion
Prediction with Uncertainty Consideration

Recently, deep learning networks became the mainstream of motion prediction. Since motion prediction was often seen as a sequence-to-sequence task, RNN-based methods were naturally used for this task. For example, LSTM-3LR (Jain et al. 2016) and ERD (Fragkiadaki et al. 2015) introduced LSTM into the task. ERD added encoder and decoder before and after LSTM to achieve better results. LSTM-3LR replaced bone information with speed information. However, RNNs suffer from error accumulation and were unable to effectively model spatial information. Then feed-forward neural networks were applied to the field. They tried to use convolution kernels to extract information in space and time. TE (Butepage et al. 2017) and QuaterNet (Pavllo, Grangier, and Auli 2018) used CNNs to capture temporal information. convSeq2Seq (Li et al. 2018) and CHA (Li et al. 2019) designed hierarchical CNNs to model spatial and temporal information simultaneously. The TrajectoryCNN (Liu et al. 2020) proposed the features of trajectory space that were more easily handled by CNN. The siMLPe (Guo et al. 2023) proposed to use simple multi-layer MLPs to extract temporal and spatial information and achieves excellent results. Some probabilistic prediction works (Yuan and Kitani 2020; Mao, Liu, and Salzmann 2022) predicted multiple possible motion sequences. However, due to the different evaluation metrics, our method does not compare with these works.

Recently, GCNs achieved the state-of-art results in motion prediction task. LTD (Mao et al. 2019) saw pose graph as a fully-connected graph and modeled temporal information by DCT representations which were followed by lots of later work. JDM (Su et al. 2021) and MPT (Liu et al. 2021) proposed GCNs that took velocity information as input. In the graph representation, DMGNN (Li et al. 2020) and MSR-GCN (Dang et al. 2021) used muti-scale graph and extracted information from single scales and cross scales. STSGCN (Sofianos et al. 2021) and GAGCN (Zhong et al. 2022) built space-time separable GCN to extract information in graph. SPGSN (Li et al. 2022) did scattering decomposition of the graph and GCNs were performed on all graphs. In the learning strategy, PGBIG (Ma et al. 2022) performed muti-stage prediction whose output corresponded to a smooth sequence in every stage. UA-HMP (Ding and Yin 2021) integrated probabilistic prediction into deterministic methods. AuxFormer (Xu et al. 2023) introduced a model learning framework with auxiliary task which was recovering corrupted coordinates depending on the rest coordinates. DMAG (Gao et al. 2023) used frequency decomposition and feature aggregation respectively to encode the information. Meanwhile, Transformer was also attempted for human motion prediction (Mao, Liu, and Salzmann 2020; Cai et al. 2020; Aksan et al. 2021). Self-attention mechanism of Transformer had strong modeling ability on sequence data and was able to model the dependencies of joints. In our model, we synthesize the modeling ability of self-attention mechanism and the effectiveness of graph neural networks.

Firstly, some notations and variables in this paper would be enumerate. We denote $X_{obs}=[x_{1},x_{2},\cdot,x_{T_{in}}]$ as the past pose sequences and $X_{pre}=[x_{T_{in}+1},x_{T_{in}+2},\cdot,x_{T_{in}+T_{out}}]$ as the future pose sequences, where $x_{i}\in R^{N\times d}$ with N joints and d-dimension space (d is 3) is represented as human pose at time i. The goal of motion prediction is to learn a mapping function that maps $X_{obs}$ to $X_{pre}$.

The overview of proposed model is showed in Fig. 3. An encoder-decoder structure is adopted to conduct end-to-end prediction. First, an encoder with proposed Self-attention Graph Blocks (SAGGBs) and Temporal Convolutional Network modules (TCNs) would encode the input into high-dimensional feature spaces. Then a lightweight decoder makes predictions for future pose sequence estimation.

The encoder contains a series of residual blocks. Each block consists of SAGGB and TCN which respectively extract the spatial and temporal information. After feature extraction through all blocks, the encoder would output a feature map of $R^{T_{in}\times V\times C_{f}}$, where $C_{f}=128$ is used in this work. The decoder is a lightweight design that includes 4 CNNs and a MLP. The CNN module uses time dimension as feature channel to extract information. The first CNN projects the input from $R^{T_{in}\times V\times C_{f}}$ to $R^{T_{out}\times V\times C_{f}}$. The MLP projects the $C_{f}$ to 3.

For GCNs, either natural connection or full connection applies the same graph weights among all samples, where the the diversity of actions and the complexity of behaviors are ignored. For instance, for action walking, the dependence between feet should be more of a concern but for action eating the dependence is not so important. Such dependency reflects the variation of uncertainty with motion diversity. Hence, We propose a Self-attention Graph Generate Block (SAGGB) module to generate data-driven graph to model such information. Let’s recall $x_{i}\in R^{n\times d}$ as the human pose at time $i$. We apply self-attention mechanism to generate graph $A_{i}$ for every pose

where $\sigma(.)$ and $\phi(.)$ are mapping function to generate query and key for every joint; $d_{k}$ is the dimension of query and key. For each pose, SAGGBs generate attention map about the joints as the graph to this pose. As can be noticed, SAGGBs generate different graphs with different weights for different poses.

However, the same pose in different pose sequences could have different means. Consequently, it’s more sensible to generate special graph for every pose sequence rather than pose. We denote the graph $A_{sample}$ for a given sample as below

By constructing a new graph for each sequence, the model is able to extract implicit higher-dimensional information from different input poses.Based on the special graph for each sample, the output $X^{l+1}$ of SAGGB block with given input $X^{l}$ can be defined as:

To train the model for human motion prediction, $L_{2}$ loss is usually used in existing approaches:

As mentioned before, such averaging form treat all timestamps equally, where problem arises in two aspects. First, the prediction of the later frames could be difficult due to the uncertainty of the action semantics. However, the loss with averaging form gives more weight to the later frames with greater uncertainty and less structured knowledge, which makes the model fit more noise and uncertainty. Second, human motion is a continuous process, thus the distribution of one frame is heavily dependent on the state of the previous frame. This dependency is passed frame by frame. Therefore, the prediction of the first frame in the prediction sequence is crucial. The first frame in prediction determines the initial position of the model from the deterministic space to the uncertain space. Meanwhile, the dependency relationship shows that the loss of the first frame prediction largely determines the expression of the entire model.

Based on the above two issues, we propose an Adaptive-Salient Loss in this work.

Human3.6M dataset (H3.6M) is a widely used motion prediction dataset consists of 15 actions performed by 7 actors(S1,S5,S6,S7,S8,S9 and S11). The human body is represented as 32 joints. Follow (Li et al. 2022; Ma et al. 2022), global information including global rotation and movement is removed. Meanwhile, all samples are down-sampled to 25 frames per second and we test our model on S5.

Follow  (Ma et al. 2022), Mean Per Joint Position Error (MPJPE) is employed as criterion which calculates the average $L_{2}$ distance at each timestamp.

The proposed method is compared with DMGNN (Li et al. 2020), LTD (Mao et al. 2019), SPGSN (Li et al. 2022) and PGBIG (Ma et al. 2022). All methods are tested under the same conditions. Meanwhile, the proposed method is also compared with STSGCN (Sofianos et al. 2021) and GAGCN (Zhong et al. 2022) in H3.6M under their evaluation criteria. Inspired by  (Du et al. 2023), we further employ Jitter metric to evaluate the quality of predicated motion sequences.

where $\Delta x_{t}$ is the Euclidean distance between prediction pose and future pose at frame $t$, $T$ is the total frames and $\mu$ is the frame per second.

To show the performance of our model, we list the quantitative results for both short-term prediction(400ms) and long-term prediction(1000ms) on H3.6M, CMU Mocap and 3DPW.

Statistical analysis on prediction quality is shown in Tab. 7 using Jitter metric (Du et al. 2023). To be specific, the sequences predicted by our method performed better in realism and stable with the increase of prediction frames. Visualization of Joint Trajectory is shown in Fig. 4. The trajectory of a single joint point projected on the x-y plane of a prediction sequence is shown. More turns appear at the end of the PGIBIG’s and STSGCN’s predictions to mimic the real joint motion. Comparing to baselines, ours predicted results are more reasonable and smooth. Further details in prediction result is shown in the supplementary video.

The ablation study results for hyper parameters is shown in Fig. 5. These results confirm the validity of our proposed loss and also support our hypothesis. The salient factor $\omega$ and weight factor $\lambda$ is determined as $10$ and $0.3$, respectively. In addition, we also investigate the validity of our model structure. We have conducted the ablation study of proposed SAGGB and investigate the block number for encoder, where the result is shown on Tab. 8.

Our method achieves high quality and realism motion prediction. However, there are still two limitations: 1. The popular $L_{2}$ Loss, e.g. MPJPE may be not sufficient enough to fully express the quality of the prediction and be an indicator for a prediction model. Especially long-term prediction results with similar MPJPE could perform quite different on motion stable and realism. Further evaluation indicators could be conducted to improve model training and prediction quality evaluation. 2. Current benchmark datasets are originally created for either pose estimation or action recognition purpose. Hence, some key characteristics of motion prediction are not considered well, e.g. various possible action modes from various subjects. In the future, how to construct a dataset that is more suitable for human action prediction is worthy of future research.