Motion Prediction via Joint Dependency Modeling in Phase Space

Previously, statistical approaches broadly fall within the scope of latent variable models, which leverage Gaussian processes (Wang et al., 2005) and hidden Markov models (Lehrmann et al., 2014) to capture the temporal dynamic of human motions. Recently, the success of deep learning methods in various fields (Zhu et al., 2020; Fuli et al., 2019; Xun et al., 2018; Jianfeng et al., 2021; Liu et al., 2017; Yinwei et al., 2019; An-An et al., 2021; Liu et al., 2021a) lead to diverse neural network designs for modeling motion contexts (Liu et al., 2021d; Li et al., 2019; Gao et al., 2019; Chen et al., 2020). The general framework encompasses a sequence-to-sequence model, where the observed pose sequence is encoded to a latent motion context that is decoded to output future sequence. A major line of work utilizes RNN modules such as Long Short Term Memories (LSTM) (Fragkiadaki et al., 2015) or Gated Recurrent Units (GRU) (Martinez et al., 2017) as the encoder and decoder. Other options such as convolutional networks (CNN), hierarchical representation, and graph convolutional networks have also been explored.  (Li et al., 2018) proposes a CNN based model to capture long-term temporal and spatial correlations. (Liu et al., 2019) represents human skeleton with a Lie algebra representation to encode anatomical constraints, and employs a hierarchical recurrent network to encode local and global contexts. (Mao et al., 2019) designs a generic graph to encode the spatial dependencies of human pose and a simple feed-forward deep network to capture dynamic context. (Cui et al., 2020) builds a deep generative model based on a novel dynamic GNN and adversarial learning.  (Li et al., 2020) introduces a multiscale graph computational unit to extract and fuse features for motion feature learning. (Mao et al., 2020) develops an attention-based model that explicitly leverages historical information for motion prediction. In this paper, we present an alternative network, which improves the modeling of spatio-temporal contexts along with the capability of capturing both explicit and implicit inter-joint dependencies.

A fundamental component for the motion prediction task is to represent the pose sequence in the most suitable and effective way so as to facilitate modeling of motion contexts. Most existing works (Fragkiadaki et al., 2015; Martinez et al., 2017; Li et al., 2018; Liu et al., 2019) represent the pose as joints along a kinematic graph with joint orientations parameterized as axis angles. A related approach characterizes the skeletal structure as a graph (Li et al., 2020; Cui et al., 2020; Mao et al., 2019, 2020; Guo and Choi, 2019). A critical issue is that these schemes usually treat the joints on an equal standing, failing to account for the fact that the kinematic chain is a hierarchical structure. This raises severe difficulties in effectively capturing the joint dependencies, manifesting as large prediction errors for end effectors such as hands and foot. We therefore propose to discard with kinematic graphs, treat each joint as distinct entities, and directly learn the correlations and dependencies via data, facilitated by additional knowledge of skeletal anatomy.

First, we move away from the widely adopted kinematic graph scheme and instead focus on each joint rather than the entire pose. In particular, a joint $j$ is characterized by its trajectory $T_{j}=[D^{j}_{0},D^{j}_{1},\cdots,D^{j}_{N}]$ where $D^{j}_{i}\in\mathbb{R}^{3}$ denotes the position vector of joint $j$ in the $i^{th}$ frame (time). Furthermore, we augment a joint trajectory with joint instantaneous displacements by computing the joint displacements between adjacent frames, thereby obtaining phase space trajectory for each joint. Specifically, joint $j$ at frame $i$ is represented as a tuple $(D^{j}_{i},\omega^{j}_{i})$ where $D^{j}_{i}$ characterizes its position while $\omega^{j}_{i}\in\mathbb{R}^{3}$ captures the displacement at the $i^{th}$ frame. Displacement $\omega^{j}_{i}$ can be conveniently computed by:

Mathematically, the phase space is the cotangent bundle of the pose configuration space and by explicitly including the displacement we achieve a complete characterization of the dynamics of the system at any given frame (time) (Craig, 2009). The displacement information also provides valuable motion contexts for future prediction. (Martinez et al., 2017) models angular velocities to solve the problem of the discontinuities between the observed sequence and the first-frame prediction. (Mao et al., 2019) tries to encode temporal information in the frequency space via the discrete cosine transform. Our proposed method instead uses the direct joint instantaneous displacements to explicitly model trajectory features, therefore, the network does not have to learn to extract the implicit motion dynamics of the joints.

An overview of the proposed TEID network is shown in Fig. 2. An important highlight of our approach consists in rejecting the widely adopted kinematic graph based pose representation which inherently invokes an unnecessary complexity burden due to the hierarchical nature of the joints representation. Instead, we examine the phase space trajectories at individual joint levels. Taking phase space trajectory representations as inputs, TEID comprises an explicit modeling component as well as an implicit modeling module to capture explicit and implicit dependencies between joint trajectories, respectively. This allows extrapolation and connection of the individual joint trajectories to form a single coherent pose sequence.

More specifically, TEID first models the known and explict dependencies between joints. Following (Liu et al., 2021b), three kinds of explicit joint dependencies are encoded: 1) The natural skeletal connection between joints. 2) The correlation between symmetric arms and legs, e.g., the left arm and the right arm. 3) The synchronization tendency between an arm and a leg in opposite sides, e.g., the left arm and the right leg. Then, TEID considers the hidden and implicit denpendencies between joints and the naturalness of the entire pose. The implicit dependency modeling encompasses a global optimization module that examines the implicit relationship between the predicted joint trajectory extrapolations and the entire predicted motion sequence to ensure consistency and naturalness of the final output.

Explicit Dependency Modeling  Effectively capturing the precise spatial relationships between closely related joints is indispensable to understanding and modeling motion. Existing methods have expended significant efforts in this direction such as considering kinematic trees (Liu et al., 2019) or graphs (Li et al., 2020; Cui et al., 2020; Mao et al., 2020) and examining the problem in the frequency domain via discrete Fourier transforms (Mao et al., 2019). Yet, the fixed graph or kinematic tree structures in such methods make it difficult to incorporate prior knowledge, while using frequency domain usually involves some intricate or cumbersome network designs to extract joint relationships.

In our approach, prior anatomical knowledge of motion can be readily leveraged. Technically, explicit dependencies that can be tapped into include bone connections, symmetrical properties of the arms and legs, as well as more complicated examples such as coordination of different limbs in opposite sides. As depicted in Fig. 2, the phase trajectory inputs for a joint $j$ along with all its explicitly related joints are collectively fed into a spatio-temporal module. Specifically, this convolutional module consists of three parallel layers of deformable convolutions with different dilation rates, which allow simultaneous extraction of multi-scale spatial features from the phase space trajectories. Various dilation rates $d\in\{1,2,3\}$ are engaged to facilitate multi-scale processing of the dynamic information. For each joint, the consideration of only its explicitly (closely) related joints rather than all the joints is beneficial in smoothing out noises and irrelevant interferences. Finally, we obtain for joint $j$ a feature of size $m\times C(j)\times N$ where $m$ is the number of channels for the convolution operation, $C(j)$ is the number of explicitly related joints pertaining to joint $j$, and $N+1$ is the length of historical sequence (which yields $N$ displacements).

The obtained features are then fed into a GRU network to generate principled trajectory predictions for future frames $N+1:N+n$.

Implicit Dependency Modeling  We further refine the principled trajectory prediction for each joint by engaging in implicit dependency modeling, which considers the relationships between predicted trajectories of all joints. After the previous step, the predictions of different joint trajectories are independent to each other, the dependencies between a single joint and the entire pose sequence are also ignored. In order to ensure that the predicted motion sequence is harmonious and reasonable from a global perspective, we introduce a global refinement module which can optimize the predicted joint features with respect to the entire predicted pose sequence.

As illustrated in Fig. 2, we feed the forecasted displacement vector sequences $\tilde{\omega}\in\mathbb{R}{{}^{3\times n\times J}}$ into two $1\times 1$ convolution modules for generating two feature matrices $\gamma$ and $\varphi$, respectively. Then, we reshape them to $\mathbb{R}{{}^{3\times K}}$, where $K=n\times J$ is the total number of joint positions in $n$ future frames. Formally,

Next, we apply matrix multiplication on the transpose of $\gamma$ and $\varphi$, and utilize a softmax operation to obtain a global affinity matrix $A\in\mathbb{R}^{K\times K}$:

The affinity matrix $A$ models the correlation between any two joints in the predicted $n$ future frames. Within the affinity matrix, element $A^{j,j^{\prime}}_{i,i^{\prime}}$ measures the influence of $j^{\prime th}$ joint in the $i^{\prime th}$ future frame on the $j^{th}$ joint in the $i^{th}$ future frame. More precisely, the element of the matrix measures the affinity between any two joint displacement features even when the two joints are in different frames. Finally, we refine the predicted displacement vector feature of $j^{th}$ joint in the $i^{th}$ future frame (generated after explicit dependency modeling) by:

where $\tilde{\omega}^{j}_{i}$ is the displacement prediction of the ${j}^{th}$ joint in the ${i}^{th}$ future frame, and $\tilde{\omega}^{j^{\prime}}_{i^{\prime}}$ is the displacement prediction of the ${j^{\prime}}^{th}$ joint in ${i^{\prime}}^{th}$ future frame, which might have influence on $\tilde{\omega}^{j}_{i}$. The final displacement vector $\hat{\omega}$ obtained by Eq. 4 can be used to restore the new joint position by incorporating the joint position in the last observed frame.

Previous works such as (Cui et al., 2020; Mao et al., 2020) have generally employed the standard L2 loss. However, the motion ranges of different joints are quite different. Some joints may stay motionless in an action while some joints may undergo large displacements. It is grossly inappropriate to assign the same weight to all the joints. We assign a spatio-temporal weight to each joint. We give higher weights to joints with larger historical position changes (which enforces the network to consider these prominent joints) and to earlier frames in the prediction (which is to reduce accumulation errors). Empirically, this is beneficial to improve the performance of short-term forecasting and can effectively reduce the error accumulation in long-term forecasting. Formally, the loss function can be formulated as:

where $\hat{\omega}^{j}_{i}$ is the displacement vector prediction of the ${j}^{th}$ joint in the ${i}^{th}$ frame, while $\omega^{j}_{i}$ represents the corresponding ground truth. $w^{j}_{i}$ denotes the weight. The loss enforces the predicted frame-wise displacements to approach the ground truth.

Human3.6M is the largest dataset for human motion prediction task, containing 3.6 million human poses and corresponding videos (images). The videos are recorded by a vicon motion capture system. The dataset consists of 7 subjects, each of them performed 15 activities (e.g., discussion, eating, and sitting). There are 32 skeletal joints involved to characterize the human pose. Following the evaluation protocol of previous works (Fragkiadaki et al., 2015), we removed duplicate points, performed a down sampling to 25 frames per second (FPS), and set subject 5 as the test set, with training done on the remaining 6 subjects.

CMU MoCap is released by Carnegie Mellon University. 12 infrared cameras record the position of 41 markers taped on the human body to capture 3D skeleton motion. We adopted the same preprocessing schema as that in (Mao et al., 2019), 7 actions were selected for evaluating the performance of the model (e.g., walking, soccer, and jumping).

We use PyTorch to implemente our method. For training, we utilize the ADAM optimizer with a batch size of 16, a learning rate of 0.001, and a dropout of 0.05. The model is trained for 50 epochs on a Nvidia GeForce Titan Xp GPU. To avoid the problem of exploding gradients, gradient clipping is utilized at a threshold of 5. We use a single convolutional feature channel for the explicit dependency modeling module and the GRU unit size is set to 128.

We evaluate our method and existing works on two popular benchmark datasets. For fair comparison, we follow the evaluation metric in existing works (Mao et al., 2019, 2020). Specifically, the standard Mean Per Joint Position Error (MPJPE) is adopted to measure the mean Euclidean distance between the predicted joint positions and the ground truth. The results are reported on both short-term ($80-400$ms) and long-term ($400-1,000$ms).

H3.6m  We benchmark the proposed method against 8 existing works, including Res-GRU (Martinez et al., 2017), ConSeq2Seq (Li et al., 2018), HMR (Liu et al., 2019), FC-GCN (Mao et al., 2019), TrajNet (Liu et al., 2021c), SDMTL(Liu and Yin, 2020), LDR (Cui et al., 2020), and HRI (Mao et al., 2020). The experimental results are demonstrated in Table 1 and Fig. 3. We report the results of 11 diverse actions and the average over all actions. There are different levels of complexity for different activities. Existing methods tend to perform well for motions with high degrees of periodicity such as “Walking” or regularity such as “Eating”. However, their performance suffers major dips when modeling more stochastic and irregular movements such as “Posing” or “Directions”. On the contrary, a major highlight of our model is the ability to deliver accurate predictions even when faced with highly complex and aperiodic action types. We observe that our method consistently delivers superior performance over state-of-the-art methods, which achieves a substantial 15.4% accuracy improvement in average comparing to state-of-the-art approach. The improvement is even more significant for long-term predictions.

We further visualize sample motion prediction results in Fig. 4, where the results for “Directions” and “Eating” activities are presented. The first three frames correspond to the observed sequences and the subsequent frames are the predicted poses. Each row illustrates the results of a method. It is easy to see that the forecasted sequences obtained by our method are more similar with the ground truth as well as being more natural. A significant issue arising in existing methods for highly stochastic actions is that the predicted motions tend to converge to static motionless states. For example, the HRI (Mao et al., 2020) predictions for “Directions” have very limited range of motions, which appears unnatural. Incorporating displacements and adopting a joint phase space trajectory representation alleviates this plaguing issue, as it provides richer and smoother individualized motion contexts that are easier to model than the pose representation. In the illustrations of the “Eating” action, the hands and foot coordination for our method also appears more coherent. We attribute this to our explicit modeling of the joint dependencies that serves to avoid interference and excess of information from the closely correlated joints.

CMU MoCap  We also evaluate our method on the CMU MoCap dataset. As shown in Table 2, our method consistently and substantially outperforms all the existing methods, especially for complex activity types. For example, on complex activity types such as “Basketball”, “Directing Traffic”, and “Soccer”, our method significantly outperforms state-of-the-art methods by 17.4%, 14.7%, and 25.4% , respectively.

For further evaluation, we show the visual results on the CMU-MoCap dataset in Fig. 5. Our method outperforms all state-of-the-art methods in both long-term and short-term predictions and is proven to be effective and robust in capturing temporal dynamics. For the “Jumping” action, our model successfully captures the fast rise of legs while other methods fail to extract such a trend. More specifically, Res-GRU and FC-GCN fail to predict the subtle correlation between the legs; HRI generates unreasonable prediction for the left foot. For the “Directing Traffic” action, we can observe that our method generates more accurate hand motions than state-of-the-art methods, while Res-GRU exhibits discontinuity between the first frame of prediction and the historical sequence; HRI yields good short-term predictions but is not accurate in the long-term.

We investigate the effectiveness of different modules within our TEID through the following ablation studies. Experiments are performed on the H3.6m dataset, with results reported in Table 3.

Explicit dependency modeling module  The explicit dependency modeling block in our TEID serves to explicitly account for prior knowledge such as natural anatomical connection between joints and coordination of different limbs. From the results in Table 3, we observe that the removal of this module will bring slight accuracy decay in the short-term while the long-term prediction accuracy is greatly affected. We interpret this as the failure to leverage prior anatomy knowledge will be detrimental in effectively filtering out joint correlations, resulting in error accumulation for the long-term.

Implicit global optimization module  This module is designed to globally optimize the position of a joint with respect to the entire forecasted motion sequence. Removing this block results in slight dip in accuracy. This reveals that modeling only explicitly related joints while ignoring irrelevant joints is able to deliver relatively accurate motion prediction. However, implicit relation modeling considers the hidden dependencies between each pair of joints ( e.g., the potential correlation between left hand and right hand joints), which improves the naturalness of the predicted pose sequence.

Phase space trajectory representation  Instead of explicitly incorporating the displacement vector as inputs, we also try employing only the position vectors for each joint. From the quantitative results in Table 3, we observe a significant deterioration in prediction accuracy, testifying to the empirical effectiveness of including displacements. This agrees with our intuition that a complete characterization of the dynamic system with a phase space representation plays an important role in motion prediction.