SDMTL: Semi-Decoupled Multi-grained Trajectory Learning for 3D human motion prediction

Inspired by the architecture of residual block (He et al. 2016), we propose a new SE module to model the spatial correlations among joints of the human body, and the detailed architecture of SE is shown in Figure 2. There are two major differences from the commonly used residual block: (1) we discard the batch normalization (BN) layer. Different from traditional classification tasks, the data distributions of human movements should be different, and using BN layers will degrade the performance of our predictive model. Therefore, we did not use the BN layers. (2) We add a $1\times 1$ Conv layer during the skip connection. This enables the SE to learn the pixel-level features from the inputs and then augments the output features with the pixel-level features.

Inspired by the squeezed channel scheme of RMB (residual multiplicative block) (Kalchbrenner et al. 2017), we build an RSE module using SEs, and the detailed implementation is shown in Figure 2. We have empirically shown that the squeezed channel scheme in RSE, i.e., first reduce the channel number of followed two stacked SEs by half and then restore the channel number to that of the inputs, can not only promote the computational efficiency but also improve the performance of the module to some extent.

As is shown in Figure 2, BSME is built with SEs to model motion dynamics of two adjacent inputs and provides an extra interface that can further fuse other information via a shortcut and element-wise summation according to the specific need, where $\emph{{F}}^{l}_{t}$ denotes the tensor of the $l$-th level at the $t$-th time-step, and $\emph{{X}}^{l}_{i}$ denotes the information that can be fused. Compared with the current pose, the spatial structure of early poses is increasing varying with time, and thus the spatial modeling of the previous pose is more complex than that of the current pose. Therefore, we use more SEs to encode the spatial features of previous input ${\emph{{F}}^{l}_{t-1}}$ and use less SEs to encode the spatial features of the current input ${{\emph{{F}}}^{l}_{t}}$.

BSME is built based on two insights: (1) Semi-decoupled motion-sensitive features. Inspired by the mechanisms of the human brain that receives spatial and temporal information via different pathways respectively, we model the spatiotemporal representation in a semi-decoupled manner, i.e., first decouple the process of spatial and temporal modeling and then fuse them together. Specifically, we first model the spatial features of each input using stacked SEs and then obtain the motion features of the adjacent inputs by element-wise subtraction, and the motion features are further process using another stacked SEs. Finally, the semi-decoupled spatiotemporal features can be obtained by fusing the motion features with spatial features via short connection and element-wise summation. During the subtraction, the static features are eliminated and the remained features are sensitive to the human motion, therefore, the proposed BSME is motion-sensitive. (2) Multi-grained features. The blue boxes in Figure 2 denote the $1\times 1$ Conv layers without activated function, enabling the network to obtain the axis-level features. With the multi-level short connections in BSME, we can get multi-grained features as the output by gradually fusing these axis-level features. Moreover, we empirically show that these multi-grained features can greatly help to capture rich motion dynamics for predictions.

The encoder is to encode spatial correlations among joints of the human body, and the decoder is to decode spatial information of the human body. Therefore, the process of encoder and decoder is similar, and the decoder can be regarded as the inverse process of the encoder. In this paper, we propose an encoder and decoder with similar architecture, as is shown at the right of Figure 3. Specifically, for the encoder, a $1\times 1$ Conv layer is first applied to model the axis-level features of joints, and then stacking $l_{en}$ SEs enlarges the spatial receptive field to model the spatial correlations among joints of the human body; for the decoder, stacking $l_{de}$ SEs reconstructs the spatial features of the future pose, and we apply another Conv layer and set the number of output channels to 1 since we focus on predicting a single pose at one time.

As is shown in Figure 3, the multi-grained trajectory modeling is built with BSMEs at multi-levels to summarize the temporal information at multi-granularity and gathering the outputs of each scale to model temporal evolutional directions of their trajectories. Therefore, our multi-grained trajectory modeling both contains temporal dynamic summarization with adaptive granularities and temporal evolutional trajectory modeling.

For temporal dynamic summarization with adaptive granularities, we focus on summarizing temporal dynamics at multi-granularity using BSMEs on multi-levels and model each granularity at each level. In the $l$-th level, we model the temporal information at scale $2^{l}$. The BSMEs in the same level share parameters, which can build a lightweight representation for each granularity and also improve the performance potentially in the mean time. At the $\left\lceil{{{\log}_{2}}T}\right\rceil$-th level (“$\left\lceil\cdot\right\rceil$” denotes a ceil function), we can model the global temporal information of previous $T$ poses. Moreover, we empirically show that more levels can further improve the performance of the model, and therefore we stack more levels with the same inputs at the $(\left\lceil{{{\log}_{2}}T}\right\rceil+1)$-th level to the $(\left\lceil{{{\log}_{2}}T}\right\rceil+{l_{m}})$-th level. Here, we empirically show that setting the total number of levels to ($T$-1) works the best. Therefore, the total number of time scales is adaptive to the length of the input $T$.

The information flow of the extra interface of BSMEs is hidden in Figure 3. In this paper, let $\{\emph{{F}}^{0}_{i}\}$ ($i=1,2,\cdots,T$) be the output of the $i$-th encoder, we initialize the input of the extra interface to $\{\emph{{F}}^{0}_{i}\}$ as $\emph{{X}}^{l}_{j}=\emph{{F}}^{0}_{k}$ ($k={{2^{l}}j}$, $j=1,2,\cdots,\left\lceil{\frac{T}{{{2^{l-1}}}}}\right\rceil$), where $\emph{{X}}^{l}_{j}$ denotes the input of the extra interface of the $j$-th BSME in the $l$-th level. In this way, we can promote the information flow among different levels to better capture multi-granularity features and further enhance the captured semi-decoupled spatiotemporal features by gradually fusing the spatial features from the low layers.

For temporal evolutional trajectory modeling, we focus on capturing the information of temporal evolutional directions at each granularity. For this, we gather the outputs of BSMEs at each level by concatenating along the time axis and then apply one Conv layer to conveniently capture the evolutional directions of this granularity since the temporal dimension is set as the channel of the input tensor.

To sum up, with the elegant design in both BSME and multi-grained trajectory modeling, the spatiotemporal information flows from the low level to a higher level in a repeated semi-decoupled manner.

Different human motion reflects different multi-granularity, and therefore it is necessary to aggregate the information with proper granularities. As is shown in Figure 3, stacking 3 FC layers with sigmoid as the activation function is applied to the top level of multi-grained trajectory modeling to learn a group of weights for each level, denoting by $\{{\alpha_{l}}\}_{l=1}^{l=T-1}$. Given the features at the $l$-th level as $\emph{{F}}^{l}$ as is denoted in Figure 3, the multi-granularity information can be aggregated by weighted summation as equation 1.

(1) Human3.6M (Ionescu et al. 2013) (H3.6M) is the most widely used dataset for 3D human motion prediction, containing 7 actors performing 15 daily activities such as walking and smoking. (2) CMU mocap dataset (CMU-Mocap)¹¹1http://mocap.cs.cmu.edu/: consistent with the baselines, eight actions are selected for evaluations, including running, basketball, etc.

Consistent with baselines, we adopt the same training set, validating set, and testing set for experiments and apply the same data processing. All models are implemented with TensorFlow. Adam optimizer is used to optimize our models on one GTX 1080Ti GPU by minimizing our loss, and the learning rate is initialized to 0.0001. The number of output channels (i.e. $C$) is set to 64 and the kernel size (i.e. $k$) is set to 3. In experiments, we empirically show that these hyperparameters work the best, i.e. $l_{en}$ =1, ${l_{de}}$ =1 and $\alpha$=0.3. The skeletal representation used in this paper is the same as that in (Liu et al. 2019a). We use MPJPE in millimeter as our metric. We predict 10 future poses (400 milliseconds) for short-term prediction and predict 25 future poses (1 second) for long-term prediction, conditioning on 10 historical poses. All experimental settings are consistent with the baselines. More details and experimental results can be found in the supplemental materials.

(1) Quantitative results on H3.6M. As is shown in Table 1, compared with baselines, our model achieves the best performance with a unified model for both short-term and long-term prediction in most cases, demonstrating the effectiveness of our method. This contrasts with LTD (Mao et al. 2019) that requires training different models with different settings to achieve their best performance. Moreover, on average, the performance gaps of early time-steps (i.e. <500ms) between the short-term model and the long-term model of our method are significantly smaller than that of LTD, further showing the effectiveness of our method powerfully.

Our superior performance benefits from two folds. ($a$) In contrast to the coupled (Martinez, Black, and Romero 2017; Li et al. 2018) or decoupled (Mao et al. 2019) spatiotemporal modeling as done in prior work, the proposed BSME models the spatiotemporal information in a semi-decoupled manner, i.e., first decoupled the process of spatial and temporal modeling and then fuse them together. Such a semi-decoupled strategy simplifies the spatiotemporal modeling of the network that forces different components of the network to focus on modeling either spatial information or temporal information. ($b$) In contrast to the previous work that either models multi-granularity information via fixed modes or focuses on a single granularity modeling, we capture rich multi-granularity information via multi-grained trajectory learning by summarizing temporal dynamics with adaptive granularities and modeling temporal evolutional directions at each granularity.

(2) Quantitative results on CMU-Mocap. As is shown in Table 2, our method consistently outperforms the baseline (Mao et al. 2019) on average, demonstrating the effectiveness of our method again. This thanks to our semi-decoupled spatiotemporal modeling with BSMEs and our novel network that captures rich multi-granularity information.

To further show the effectiveness of our method, we visualize the predictions of the long-term model frame-by-frame on H3.6M and CMU-Mocap respectively, and the results are reported Figure 4 and Figure 5.

(1) Qualitative results on H3.6M. As is denoted in Figure 4, for the right hands in both Figure 4(a) and Figure 4(b), our predictions are closer to the groundtruth than that of LTD, demonstrating the effectiveness of our proposed method. This may benefit from the advantages of our method that captures rich motion dynamics in a semi-decoupled manner via both the summarization of temporal dynamics with adaptive granularities and the modeling of temporal evolutional directions at each granularity, which contrasts to that of LTD that focuses on modeling motion dynamics at a single granularity. Therefore, we capture rich motion dynamics for accurate predictions.

(2) Qualitative results on CMU-Mocap. As is shown in Figure 5, for the hands in Figure 5(a) and the right legs in Figure 5(b), our predictions are significantly better than that of LTD, showing the effectiveness of our method again. Specifically, for the predictions of LTD, the right legs of predictive poses change slightly over time, contrasting to the groundtruth. By contrast, the right legs of our predictive poses change greatly, and the temporal evolutional law is basically consistent with that of the groundtruth, demonstrating the effectiveness of our method powerfully. Moreover, although the predictions at the longer time-steps are slightly worse than that of LTD, as is shown in Figure 5(b), the generative poses are still reasonable. The main reasons for our superior performance are two-fold: ($a$) the proposed BSME captures semi-decoupled spatiotemporal features and it is sensitive to the motion; ($b$) we capture rich motion dynamics of human movement via multi-grained trajectory learning. This is different from that of LTD that focuses on modeling motion dynamics at a single granularity and ignores the nature of human motion at more granularities. Therefore, we can better capture motion dynamics for predicting the complex human motion.