3D Pose Estimation and Future Motion Prediction from 2D Images

Previous works investigated the usage of Lie algebra-based pose representation [9, 33] to provide implicit physical constraints for estimating the 3D pose of an articulated object. We further attempt to use the Lie algebra pose representation to consolidated the prediction. The self-projection module, which is a derived version of the Mixture-of-Expert layer for regressing both the joint relative coordinates and the Lie algebra pose representation, is shown in 3. A Mixture-of-Expert design of soft parameter sharing is utilized. The output from the RNN cell is the input of a shared layer followed by the two separated expert paths by using softmax gating.

Given a kinematic chain of m joints with the corresponding parametrized Lie algebra pose $\mathrm{\mathbf{p^{\text{Lie}}}}=\left({\xi_{1}^{1}}^{\intercal},\cdots,{\xi^{1}_{m_{1}}}^{\intercal},\cdots,{\xi^{K}_{1}}^{\intercal},\cdots,{\xi_{m_{K}}^{K}}^{\intercal}\right)^{\intercal}$, we are able to obtain the location of the Joint $\mathbf{J}_{i}$ by a completely differentiable forward kinematics

At time step $t$, the desired output from both the coordinate pose path and the Lie algebra path should be well-aligned after the forward kinematic inference if the predictions are perfect. The training process should be able to penalize the bad prediction where the two outputs are very different after the kinematic chain forwarding. We define this process as $\mathbf{p}^{\text{Coord}}_{(t)}=f_{\text{kin}}(\mathbf{p}^{\text{Lie}}_{(t)})$. Note that this kinematic chain forwarding process contains inevitable tiny biases during the calculation, therefore we consider this as a regularization step for a more stable training process. The L2 norm of the outputs of these two paths is then used as a regularization term for supervision, denoted as

The Pose Lifting Network (PLN) is designed to handle the task of lifting a 2D pose sequence to its corresponding 3D pose trajectory on its own duty, as well as behave as the encoder network in the overall framework.

As shown in Figure 4, the architecture design of the PLN is summarized as the following: it consists of a feed-forward neural network (FFNN), then a 2-layer bi-directional Gated Rectified Unit (GRU), and finally a Self-Projection (SP) layer for producing the 3D pose trajectory. There are two linear layers in the FFNN and both are followed by a Rectified Linear Unit (ReLU) as the non-linearity. Dropout is added after all non-linearity layers except for the output layer from the SP layer, which does not have an activation function.

More formally, we define the input of the PLN as a length $T$ sequence of 2D pose $\mathbf{\hat{P}}^{\text{2D}_{(1:T)}}=\langle\mathbf{p}^{2D}_{(1)},...\mathbf{p}^{2D}_{(T)}\rangle$. There are two outputs from PLN: a latent representation of the entire sequence, $z_{T}$, and a predicted 3D pose sequence $\mathbf{\hat{P}}^{\text{3D}}_{(1:T)}$. We note that $\mathbf{\hat{P}}^{\text{3D}}_{(1:T)}=\langle\mathbf{\hat{p}}^{3D}_{(1)},...\mathbf{\hat{p}}^{3D}_{(T)}\rangle$ where at each time step $t$, $\mathbf{\hat{p}}^{\text{3D}}_{(t)}=\langle\mathbf{\hat{p}}^{\text{Coord}}_{(t)}$, $\mathbf{\hat{p}}^{\text{Lie}}_{(t)}\rangle$ with $\mathbf{\hat{p}}^{\text{Coord}}_{(t)}\in\mathbf{R}^{16\times 3}$ and $\mathbf{\hat{p}}^{\text{Coord}}_{(t)}\in\mathbf{R}^{16\times 6}$. The latent representation, $z_{T}$, is a tensor of shape $(\text{RNN layer \#}\times\text{Hidden Size}\times 2)$. With abbreviating the forward pass as a function mapping $\phi_{\text{PLN}}$, the loss function for PLN is

The overall architecture of the Motion Generator Network (MGN) is similar to the PLN with the following differences, as shown at the right side of Figure 4. Starting from the first step of motion generation at time $T+1$, the input is the predicted 3D pose $\mathbf{\hat{p}}^{\text{3D}}_{(T)}$ instead of the ground truth 3D pose $\mathbf{p}^{\text{3D}}_{(T)}$. To our knowledge, this setup also differs from all previous works [8, 9, 11, 10]. We follow the common practice presented in previous works [11] to predict the residual of the motion from the previous time step.

Mathematically, at time step $T+1$, the input of the MGN is $\mathbf{\hat{p}}^{\text{3D}}_{(T)}$ and the latent representation $z_{T}$. As before, we use a function mapping $\phi_{\text{MGN}}$ to abbreviate the forward pass in MGN. Consider the future motion prediction is a $K$-step process, for every time step $t$, where $T+1\leq t\leq T+K$, the predicted 3D pose $\mathbf{\hat{p}}^{\text{3D}}_{(t)}=\mathbf{\hat{p}}^{\text{3D}}_{(t-1)}+f_{\text{MGN}}(\mathbf{\hat{p}}^{\text{3D}}_{(t-1)},z_{t-1})$. We shorten the entire step-by-step process as $\mathbf{\hat{P}}^{\text{3D}}_{T+1:T+M}=\phi_{\text{MGN}}(\mathbf{\hat{p}}^{\text{3D}}_{(T)},z_{t-1})$. The loss function for the MGN is then

We investigate two baseline architectures, the vanilla Gated Rectified Unit (GRU) and a Convolutional Encoder-Decoder (ConvED) to handle both the temporal and spatial context inside the pose trajectory, and then we propose a Global Refinement (GR) module built upon the two baselines for pose sequence refinement. The overall design in demonstrated in Figure 6. Overall, the refinement is performed over the entire predicted pose trajectory, $\mathbf{\hat{P}}^{\text{3D}}_{(1:T+M)}$. The refinement module contains two paths and a trainable adaptive weighting vector, $\mathbf{\alpha}\in(0,1)$. We denote the two forward passes that happened in the two paths as $\phi_{\text{GRU-R}}$ and $\phi_{\text{ConvED}}$. Formally, with $\odot$ refers to the Hadamard product, the forward pass of the GR Module can be represented as $\phi_{\text{GR}}(\mathbf{\hat{P}}^{\text{3D}}_{(1:T+M)})=\mathbf{\alpha}\odot\phi_{\text{GRU-R}}(\mathbf{\hat{P}}^{\text{3D}}_{(1:T+M)})+(\mathbf{1}-\mathbf{\alpha})\odot\phi_{\text{ConvED}}(\mathbf{\hat{P}}^{\text{3D}}_{(1:T+M)})$, which gives us another loss function

The idea of using a 2D convolutional architecture to process pose trajectories can become intuitive if the pose trajectory is converted to a spatio-temporal grid representation which is identical to common digital images. A demonstration of the spatio-temporal pose grid representation is shown in Figure 5. Suppose the human skeleton model $J$ joints, and there are $N$ time steps and in total for the human pose trajectory. A built pose grid is then of shape $(J,N,D)$ where the height is the joint index dimension and the width is the temporal dimension. We only show the $D=3$, i.e., a 3-channel case to save the space but it can be easily generalized to a higher dimension case when the Lie Algebra representation is included (i.e., from 3 channels to 9 channels). While the adjacent sub-regions in the grid represent closely related joints, different sizes of receptive fields across different convolutional layers can, therefore, learn the spatial relationship of joints across the time dimension implicitly.

The overall loss function which is a weighted combination of the loss functions defined in the above sections:

In our experiments, we use $\beta=0.2$ and $\lambda=0.01$. The value of the weights are chosen by empirical results. It is worth to notice that the $\Omega_{\text{SP}}$ term behaves more like a constraint noise regularizer as there are always small biases caused by the re-projection process, i.e., the process of converting the Lie algebra-pose to the coordinate-based pose. Using a weighted loss in our framework can be considered from two aspects. First, $\beta$ is used to control the balance of the importance between pose estimation and future motion prediction. Consider the stochasticity and a usually longer length of the temporal scope for motion generation, the weight of the motion generation loss term is reduced to 0.2. Second, the regularizer weight, $\lambda$, is used to re-scale the regularizer to a similar scale of the other terms in the loss function.

It has been shown in previous works that directly regressing the future poses with the ordinary ”XYZ” pose representation [8, 9, 11] can easily produce unrealistic poses, we first validate this observation with our PoseMoNet by turning off the supervision at the Lie algebra pose representation outputted by the self-projection module. It is worth to highlight that, although the ordinary ”XYZ” pose representation is most commonly used and there is no significant shortage of using it, we argue and empirically show that only the plain pose representation is insufficient in our work. We have 2 fundamental differences compared with existing works. First, to the best of our knowledge, a 3D MoCap pose sequence is required as the input for all the existing 3D future motion prediction method, where in our work, we only use 2D pose keypoint sequence as the only input to the task. Secondly, in our work, we are looking 1 more step further by considering a real world application scenario, we only leverage estimated 2D keypoint location without any human intervention, which make the problem more difficult and significantly different. The result shows that both training and prediction are unstable and can easily lead to divergence. Therefore, we omit the quantitative comparison for this ablative experiment and show an expressive qualitative visualization.

We demonstrate the visual results in Figure 7. As mentioned, we first train our framework without supervision on the Lie algebra pose representation. The result is shown in the middle column, denote as ”XYZ”. We also replicated the model proposed as Res-GRU [11] and show the result in the right column. We see that the previous competitive method may also suffer from the issue caused by no physical constraints. Finally, we add the results predicted by our framework trained completely with both pose representation and all loss terms. Not surprisingly, while other setups start to diverge on the predictions under a longer-than-usual context, the prediction from our framework, even after a 1600ms temporal context, is still natural and dynamic. This observation consolidates the effectiveness of including the Lie algebra pose representation in the training phase as well as using it to provide self-regularization with implicit physical constraints.

We demonstrate the advantage of building the multitask framework over training two standalone models with ablative experiments. The first group of models is Pose Liter Networks (PLN) being trained without the Motion Generator Networks (MGN) followed after. Different input lengths are covered in our experiments: the sequences of 5, 9, 27, and 54 frames, are used to train 4 PLNs, respectively. We then train another group of 4 models with the exact same configuration but with the Motion Generator Network trained together. The comparison of the quantitative results between these two groups is shown in Table 1.

The quantitative results provide us a better understanding of the benefits of introducing the multitask setting. We observe that, for varying lengths of input sequences, training the PLN along with the MGN leads to better quantitative results under both protocols. Surprisingly, if we focus on protocol 2, P-MPJPE, the distance metric after the alignment at scale, rotation, and transformation, we notice a more significant boost over protocol 1 (MPJPE). We argue that by including the MGN, i.e., providing a broader temporal context in the future, our PLN obtained stronger constraints and resulted in predicting results that are closer to the ground truth, as shown in Figure 8.

For the sake of the completeness of the ablative study, we also performed experiments on removing the video pose estimation task, i.e., the supervision for 3D pose estimation is turned off and therefore only 2D pose sequence is used directly as input to generate future motion, and the training process suffered from huge fluctuation and diverged quickly, which consolidated the fact that 2D and 3D pose are not bijective therefore it can lead to massive ambiguity even in the latent space function mapping.

The proposed global refinement module consists of two baseline refinement networks mentioned previously. The effectiveness of the global refinement module is validated from two perspectives. To select the best architecture for pose trajectory refinement, we compared different settings for refining the entire pose trajectory under the following settings: using only the GRU-based model, using only the ConvED, and combining them to the proposed global refinement module. Quantitative results are obtained with different lengths of input video sequences and reported in Table 2. We see that the GRU alone provides more improvement for pose estimation, which consolidated our observation that a significant amount of errors on pose sequence estimation is caused by the non-smooth prediction among consecutive frames. On the other hand, ConvED focuses more on the spatiotemporal context of the pose sequence, which not surprisingly enhanced the result of the motion generator more than the GRU. On combining results from both paths, the adaptive weighting mechanism has been shown to be helpful in terms of quantitative performance.

Besides, the effectiveness of the refinement module along with the multitask vs. single task setting is also examined by first attaching the GR module only on the estimated 3D pose sequence, i.e., the output from the Pose Lifting Network, then we apply the GR module on the entire estimated pose sequence (predicted 3D pose + predicted future motion). Qualitative results is briefly reported in Table 2.

As shown in Table 3, we compare with previous state-of-the-art methods for 3D pose estimation on the Human3.6M dataset. Different collections of methods are included. We have evaluated the methods proposed in [2, 1] which focus on 3D pose estimation from a single image. For a fair comparison, we also compete with methods that leverage a video sequence as input [6, 5, 20, 21, 14, 49]. It is important to note that a recently proposed work [14] leverages extra data augmentation step to improve the 2D keypoint detection result, which further pushed the performance. Two results, one without the data augmentation step (which is using the same input compared with other methods) and one with their proposed data augmentation technique, are reported from [14] and we include both results. Unfortunately, as the source code is not publicly available at the time we conduct experiments, we are not able to reproduce the proposed data augmentation. For the sake of completeness, we report both results in Table 3.

We can see that compared with the state-of-the-art methods, by exploiting the temporal information along the sequence as well as the motion dynamics from future motion prediction task, our approach achieves the decent performance with the smallest error averaging across all actions and holding the 1st place for most categories of actions, such as Eating, Greet, Purch, etc. Interestingly, two methods [6, 5] that use very large context information ($\geq$ 243 frames) outperform our framework on action types ”Walking” and ”Walking Together”. These two types of actions are highly predictable if being estimated from a larger context due to its cyclic nature, where the pose estimation task can be benefited. On the other hand, our method performs particularly well on more random actions, such as ”Eating”, ”Photo”, ”Purch” and ”Smoke” compared with existing methods. This observation meets the fact that a human does not need to see a relatively long presence of other people to estimate and understand their pose from 2D inputs.

Furthermore, our framework only needs to observe a relatively short sequence (54 frames) to produce sound predictions for both 3D pose estimation and motion prediction, which makes it comes with the potential to be deployed easier in a real-world real-time situation. We argue that the longer the context is, the more diminishing of returns, even potential negative effects, can be observed. This can be seen in the cases of estimating pose stochastic activities such as ”Sitting”, ”Purch”, ”Posing” and ”Discussing”. Our method has more stable and quantitatively better results compared to methods that leverage a larger context [5, 14]. We have also noticed that existing works can produce better quantitative results with even a single-frame input [5] though longer contexts are used in our framework, we argue that the architecture in [5] at the single-frame setting requires significantly more computations as it is a 5-block residual architecture, also with more parameters, where our recurrent-based architecture can be more efficient. It is also worth to mention that the future motion predictions task suffer significantly from using an estimated 2D keypoint sequence as the original input, it also affects the performance of pose estimation, particularly when the sequence is short (i.e., 9-frame).

Consider that the skeleton visualization of the 3D poses is somehow abstract, we attempt to perform a reverse fitting process with the SMPL [26] shape model. A grounded visualization of the estimated poses can be obtained and is shown in Figure 9. The predictions produced by our proposed framework can not only achieve decent quantitative performance but they are also physically reasonable that rarely the body parts have collision or cross each other. Furthermore, we also provide visualization for HumanEva-I here to demonstrate the generalization ability of the proposed work.

It is worth highlighting that our proposed work has a fundamental difference with the existing motion prediction literature where makes it is infeasible to have a completely fair and controlled experiment: we do not use the historical 3D ground truth as the input of the motion prediction task. In Table 4, all the previous state-of-the-art methods use past 3D ground truth as the context as well as a ground truth seed pose for generating the future poses. We highlight our results with bold font whenever our results can achieve the top-3 overall ranking. In spite of this, it is still clear that our quantitative results are comparable with methods that leveraged 3D ground truth. This helps to demonstrate the further potential of our proposed framework.

Following the literature, we evaluate the future motion generation task on four commonly used test action classes, namely, discussion, greeting, posing, and walking dog. Mean Angle Error (MAE) is reported for this task. We compare with milestone works in the literature including ERD [10], S-RNN [31], Res-GRU [11], HMU [7] and a recently proposed Lie algebra-based method, HMR [9]. Unfortunately, as mentioned before, there is no easy way to waive the disadvantage of not using 3D ground truth but we are still able to obtain comparable quantitative results.

From Table 4, we can see that we can achieve comparable quantitative performance with the state-of-the-art methods [8, 34] within a 560ms time interval. Interestingly, the quantitative advantage of our method starts to deviate after 720ms. First, we believe this observation is acceptable, as there is, indeed, a potential drawback of using only the 2D pose sequence as the input. Besides, we argue that there are always trade-offs between the quantitative result and the quality of generated future motion as it has been pointed out that a very simple baseline, Zero-velocity [11], which always produces the static mean pose outperforms many existing works quantitatively.

We further validate the above observation by a qualitative visualization provided in 10. Surprisingly, we found that the relatively high error measured under the MAE metrics not necessarily translates to a ”bad” prediction of the future motion due to the fact that MAE is ambiguous which leads to the result of different sets of angles that may yield similar poses. The visualization further consolidated that we are still able to generate sound poses for future motions. In particular, many previous methods have the problem that the predicted future motion starts to either drift or converge to a static pose [10, 7] but in our case, for example, the walking action can keep being dynamic after a relatively long period.

Note that the shape visualization is only for a demonstrative purpose, we do not estimate the shape parameters in our framework. This fact also proves the effectiveness of our framework from another aspect where the predicted future motions are realistic and dynamic even we do not explicitly model the shape and constraints provided with the shape fitting process.

For pursuing a relative fair comparison, with the lighted shined by state-of-the-art methods, we have also either re-implemented [11, 8] or adopted public code base[34] to retrain previous methods by feeding them the estimated 3D historical pose sequence as the input. Due to the fact that different skeletal structures are used (different number of joints) as well as the input is no longer historical 3D ground truth poses, the quantitative results shacked significantly and we reported them in Table 5. Clearly, the performance of the previous methods is significantly affected by the noisier input and our method is achieving the best or the second-best position among most of the time horizons.