Self-Attentive 3D Human Pose and Shape Estimation from Videos

Given an input video $V=\{I_{i}\}_{i=1}^{N}$ of length $N$ containing a single person, our goal is to learn a model that recovers the 3D human body of each frame. We present the Self-attentive Pose and Shape Network (SPS-Net), comprising four components: 1) feature encoder $E$, 2) self-attention module $A$, 3) forecasting module $F$, and 4) three parameter regressors $R_{\mathrm{shape}}$, $R_{\mathrm{pose}}$, and $R_{\mathrm{camera}}$.

As shown in Figure 2, we first apply the encoder $E$ to each frame $I_{i}\in V$ to extract the feature $f_{i}=E(I_{i})\in\mathbb{R}^{d}$, where $d$ denotes the number of channels of the feature $f_{i}$. Next, the self-attention module $A$ takes all the encoded features $\{f_{i}\}_{i=1}^{N}$ as input and outputs the corresponding latent representations $\{h_{i}\}_{i=1}^{N}$, where $h_{i}\in\mathbb{R}^{d}$ denotes the latent representation for $I_{i}$, containing temporal information of past and future frames. The forecasting module $F$ takes each encoded feature $f_{i}$ as input and forecasts the feature of the next time step $f_{i+1}^{\prime}=F(f_{i})\in\mathbb{R}^{d}$. The latent representations $\{h_{i}\}_{i=1}^{N}$ and the predicted features $\{f_{i+1}^{\prime}\}_{i=1}^{N}$ of the same time step (e.g., $h_{i}$ and $f_{i}^{\prime}$) are passed to a feature fusion module to derive the fused representations $\{F_{i}\}_{i=1}^{N}$, where $F_{i}\in\mathbb{R}^{d}$ contains both global temporal and local motion information. The pose parameter regressor $R_{\mathrm{pose}}$ takes each fused representation $F_{i}$ as input and renders the pose parameters $\theta_{i}$ for each frame $I_{i}$, where $\theta_{i}=R_{\mathrm{pose}}(F_{i})\in\mathbb{R}^{72}$. The shape parameter regressor $R_{\mathrm{shape}}$, on the other hand, takes all the fused representations $\{F_{i}\}_{i=1}^{N}$ as input and regresses the shape parameters $\beta\in\mathbb{R}^{10}$ of the input video $V$.

Given a sequence of features $\{f_{i}\}_{i=1}^{N}$ encoded by the encoder $E$, our goal is to leverage temporal cues in the input video to provide more information that helps regularize the estimation of human pose and shape. Existing methods exploit temporal information by resorting to an RNN-based model, e.g., GRU (Kocabas et al., 2020) or LSTM (Lee et al., 2018b; Rayat Imtiaz Hossain and Little, 2018). However, training RNN-based models is difficult to capture long-range dependencies (Vaswani et al., 2017; Pascanu et al., 2013).

Motivated by the attention models (Vaswani et al., 2017; Zhang et al., 2019a; Parmar et al., 2018) which have been shown effective to jointly capture short-range and long-range dependencies while being more parallelizable to train (Vaswani et al., 2017), we develop a self-attention module to learn latent representations $h$ that jointly observe past and future video frames for producing temporally consistent pose and shape predictions.

The attention weights $\{a_{i}^{l}\}_{l=1}^{N}$ are computed by

We then apply a weighted sum layer to sum over all input features $\{f_{l}\}_{l=1}^{N}$ with the associated attention weights $\{a_{i}^{l}\}_{l=1}^{N}$. In addition, we add a residual connection (He et al., 2016) to pass the input feature $f_{i}$ to the output of the self-attention module. Specifically, the latent representation $h_{i}$ is described by

In addition to considering global temporal information as in the self-attention module $A$, we exploit visual cues from human motion to encourage our model to generate temporally smooth predictions. Motivated by methods that focus on tackling human motion prediction (Kanazawa et al., 2019; Zhang et al., 2019b), we develop a forecasting module $F$ that takes each encoded feature $f_{i}$ as input and forecasts the feature of the next time step $f_{i+1}^{\prime}$. As the feature of the next time step is available (given by the encoder), we train the forecasting module $F$ in a self-supervised fashion with a feature regression loss:

We note that since the feature of the next time step of $f_{N}$ is not available, we do not compute the feature regression loss on $f_{N+1}^{\prime}$.

To jointly consider the latent representations $\{h_{i}\}_{i=1}^{N}$ that contain global temporal information and the predicted features $\{f_{i+1}^{\prime}\}_{i=1}^{N-1}$ that contain local motion information for predicting the parameters for 3D human pose and shape estimation, we have a feature fusion module that fuses $\{h_{i}\}_{i=1}^{N}$ and $\{f_{i+1}^{\prime}\}_{i=1}^{N-1}$ at the same time step to derive the fused representations $\{F_{i}\}_{i=1}^{N}$. We note that since our encoder $E$ is pre-trained on single-image pose and shape estimation task and fixed during training as in prior work (Kanazawa et al., 2018; Kocabas et al., 2020), the feature $f_{i}$ encoded by the encoder $E$ is static and does not contain motion information. Therefore, we use the predicted feature $f_{i}^{\prime}$ from the forecasting module $F$ that contains motion information for feature fusion.

As shown in Figure 4, our feature fusion module is composed of a fully connected (FC) layer, followed by a softmax layer. Given a latent representation $h_{i}$ and a predicted feature $f_{i}^{\prime}$, we first apply the FC layer to each input feature to predict a weight. The predicted weights are then normalized using a softmax layer. The two input features are then fused by $F_{i}=a_{h_{i}}\cdot h_{i}+a_{f_{i}^{\prime}}\cdot f_{i}^{\prime}\in\mathbb{R}^{d}$. We note that since $f_{1}^{\prime}$ is not available, we define $F_{1}=h_{1}$.

Next, we pass all the fused features $\{F_{i}\}_{i=1}^{N}$ to the shape $R_{\mathrm{shape}}$, pose $R_{\mathrm{pose}}$, and camera $R_{\mathrm{camera}}$ parameter regressors to predict the corresponding parameters, respectively. Similar to one prior work (Kanazawa et al., 2018), we adopt an iterative error feedback scheme to regress the parameters. To train the proposed SPS-Net, we impose a SMPL parameter regression loss $\mathcal{L}_{\mathrm{SMPL}}$ on the estimated pose $\{\hat{\theta}_{i}\}_{i=1}^{N}$ and shape $\hat{\beta}$ parameters, a 3D joint loss $\mathcal{L}_{\mathrm{joint}}^{3D}$ on the predicted 3D joints $\{\hat{X}_{i}\}_{i=1}^{N}$, and a 2D joint loss $\mathcal{L}_{\mathrm{joint}}^{2D}$ on the reprojected 2D joints $\{\hat{x}_{i}\}_{i=1}^{N}$ (Kanazawa et al., 2018; Kocabas et al., 2020). Specifically, the SMPL parameter regression loss $\mathcal{L}_{\mathrm{SMPL}}$, the 3D joint loss $\mathcal{L}_{\mathrm{joint}}^{3D}$, and the 2D joint loss $\mathcal{L}_{\mathrm{joint}}^{2D}$ are defined as

As shown in Figure 5, our discriminator $D$ is composed of a self-attention module $A_{D}$ and a classifier $C_{D}$. We first concatenate the estimated shape parameters $\hat{\beta}$ with each of the estimated pose parameters $\{\hat{\theta}_{i}\}_{i=1}^{N}$ to form the joint representations $\{\hat{J}_{i}\}_{i=1}^{N}$, where $\hat{J}_{i}=[\hat{\beta},\hat{\theta}_{i}]\in\mathbb{R}^{82}$. We then pass all joint representations $\{\hat{J}_{i}\}_{i=1}^{N}$ to the self-attention module $A_{D}$ to derive the latent representations $\{\hat{H}_{i}\}_{i=1}^{N}$, where $\hat{H}_{i}\in\mathbb{R}^{82}$ is the latent representation of $\hat{J}_{i}$. To derive the motion representation $\hat{M}$ of $\hat{\Theta}$, we average all the latent representations $\{\hat{H}_{i}\}_{i=1}^{N}$, i.e., $\hat{M}=\frac{1}{N}\sum_{i=1}^{N}\hat{H}_{i}\in\mathbb{R}^{82}$. The motion representation $M\in\mathbb{R}^{82}$ of $\Theta$ can be derived similarly. The classifier $C_{D}$ takes the motion representations $\hat{M}$ and $M$ as input and distinguishes whether the input motion representations are realistic or not. Specifically, we have an adversarial loss $\mathcal{L}_{\mathrm{adv}}$ which is defined as

Leveraging the unpaired data from the AMASS (Mahmood et al., 2019) dataset serves as a weak supervision to encourage the SPS-Net to recover a sequence of 3D meshes with realistic motions.

We note that our discriminator $D$ is different from that of the VIBE (Kocabas et al., 2020) method in two aspects. First, our discriminator has a self-attention module, while the discriminator of the VIBE (Kocabas et al., 2020) method has two GRU layers. Second, we use self-attention to derive a representation for each frame that contains temporal information by jointly considering short-range and long-range dependencies across video frames, whereas the VIBE (Kocabas et al., 2020) method leverages self-attention to derive a single representation for the entire pose sequence.

We implement our model using PyTorch (Paszke et al., 2019). Same as prior work (Kanazawa et al., 2018; Kocabas et al., 2020), we adopt the ResNet-$50$ (He et al., 2016) pre-trained on single-image pose and shape estimation task (Kanazawa et al., 2018; Kolotouros et al., 2019a) to serve as our encoder $E$. Our encoder $E$ is fixed and outputs a $2,048$-dimensional feature for each frame, i.e., $f_{i}\in\mathbb{R}^{2048}$. We set the length of the input sequence to $32$ with a batch size of $16$. Both the attention network $Q$ and the attention network $K$ in the self-attention module $A$ consist of $2$ fully connected layers, each of which has a hidden size of $2,048$, followed by a LeakyReLU layer. As for the forecasting module $F$, unlike prior methods (Zhang et al., 2019b; Kanazawa et al., 2019) that use 1D convolution layers, our forecasting module $F$ is composed of $2$ fully connected layers, each of which has a hidden size of $2,048$, followed by a LeakyReLU layer. Both the attention network $Q$ and the attention network $K$ in the self-attention module $A_{D}$ also consist of $2$ fully connected layers, each of which has a hidden size of $82$, followed by a LeakyReLU layer. The classifier $C_{D}$ in the discriminator $D$ is composed of a fully connected layer, followed by a sigmoid function. The input and output dimensions of the classifier $C_{D}$ are $82$ and $1$, respectively. Similar to the HMR (Kanazawa et al., 2018), the SMPL (Loper et al., 2015) parameter regressor $\{R_{\mathrm{pose}},R_{\mathrm{shape}}\}$ is composed of $2$ fully connected layers with a hidden size of $1,024$. The shape $R_{\mathrm{shape}}$, pose $R_{\mathrm{pose}}$, and camera $R_{\mathrm{camera}}$ parameter regressors are initialized from the pre-trained weights of the HMR (Kanazawa et al., 2018) approach. The weights of the self-attention module $A$, the forecasting module $F$, the feature fusion module, and the discriminator $D$ are randomly initialized. We use the ADAM (Kingma and Ba, 2014) optimizer for training. The learning rates for the SPS-Net and the discriminator $D$ are set to $5\times 10^{-5}$ and $1\times 10^{-4}$, respectively. Following the VIBE (Kocabas et al., 2020) method, we set the hyperparameters for the loss functions as follows: $\lambda_{\beta}\mathrm{=}0.06$, $\lambda_{\theta}\mathrm{=}60$, $\lambda_{\mathrm{joint}}^{3D}\mathrm{=}300$, $\lambda_{\mathrm{joint}}^{2D}\mathrm{=}300$, and $\lambda_{\mathrm{adv}}\mathrm{=}2$. For the other hyperparameters, we set $\lambda_{\mathrm{feature}}\mathrm{=}1$, $\lambda_{\mathrm{mask}}\mathrm{=}300$, $\lambda_{\mathrm{camera}}\mathrm{=}0.1$, $\lambda_{\mathrm{param}}^{\beta}\mathrm{=}0.06$, $\lambda_{\mathrm{param}}^{\theta}\mathrm{=}60$, and $\lambda_{\mathrm{param}}^{\mathrm{camera}}\mathrm{=}0.1$. We train our model on a single NVIDIA V$100$ GPU with $32$GB memory for $120$ epochs. For each epoch, there are $500$ iterations.

We describe the datasets and the evaluation metrics below.

We compare the performance of our SPS-Net with existing frame-based methods (Yang et al., 2018; Chen et al., 2019a; Kocabas et al., 2019; Mehta et al., 2017b; Cheng et al., 2020; Wandt and Rosenhahn, 2019; Kolotouros et al., 2019b; Sengupta et al., 2020; Omran et al., 2018; Choutas et al., 2020; Zanfir et al., 2020; Kanazawa et al., 2018; Kolotouros et al., 2019a) and video-based approaches (Kanazawa et al., 2019; Arnab et al., 2019; Doersch and Zisserman, 2019; Sun et al., 2019; Kocabas et al., 2020). Table 1 presents the quantitative results on the 3DPW (von Marcard et al., 2018), MPI-INF-3DHP (Mehta et al., 2017a), and Human3.6M (Ionescu et al., 2013) datasets.

Experimental results on all three datasets show that our method performs favorably against existing frame-based and video-based approaches on the PA-MPJPE, MPJPE, PVE, and PCK evaluation metrics. However, the acceleration error of our method is inferior to that of the HMMR (Kanazawa et al., 2019) approach. The reason for the inferior performance is that the goal of the HMMR (Kanazawa et al., 2019) method lies in predicting past and future motions given a single image. While we have a forecasting module $F$ that predicts the feature of the future frame based on past information, we do not aim to optimize the performance on the human motion prediction task but instead focus on learning to estimate 3D human pose and shape of the current frame. On the other hand, as noted by the VIBE (Kocabas et al., 2020), the HMMR (Kanazawa et al., 2019) method applies smoothing to the predictions, leading to overly smooth pose predictions at the expense of sacrificing the accuracy of pose and shape estimation.

In addition to quantitative comparisons, we present 1) visual comparisons with the VIBE (Kocabas et al., 2020) and SPIN (Kolotouros et al., 2019a) methods, 2) visual results of occlusion handling, and 3) visual results of different viewpoints.

Without the camera parameter consistency loss $\mathcal{L}_{\mathrm{camera}}$, there is no explicit constraint imposed on the prediction of camera parameters, leading to performance drops of $1.7$ in PA-MPJPE and $3.5$ in PVE. When removing the mask loss $\mathcal{L}_{\mathrm{mask}}$, our model does not have any constraints to regularize the 3D mesh. Performance drops of $5.9$ in PA-MPJPE and $7.0$ in PVE occur. Without the self-supervised parameter regression loss $\mathcal{L}_{\mathrm{param}}$, our model does not learn to produce plausible predictions when the occlusion or out-of-view issues occur, resulting in performance drops of $5.4$ in PA-MPJPE and $4.6$ in PVE. When removing the adversarial loss $\mathcal{L}_{\mathrm{adv}}$, our model does not learn to render 3D meshes that have realistic motions. Performance drops on all three evaluation metrics occur, which also concur with the findings in the HMR (Kanazawa et al., 2018) and VIBE (Kocabas et al., 2020).

Figure 9 presents two visual comparisons with the variant methods of our SPS-Net (i.e., Ours w/o $\mathcal{L}_{\mathrm{camera}}$ and Ours w/o $\mathcal{L}_{\mathrm{mask}}$). Our visual results show that both the camera parameter consistency loss $\mathcal{L}_{\mathrm{camera}}$ and the mask loss $\mathcal{L}_{\mathrm{mask}}$ allow our model to predict more accurate pose and shape estimates.

The ablation study on loss functions shows that all four losses are crucial to the SPS-Net.

We observe that when the hyperparameter is set to $0$ (i.e., the corresponding loss function is removed), our SPS-Net suffers from performance drops. When the hyperparameters are set within a suitable range (i.e., around $300$ for $\lambda_{\mathrm{mask}}$ and around $0.1$ for $\lambda_{\mathrm{camera}}$), the performance of our SPS-Net is improved, demonstrating the effectiveness of the corresponding loss function. When the hyperparameters are set to large values (e.g., $1\times 10^{4}$ for $\lambda_{\mathrm{mask}}$ and $1\times 10^{3}$ for $\lambda_{\mathrm{camera}}$), our model training will be dominated by optimizing the corresponding loss, leading to performance drops.

The sensitivity analysis of hyperparameters shows that when each hyperparameter is set within a suitable range, the performance of our method is improved and remains stable.

We present the failure cases of our method in Figure 11. As our SPS-Net assumes that the input video frames contain a single person, if missing detection happens, our method will not be able to perform human pose and shape estimation.