Synthetic Training for
Monocular Human Mesh Recovery

We aim to train a single-shot model for whole-body 3D human mesh recovery, including body, hands, and expression from a single uncalibrated image. By involving the statistical model SMPL-X [15] to encode the whole-body 3D human mesh, the task is converted to estimate the SMPL-X parameters from a 2D image. Fig. 3 shows an overview of the proposed framework. Take a single 2D image as input, the outputs of our network are global body/camera and partial hands/expression parameters for the SMPL-X model that derives the 3D mesh. In our proposed framework, a two-branch design is adopted to estimate parameters of 1) large-scale body and camera, 2) small-scale 3D hands pose, and facial expression, respectively. Since we lack the paired data to support the directly supervised training of the partial branch, we propose a synthetic training scheme (STS) to help regress hands and face parameters with the unpaired 3D data in a self-supervised manner. Besides, for better generalization and richer representation learning, we supervise the 3D pose of the estimated 3D mesh with 2D pose using the proposed D2S projection. We will introduce the network architecture and main modules of our framework in the following.

As shown in Fig. 3, a two-branch framework is designed to predict the SMPL-X and camera parameters from a 2D image for whole-body 3D mesh recovery. Given 2D images, we use OpenPose [22] and a shape encoder to predict the whole-body 2D keypoints and shape embedding, respectively. Then, to balance the scale differences between different body parts, we develop a scale normalization layer to transform the 2D keypoints of the face, hands, and body into a standardized space. In detail, keypoints of different parts are first translated to their center by subtracting the mean value and then scaled by dividing the half size of their bounding boxes. Then the re-scaled 2D keypoints of body and hands/face are separately put into the two sub-networks for global and partial parameter retrieval.

In global branch, with the re-scaled 2D body keypoints and the extracted shape embedding (i.e, 512-D global feature vector from Resnet-50 [44]), we use the global recovery sub-network to estimate global parameters. In this process, following [7], we employ a bilinear transformation layer to re-organize the global features for better performance. Then, we put the yielded skeleton-disentangled representation into a Multi-Layer Perceptron (MLP) to regress the body shape $\boldsymbol{\beta}$, pose parameters $\boldsymbol{\theta_{b}}$, and the camera parameters $(s,\boldsymbol{R},\boldsymbol{t},d)$. $\boldsymbol{R}\in\mathbb{R}^{3}$ is the 3D rotation of the entire human mesh, $\boldsymbol{t}\in\mathbb{R}^{2}$ and $s\in\mathbb{R}$ are translation and scale in image plane, respectively. $d\in\mathbb{R}$ is a distance factor of D2S projection, which will be introduced in Sec.III-E.

In partial branch, with the re-scaled 2D face and hand keypoints, we use the partial recovery sub-network to predict the small-scale 3D hands pose $\boldsymbol{\theta_{h}}$ and facial expression $\boldsymbol{\psi_{f}}$ parameters. Especially, since both face and hand has the structure prior (e.g., symmetry, connection), we leverage the GCNs to explore their structural and spatial relations. The design of the GCN-based face regressor follows [45] and consists of four semantic graph convolution layers. Similarly, the GCN-based hand regressor estimates hands pose parameters $\boldsymbol{\theta_{h}}$. The only difference is that the face regressor employs an additional MLP to estimate the expression from whole-face features.

With these estimated parameters from both the global and partial branches, we directly connect all these parameters and put into the SMPL-X model described in Sec.III-B to obtain the final 3D human mesh. In other word, the parameter vector $\mathbf{\Theta}=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\psi},s,\boldsymbol{R},\boldsymbol{t},d\}$ is adopted to represent the 3D human whole-body in camera coordinates.

Loss function. The supervision of the proposed framework is conducted in both 3D and 2D to make full use of existing data. The total loss $L_{\mathbf{\Theta}}$ is the weighted sum of parameter $L_{pm}$, 3D keypoints $L_{3D}$, 2D projected keypoints $L_{2D}$ and the rationality $L_{r}$ where $w_{pm},w_{3D},w_{2D},w_{r}$ are weights of these loss items. Since the ground truth parameters are not fully available in most datasets, the supervision is usually carried out on the different aspects of the recovered 3D human mesh. As stated in Eq.1, 3D keypoints can be derived from human mesh via a linear keypoint regressor. is employed to supervise the 3D keypoints of the estimated whole-body mesh using the $L2$ loss.

In 3D level, we employ $L2$ loss $L_{pm}$ and $L_{3D}$ to supervise the estimated parameters and the 3D poses (derived by Eq.1) of all human parts respectively. In detail, $L_{pm}$ is employed to supervise the pose $\boldsymbol{\theta}$ and expression $\boldsymbol{\psi}$. The angular pose is converted from axis-angle representation to rotation matrices via the Rodrigues formula for more stable training [4, 6]. However, due to the inherent scale difference, it is hard to simultaneously capture large torsos and small body parts. Therefore, no dataset has full whole-body 3D annotations. In this case, general supervised training is not feasible. In contrast, there are some single-body-part datasets of 3D body pose, hand gesture, and facial expression available, which are not paired. Therefore, we develop a self-supervised learning scheme (STS) in Sec. III-D to train the partial branch in a learning-from-synthetic manner.

The existed datasets are mostly captured in restricted environments. The model solely trained on them is hard to work well in the wild. To solve this generalization problem, in the 2D level, we project the 3D joints of the estimated 3D mesh onto the 2D image plane using the estimated camera parameters for better generalization. The $L1$ loss $L_{2D}$ is used to make 2D pose supervision. However, the widely used weak-perspective camera model ignores the foreshortening effects in 2D images, which makes it prone to fail when the images are captured closely. Therefore, we propose a novel D2S projection in Sec.III-E to make the projection scale adjustment for each keypoint in terms of the depth difference between them. Finally, following [4], we also supervise the rationality $L_{r}$ of the keypoint angles and whole-body shape for preventing physically impossible 3D human mesh, especially when we use the unpaired body, hands, and face data.

As we stated in Sec. I and Sec. 3.3, lacking complete annotations of whole-body parts makes it hard to directly train the network end-to-end. Therefore, we develop a synthetic training scheme (STS) to supervise the network under this circumstance. The STS is designed to get reasonable synthetic data from the original unpaired 3D data and learn the hands and face recovery in a self-supervised manner.

The illustration of STS is shown in Fig. 4. Firstly, we collect the 3D pose $\boldsymbol{\theta}$, shape $\boldsymbol{\beta}$ and expression $\boldsymbol{\psi_{f}}$ parameters from these unpaired datasets by fitting the statistical models to establish a reasonable parameter bank. Specifically, we get facial expression $\boldsymbol{\psi_{f}}$ by fitting FLAME [8] model to 3D head mesh data [20] or 2D facial keypoints [46]. The body poses are brought from several Mocap datasets [47], and 3D hands poses are brought from MANO datasets [13, 18, 47]. Then, as shown in Fig. 4 (a) we randomly sample these parameters from the bank to form the complete parameter vector $\boldsymbol{\Theta}$ and put it into the SMPL-X model to derive the whole-body 3D mesh. Next (b), we map the joints of the driven 3D whole body mesh back to the 2D image plane to establish the mapping between 3D mesh and 2D pose. In this way, 2D pose along with its complete 3D annotations are generated. With these synthetic paired data, in (c) and (d), we exploit the GCN-based regression network for estimating hand gesture and facial expression respectively from their 2D landmarks. Finally in (e), the face/hand partial recovery sub-network can be trained by self-supervising the consistency between the generated 3D meshes and final regressed 3D mesh. Due to the full 3D annotations are available, we only supervise the $L_{pm}$ and $L_{3D}$.

As we introduced in Sec.I and Sec. 3.3, available 3D datasets [18, 21] are always captured in a constrained environment with the fixed camera configurations. In this case, using in-the-wild 2D pose datasets [48, 49, 50] for training is a good choice for better generalization and adopted by many methods. To supervise the 3D pose of the estimated 3D human mesh with 2D pose, we need a camera model to make 3D-to-2D projection. In previous works [4, 5, 7, 14, 18], weak-perspective projection is widely adopted. Compared with the perspective projection that needs complex camera parameters (e.g. 3D translation $\boldsymbol{T}=(T_{x},T_{y},d)\in\mathbb{R}^{3}$, focal length $\boldsymbol{f}=(f_{x},f_{y})\in\mathbb{R}^{2}$, et al.), the weak-perspective one is much more simple. It only composes of two factors, scale $s\in\mathbb{R}$ and 2D translation $\boldsymbol{t}=(t_{u},t_{v})\in\mathbb{R}^{2}$ in image plane. The 2D projection $J^{p2D}_{i}=(u_{i},v_{i})$ of the $i$-th 3D keypoint $J_{i}=(x_{i},y_{i},z_{i})$ (takes the body gravity center as the origin) can be simply derived in a weak-perspective manner using Eq.(6) and Eq.(7) while its perspective projection can be derived using Eq.(3) and Eq.(4).

We can notice that the weak-perspective projection shares the same scale $s$ among all keypoints, while the standard perspective projection provides an individual scale $\frac{f_{x}(x_{i})}{d+z_{i}}$ for each keypoint. As shown in Fig.5, the weak-perspective projection ignores the difference induced by the foreshortening effect and results in 3D-to-2D misalignment. In this case, even if the 3D pose of the estimated human mesh is correct, the weak-perspective projection may lead to the wrong 2D pose supervision. Therefore, we develop a novel D2S projection to provide an individual projection scale for each keypoint w.r.t their depth difference. To unify the different representations, we assume that $s$ and $t_{u}$ are approximating the projection factor of a virtual body center whose $z_{i}=0$. In this way, the approximation relationship between the perspective and the weak-perspective projection can be represented as

Based on this assumption, the difference ($\Delta s,\Delta t_{u}$) between the real and the approximate factor can be derived as

From Eq.(8) and Eq.(9), we can know that, when $d$ is significantly larger than $z_{i}$, the difference between the perspective and the weak-perspective projection is small. Otherwise, the difference is obvious. In other words, when a person is close to the camera, the weak-perspective camera is prone to fail especially for poses with a large $z_{i}$. The difference is mainly brought by the depth $z_{i}$. Therefore, we design a D2S projection that converts the depth differences to projection scale variances among body parts. The goal is to make up the $\Delta s$ and $\Delta t_{u}$, meanwhile, reduce the parameters that need to be estimated.

Instead of using a unified projection scale $\boldsymbol{s}$ and translation $t_{u}$ for all keypoints, the individual scale factor $\boldsymbol{s_{i}}$ for the $i$-th keypoint is estimated to re-scale the output of weak-perspective projection. So the coordinate $u_{i}$ of the projected 2D keypoint $J^{2d}_{i}$ can be derived as

from which we can see that $d$ is a key coefficient converting the per-joint depth differences into the projection scale variation. In our framework, $d$ is estimated as a part of the camera parameters. Compared with the weak-perspective projection, we just need to estimate one more factor $d$. For better generalization on in-the-wild images, large-scale 2D pose datasets [49, 50] are used to learn the estimation of $d$. The D2S projection works better in the scenarios whose depth difference between different parts is obvious and the field of view is large. For example, taking a selfie with a handheld camera. On the contrary, when people are far away from the camera, it degrades to the weak-perspective model.

We firstly introduce the three benchmarks for experiment evaluation. We also introduce several datasets that we put together to establish the real distribution of whole-body parts for model training.

CMU Panoptic Studio Dataset: We use the “Monocular Mocap” part [14] that contains multi-view images (taken from 30 viewpoints) with 3D pose annotations. It contains $40$ subjects and $834K$ body images with corresponding 3D body pose annotations. Following [14], we only focus on the evaluation of 3D body pose in this paper. We report both mean per joint position error (MPJPE) and Procrustes Aligned MPJPE (PA-MPJPE) that is MPJPE after rigid alignment of the predicted pose with the ground truth.

Human3.6M Dataset: It is a large-scale human MoCap dataset  [21] with 3D pose annotations and has been widely used as the 3D pose benchmark. It contains videos of 7 actors performing 17 activities. Videos are captured in a controlled environment. For removing the redundancy, we down-sample all videos from 50fps to 10fps. We use the dataset to quantitatively evaluate the 3D body keypoint error of the recovered mesh and follow the same protocol as in [4] for evaluation. We evaluate both MPJPE and PA-MPJPE of the 3D body keypoints.

STB Hand Dataset: Stereo Hand Pose Tracking Benchmark (STB) [17] is a 3D hand pose dataset, containing 18,000 images with corresponding depth images. The 3D hand annotations involve 21 keypoints locations. Following [11], we split it into 15,000 samples for training and 3,000 samples for testing. Besides, for a fair comparison, we use Openpose [22] for bounding box detection. We measure the MPJPE for comparison.

Datasets for STS and Model Generalization: For rationally sampling the SMPL-X model space, we use multiple 3D datasets to obtain the real whole-body parameters. 1) 3D body poses are obtained from the AMASS dataset [47]. It contains 300 subjects, more than 11000 motions (with corresponding SMPL 3D pose parameters), which is significantly richer than existing 3D pose datasets. 2) 3D hand poses are obtained from the MANO [13], Freihand [18], and AMASS [47]. MANO and AMASS contain the 3D hand pose parameters. Freihand contains image data along with the 3D hand annotations. We split their training set into two sets for training (30000 samples) and validation (2560 samples) respectively. 3) Facial expression are extracted from D3DFACS 3D face mesh dataset [20] and LS3D [46] 2D facial dataset. Besides, we employ two in-the-wild 2D pose datasets, AI-CH [50] and MPII [49], to provide 2D pose supervision. Different from the 3D datasets captured in limited environments, they can greatly help the model generalize to various poses and camera configurations.

Considering that only a part of the data has 3D hand pose annotations, the STS is used one time in every 5 iterations. The Adam [51] optimizer is adopted with betas=(0.9,0.999), momentum=0.9. The learning rate and batch size are set to 1e-4 and 16 respectively. The weights of loss items are set as $w_{pm}=20,w_{r}=0.5,w_{2D}=6,w_{3D}=60$. Training of the partial parameter retrieval branch consists of two phases. Firstly, we train the face and hand GCN-based regressor with the assistance of the FLAME head model and MANO hand model respectively. Then, pre-trained models are loaded and the entire branch is trained by STS.

In this section, we introduce the comprehensive ablation study of D2S projection on the CMU Panoptic Studio dataset. To validate the superior of the proposed D2S projection under an uncalibrated condition, we compare it with other camera models, which are the perspective projection (PP) and the weak-perspective projection (WPP) introduced in Sec. III-B. For a fair comparison, the model is trained to estimate the camera parameters of these projection models based on the 2D pose only without using the ground truth parameters. Due to the complete camera parameters of perspective projection are too complex to predict, we simplify the parameters and estimate the $\frac{f_{x}+f_{y}}{2},t^{c}$ in Eq.(3) and Eq.(4). For WPP, we just estimate the $s,t_{u},t_{v}$ in Eq.(6) and Eq.(7).

As shown in Tab. I, PP achieves comparative results with MTC, while greatly outperformed by the proposed D2S projection 8.6mm MPJPE and 4.1mm PA-MPJPE, corresponding to $13.9\%$ and $7.7\%$ error reduction. Especially, the training of the proposed D2S projection is much more stable than the PP, which is prone to collapse. WPP is outperformed by 4.3mm MPJPE and 2.1mm PA-MPJPE, corresponding to $7.4\%$ and $5\%$ error reduction. We think the reason is that PP is too hard to estimate from a single monocular image, while the WPP is so rough that may misguide the model. In contrast, the proposed D2S projection finds a good balance of estimation difficulty and projection accuracy for proper supervision. It demonstrates that the proposed D2S projection is more suitable for the un-calibrated condition in this task.

Moreover, since the Panoptic Studio Dataset contains multi-view images, we further evaluate the MPJPE and PA-MPJPE in different viewpoints. As shown in Fig. 6, the recovery error of the WPP and PP dramatically changes along with the viewpoint. The proposed D2S projection can alleviate this phenomenon to some degree, demonstrating that the proposed D2S projection is effective to make up the depth loss of weak-perspective projection.

In Tab.IV, we compare the computational efficiency with the most related works SMPLify-X [15] and MTC [14]. For a fair comparison, we directly use the released official code and test on the same dataset [19] on the same device (Tesla P40 GPU). Considering that both SMPLify-X and our method employ the OpenPose to estimate the whole-body 2D keypoints, we only compare the time consuming of predicting the whole-body 3D mesh from a single 2D image and the corresponding OpenPose output. SMPLify-X takes 82.72 seconds per image, while the proposed single-step framework only consumes 0.09 seconds per image. The time consumption is mainly costed by the 5-stage iterative optimization strategy in their methods, which demonstrate our motivation. Based on their official code, MTC costs about 12 seconds per image. The single-step design enables us to enjoy parallel computation, which greatly accelerates the calculation. In contrast, the proposed single-step method can usually be dozens or hundreds of times faster than the existing optimization-based methods.

We present qualitative results of our method on images for the whole-body mesh recovery and the face/hand partial recovery. To test the effectiveness and the stability of the proposed method (STS and D2S projection), we evaluate our model on internet images. The results of MTC [14], SMPLify-X [15] and the proposed method are presented in Fig.1. MTC and SMPLify-X can indeed align the human body in 2D images better, but their performance in depth and shape estimation is difficult to satisfy. Especially, when some body parts are invisible (such as the invisible feet at the first raw of Fig.1), the algorithm is prone to fail. The estimated body shape is weird and not coordinated. In contrast, the estimated body pose, shape, and expression of our method are much better. Benefited from the D2S projection, our method shows some prior in perceiving depth information. Considering most 3D datasets are captured in a close range, we deliberately select some videos where people are far from the camera. The results are presented in Fig.8. It demonstrates that the proposed D2S projection can handle a wide range of human-to-camera distance. Besides, qualitative results of face and hands are shown in Fig. 7. The top row presents the recovered meshes on face data for both in indoor images from [14] and in-the-wild images. The second and third rows show the recovered hand meshes on the Freihand dataset [18] and the STB dataset [17], respectively. These results demonstrate the effectiveness of the proposed method.

The proposed method successfully reduces the network’s dependence on data and improves computational efficiency. However, there are still some limitations.

Reliance on 2D pose estimation. The performance relies on the quality of the 2D pose estimation. A stronger 2D pose estimation model will lead to better performance. Also, the representation learning of the body shape needs to be further enhanced. Besides, self-occlusion, such as hands overlapping, also affects the performance. We will investigate those problems in our further work.

About face shape. Besides, due to the limitation of the SMPL-X model, we only estimate the facial expression instead of the 3D face mesh. Because the face shape of SMPL-X is changed along with the body shape and cannot be controlled individually, which makes it hard to estimate the precise facial mesh. Therefore, we show the visualization results other than evaluating the facial keypoints. We will dig into developing a whole-body statistical model whose partial mesh could be preciously controlled.