Weakly-supervised Pre-training for 3D Human Pose Estimation via Perspective Knowledge

Some works [16, 10, 41, 42] have studied the mixed training of 2D and 3D human pose datasets. Zhou et al. [16] a weakly-supervised transfer learning network with a 2D pose estimation sub-network and a 3D depth regression sub-network, which uses mixed 2D and 3D labels. Sun et al. [10] proposed a unified framework to mix 2D and 3D labels by integral regression on 3D heatmaps. Yang et al. [42] proposed an adversarial learning framework, which distills the 3D human pose structures learned from the fully annotated dataset to in-the-wild images with only 2D pose annotations. Habibie et al. [41] introduced a network architecture that comprises a new disentangled hidden space encoding of explicit 2D and 3D features and uses supervision by a new learned projection model from a predicted 3D pose. These methods have extensively verified that 2D labels can improve the accuracy of 3D pose estimation. However, these methods just extract 2D information from images and 2D labels to enhance the constraint of the 3D pose, which limits the role of 2D labels and images. In this paper, We directly extract 3D information from 2D labels and images to improve the performance of 3D human pose estimation.

Some approaches try to generate more training images through data augmentation [21, 43, 19, 8]. Varol et al. [21] propose to generate more training images by computer graphics rendering. Rogez et al. [43] propose mocap-guided data augmentation for 3D human pose estimation in the wild. Pishchulin et al. [19] fitted 3D models to humans and deformed them to generate new images in multiple views. Mehta et al. [8] tried to generate more images by changing the background and clothing in existing images. Although these methods improve the performance of the 3D human pose estimation model by generating more training images, the fundamental problem has not been solved since the gap between synthesized images and real images still exists.

Other works try to solve this problem by weakly supervised methods [13, 22, 23]. Kocabas et al. [13] introduces a self-supervised learning method to train a 3D pose model by generating 3D pose labels according to multi-view information and camera parameters. Chen et al. [22] proposes to learn a geometry representation from multi-view information. The learned representation could map to a 3D pose with a shallow network and less annotated training samples. Wandt et al. [23] proposes an adversarial reprojection network to train the 3D pose model. They predict 3D pose and camera parameters at the same time, then project the 3D pose to 2D pose by camera information. Although these methods reduce the reliance on 3D pose annotations, the performance gap between weakly supervised methods and fully supervised methods is huge. Meanwhile, camera parameters and multi-view formation can’t be provided in existing monocular images, which limits the application of weakly supervised methods.

WSP estimates the relative depths between human joints from a cropped image with two-person pair, which is shown in Figure LABEL:fig:f3. WSP consists of three parts: backbone, relative depth head, and 2D keypoints heatmap head.

The first one, backbone, extracts deep features from the input image by using ResNet. Based on the extracted deep features, the second part, the 2D keypoints heatmap head predict the keypoints heatmap by 3 deconvolution layers. Different from other methods to obtain 2D coordinates from 2D heatmaps, the estimated heatmaps here are used as masks to filter deep features for special human joints.

The relative depth head estimates the relative depths of human joints when given the deep features of two points. The inputs of relative depth head are deep features filtered by keypoints heatmaps. Deep features $F_{d}$ has a size of (D, H, W), and 2D heatmap $M$ has a size of (J, H, W). D is the channel number. H and W are the height and width of heatmaps. J represents J joints. The keypoints deep features are extracted as follows

where $j$ denotes $j$-th joint. $repeat$ means the operation of repeating $M^{j}$ of size $H\times W$ into size $D\times H\times W$. $\otimes$ denotes element-wise multiplication. We ignore other persons who may are cropped in this image patch and their relative depths are estimated in other image patches. $M^{j}$ represents the 2D heatmaps for the $j$-th keypoint. $F^{j}$ represents the filtered deep features for the $j$-th keypoint. Then, for each person, $F^{j}$ is extracted into a vector of size $D$ by the pooling layer within the bounding box of this person. Thus, the extracted deep features have a size of (J, D).

Then, we randomly select $N~{}(N<J)$ keypoints for each person to estimate their relative depths. We use a fully connected network to classify the relative depths of selected keypoints. As shown in Figure 2, for the person $A$ and person $B$ in the input image, we firstly obtain the filtered deep feature $F(P_{A})$ and $F(P_{B})$ by Equation 5, which have a size of (J, D). After randomly selecting, we get $F_{s}(P_{A})$ and $F_{s}(P_{B})$, which have a size of (N, D). For each randomly selected keypoint, we concatenated $F_{s}^{j}(P_{A})$ and $F_{s}^{j}(P_{B})$ and send them into the fully connected network to estimate their relative depths. The predicted relative depths can be denoted as

where $\oplus$ represents concatenate operation. $f(\cdot)$ represents fully connected network of input dimension $2\times D$ and output dimension $1$. $R_{AB}^{j}$ represents the estimated relative depths for $j$-th joint in person $A$ and person $B$.

Given an image patch $I_{AB}$ cropped from original image $I_{2D}$, WSP estimates the relative depths $R_{AB}^{j}$ for each joints in person $A$ and $B$ according to the Equation 5 and 6. The ground-truth of relative depths is $\hat{R}_{AB}^{j}$, which can be generated according to Equation 1, 2, 3 and 4. To numerical represent relative depths, we define that

where $P_{A}^{j}$, $P_{B}^{j}$ and $Z(\cdot)$ are the $j$-th joint in person $A$, the $j$-th joint in person $B$ and the operation to get depths, respectively.

We use cross-entropy loss and $L_{2}$ loss for relative depth learning and heatmap learning, respectively. The total training loss is

where $\mathcal{L}_{hm}$, $\mathcal{L}_{rd}$ and $\alpha$ are the loss for keypoints heatmap, keypoints relative depth, and loss weight, respectively.

$\mathcal{L}_{rd}$ is cross-entropy loss, and

where $J$, $\hat{R}^{j}$, and $R^{j}$ are the number of human joints, the ground-truth of relative depths for $j$-th joint, and the relative depth estimated by WSP for $j$-th joint, respectively.

$\mathcal{L}_{hm}$ is $L_{2}$ loss, and

where $J$, $K$, $M_{k}^{j}$ and $\hat{M}_{k}^{j}$ are the number of human joints, the number of estimated persons in the input image, the predicted keypoints heatmaps, and the ground-truth of keypoints heatmaps for $j$-th joint in person $k$, respectively.

We finetune WSP on the 3D human pose estimation dataset. For 3D human pose estimation, as [10, 13, 11], we add a head to learn 3D keypoints heatmap. The 3D coordinates can be obtained by integrating 3D heatmaps. The integrating operation to get $x$ values of 3D coordinates can be formulated as

where $p$ represents the points in 3D heatmap $\hat{M}$. $C_{x}^{j}$ is the $x$-axis values of 3D coordinates for $j$-th joints. $C_{y}^{j}$ and $C_{z}^{j}$ are obtained also according to the Equation 11 by changing the order of integrating.

Following the suggestion from [10], we use $L1$ loss to optimize the errors of coordinates. The training loss of finetuning WSP is

where $J$, $C^{j}$ and $\hat{C}^{j}$ are the numbers of human joints, the estimated coordinates by our network and the ground-truth coordinates, respectively.

The images in MCPC dataset for training are all collected from the train set in MPII, COCO, PoseTrack, and CrowdPose dataset. The train set of MCPC contains over 53k images with more than 220k persons. The images in the test set of MCPC are all from the test set in these four 2D datasets. The test set of MCPC contains over 11k images with more than 18k two-person pairs.

For relative depth pre-training, we evaluate the predicted relative depths of human joints on the test set of the MCPC dataset. Since it is formulated as a classification task, we use AUC, Accuracy, Precision, Recall, and F1 score to evaluate the predicted relative depths. The threshold for accuracy, precision, recall, and F1 score is 0.5.

Human3.6M [7] is the largest and most widely-used benchmark for 3D human pose estimation. This dataset is captured by the MoCap system in the lab environment. It consists of 3.6 million videos frames from 4 cameras’ viewpoints. 11 subjects perform 15 activities, such as discussion, eating, etc. The appearance of the human body and the background of the images is simple since the limited subjects and lab environment. Accurate 3D annotations of the human pose are generated by a motion capture system in each frame.

For evaluation, we use mean per joint position error (MPJPE), which is widely used in previous works. Some works also use another widely-used metric, PA MPJPE. They firstly align the predicted 3D pose and ground truth 3D pose with a rigid transformation using Procrustes Analysis and then compute MPJPE, which is called PA MPJPE.

MuCo-3DHP and MuPoTS-3D Datasets are 3D multi-person pose estimation datasets proposed by Mehta et al. [9]. MuCo-3DHP is the training set, which is generated by compositing the existing MPI-INF-3DHP 3D single-person pose estimation dataset [8]. The testing set, MuPoTS-3D dataset is captured outdoors and includes 20 real-world scenes with ground-truth 3D poses for up to three subjects. The 3D annotations of the MuPoTS-3D dataset are obtained with a multi-view marker-less motion capture system.

For evaluation, 3DPCK, a 3D percentage of correct keypoints is used after alignment with ground truth. It treats a joint’s prediction as correct if it lies within 15cm from the ground-truth joint location. For more rigorous testing, we also additionally use 3DPCK with thresholds of 10cm, 11cm, 12cm, 13cm, and 14cm, respectively.

We formulate the task of estimating relative depths as distinguishing which human joint is closer to the camera. When training, for each image in mini-batch, we randomly select two persons in this image to estimate their relative depths of human joints. Due to the different scales of the two-person pairs, we define that

where $S$ denotes the scale of the person. $P_{ht}$ and $P_{hd}$ are the head_top joint and head_down joint in the person, respectively.

For person $A$ and person $B$, the normalized scale gap ratio $\Delta S$ is denoted as

where $S_{A}$, $S_{B}$ are the scales of person $A$ and person $B$ generated by Equation 13. $0\leq\Delta S\leq 1$.

In order to fully verify that WSP has learned the knowledge of estimating relative depths, we define two protocols. RD protocol 1: we evaluate the estimated relative depths in different scale gap ratios ($\Delta S=1.0,0.5,0.3,0.1$). The smaller $\Delta S$ indicates that the person $A$ and person $B$ have similar scales, which means it’s more difficult to estimate their relative depths. RD Protocol 2: Removing the human ankle parts in the input image for testing. RD Protocol 2 is designed to study if WSP still can distinguish the relative depth without the ankle clues.

Three experimental protocols are used. Following previous works [11, 10], the first one uses five subjects (S1, S5, S6, S7, S8) as training data and the other two subjects (S9, S11) as testing data. We denote it as the Subject Protocol 1, which uses MPJPE as the evaluation metric. The second one uses six subjects (S1, S5, S6, S7, S8, S9) as the training data and the other one subject (S11) as testing data. We denote it as the Subject Protocol 2, which uses PA MPJPE as the evaluation metric. To evaluate the generalization of 3D pose estimation model, we also use another widely-used protocol following [35, 15], called Cross Action Protocol, which uses MPJPE as the evaluation metric. Cross Action Protocol trains on only one of the 15 actions in the Human3.6M dataset and tests on all actions, which is designed to test the generalization ability of the 3D pose model.

Following the previous work [11], we use 400k frames composited from the MuCo-3DHP dataset for training, in which half of the images are background augmented. As the setting of previous works, we also use the COCO dataset for augmentation. When training models, each mini-batch consists of half MuCo-3DHP and half COCO images. We use the strategy of auxiliary training as [10, 11] since the COCO dataset has no annotations of the 3D pose.

We use ResNet pre-trained on ImageNet as the backbone of WSP. Then, we train WSP on the MCPC dataset. We summarize the hyper-parameters as follows for pre-training: the loss weight $\alpha$ (in Equation 8) equals 50. The input image size is $256\times 256$ and the heatmap size is $64\times 64$. The batch size is 64 and the initiating learning rate is 0.001. The learning rate is decay to 0.0001 and 0.00001 at $30$th epoch and $40$th epoch, respectively. The total epoch number for training is 50. We conduct the experiments using PyTorch on the Linux platform, and 2 V100 GPUs are used. It cost about 50 hours to train WSP on the MCPC dataset.

We use WSP pre-trained on the MCPC dataset as the backbone network for 3D human pose estimation. We remove the original heads of WSP and add an up-sampling head to estimate the 3D human pose. We finetune the network on Human3.6M and MuCo-3DHP datasets, respectively. We summarize the hyper-parameters as follows for WSP finetuning: The input image size is $256\times 256$ and the heatmap size is $64\times 64$. The batch size is 64 and the initiating learning rate is 0.001. The learning rate is decay to 0.0001 and 0.00001 at $17$th epoch and $21$th epoch, respectively. The total epoch numbers for training is 25. We conduct the experiments using PyTorch on the Linux platform, and 2 V100 GPUs are used. It cost about 8 hours and 20 hours to finetune WSP on Human3.6M and MuCo-3DHP datasets, respectively.

We formulate the pre-training task as a classification task, which aims to learn to distinguish the relative depth relationship between two human joints. After training WSP on the MCPC dataset, we test WSP on the test set of the MCPC dataset. The results of classification are shown in Table 2 and Table 3.

In Table 2, WSP-A, WSP-B, WSP-C, and WSP-D represent different test settings. All of the settings are WSP with the ResNet-50 backbone. The test images are cropped from the images in the testing set of MCPC. The cropped image includes two persons, which are the targets to distinguish relative depth relationships. Since the scales of the two human bodies in the image may be very different, we define the scale gap ratio $\Delta S$ to represent this scale gap. WSP-A, $\Delta S\leq 1.0$ means that all the cropped images are used for testing, regardless of the huge scale gap between the two persons in this cropped image. WSP-B, $\Delta S\leq 0.5$ means that only a part of cropped images is used for testing. The scale gap of two persons in the cropped image must be less than or equal to 0.5, which means the testing images are more difficult to distinguish relative depth relationships. With the same knowledge, WSP-C ($\Delta S\leq 0.3$) and WSP-D ($\Delta S\leq 0.1$) mean the smaller and more difficult testing set.

WSP-A obtains 95.39% in AUC. The values of accuracy, precision, recall, and F1 score are all over 88%. These results illustrate that WSP can distinguish the relative depth relationship of keypoints when given an image with two persons. WSP-B, WSP-C, and WSP-D are designed to study the influence of the scale. The smaller $\Delta S$ means that the two persons in testing images are with more similar scales. As shown in the results in Table 2, all the evaluation metrics decrease with the smaller scale gap ratio $\Delta S$, which verifies that scale is an important visual cue to distinguish relative depth. However, although we just keep the hardest cases for evaluation in WSP-D, we still can obtain over 90% in AUC and over 81% in accuracy, precision, recall, and F1 score, respectively. The results show that WSP can estimate relative depths successfully when two persons have almost the same scales.

Due to the relative depth supervision being generated according to the ankle cues, we study the influence of ankle cues in Table 3. RA in Table 3 means removing the ankle parts in the images for testing. WSP-E, WSP-F, WSP-G, and WSP-H are designed to that WSP still can estimate relative depths successfully without the guiding of ankle clues. Although there are no ankle joints in the input images, for the testing images with almost the same scales, WSP-H still can obtain over 87% in AUC and over 79% in accuracy, precision, recall, and F1 score, respectively. The results show that the accuracy of estimating relative depths is influenced by ankle joints, but WSP can still have a good estimation of relative depths even without the clues of ankle joints.

To verify that WSP is beneficial to 3D human pose estimation, we finetune WSP on the Human3.6M dataset. The baseline is PoseNet [10, 13, 11], which is a widely-used network for 3D human pose estimation. PoseNet and WSP are the same networks, the difference is that WSP is pre-trained on the MCPC dataset. As shown in Table 4, PoseNet achieves 67.5 in MPJPE of action average based on ResNet-50. WSP achieves 54.3 in MPJPE and a relative gain of 20% with the pre-trained weights.

In order to further analyze the influence of pre-training weights, we make a comparison of PoseNet and WSP on MPJPE of joints average. We can find that the MPJPE of joints averages on $Z$ coordinates are obviously greater than the MPJPE of joints average on $X$ and $Y$ coordinates from Table 4. Here, we denote them as $MPJPE_{X}$, $MPJPE_{Y}$, and $MPJPE_{Z}$, respectively. $MPJPE_{Z}$ of PoseNet are greater than 50mm, while $MPJPE_{X}$ and $MPJPE_{Y}$ of PoseNet are about 23mm. The phenomenon shows that the error of depth predicting is the key problem to improve the performance of 3D human pose estimation. However, WSP obtains 40.2 in $MPJPE_{Z}$ of action average based on ResNet-50. Compared with PoseNet, the $MPJPE_{Z}$ decay to 40mm from 50mm, which verifies that the pre-trained WSP is beneficial to 3D pose estimation, especially in the depth prediction. We believe that the relative depth pre-training on MCPC learns how to distinguish the relative depth, which improves 3D pose estimation.

Since we use the MCPC dataset for the pre-training of WSP, to verify the improvements in Section 4.4.2 come from more data or our pre-training strategy, we make the comparison of mixing training strategy (MTS) and WSP in this section.

MTS is adding 2D datasets with 3D datasets for mixing training. MTS just gives 2D supervision when using images in 2D datasets while 3D supervision when using images in 3D datasets. Following [10, 13, 11], we also finetune WSP on Human3.6M and MCPC datasets to ensure that WSP and MTS are trained on the same data for a fair comparison. The results are shown in Table 4. PoseNet achieves 52.9 MPJPE based on ResNet-50. WSP achieves 48.7 in MPJPE based on ResNet-50. Compared with PoseNet, the relative improvement is 6%. The $MPJPE_{Z}$ of WSP decays to 35.5mm with ResNet-50, which has an improvement of 11% in estimating depths compared with 39.9mm of PoseNet. The MTS tries to improve the performance of predicting 3D pose by adding the constraint of the 2D pose, which to further enhance the structural constraint of the 3D pose. However, WSP tries to directly learn weak 3D representation from 2D datasets, which significantly improves the accuracy of depth prediction for 3D human pose estimation as the results are shown in Table 4.

Therefore, the WSP improves the performance of 3D pose estimation since the relative depth pre-training.

We compare our proposed system with the state-of-the-art 3D human pose estimation methods on the Human3.6M dataset. Following the previous works [11], WSP is finetuned on Human3.6M dataset based ResNet-50. For evaluation on Subject Protocol 1, as shown in Table 5, WSP obtains an action average MPJPE of 43.2mm, which is the state-of-the-art result for fair comparison. For evaluation on Subject Protocol 2, as shown in Table 5, WSP obtains an action average PA MPJPE of 28.2mm, which is the state-of-the-art result for fair comparison.

We compare our approach with state-of-the-art 3D human pose estimation methods on MuPoTS-3D Datasets in Table 6. MuCo-3DHP and MuPoTS-3D Datasets are outdoor multi-person 3D human pose estimation dataset. The evaluation metric is a 3D percentage of correct keypoints. Due to the multiple persons, the accuracy is computed for all ground-truth keypoints or ground-truth keypoints in matched persons. As shown in Table 6, WSP outperforms state-of-the-art methods in 3DPCK with a threshold of 150mm.

To further study the effectiveness of WSP, we compare the results of WSP and PoseNet [11] on MuPoTS-3D datasets with more strict thresholds (100mm, 110mm, 120mm, 130mm, 140mm, and 150mm) since 150mm is a broad threshold. The results are shown in Table 7. The testing threshold decreases to 100mm from 150mm, and the 3DPCK of PoseNet and WSP decreases. However, for all-person test settings, compared with PoseNet, the relative improvement of WSP increases to 3.3% from 1.4% with the more strict thresholds. For matched-person settings, the relative improvement of WSP increases to 2.7% from 0.9% with the more strict thresholds.