Graph-PCNN: Two Stage Human Pose Estimation with Graph Pose Refinement

Locating human joint based on the peak of heatmap is frequently-used in modern human pose estimators, and they modeled the target of heatmap by generating gaussian distribution around the ground truth. But due to the complex image context and human action, the joint heat may not be satisfy gaussian distribution strictly which, together with quantisation affect of image resolution downsampling, leads to an insufficient precision of this localization method. However, the peak of heatmap is always close to the true location of joint, which make it adequate to regress the true location.

To achieve the goal of obtaining refined coordinate based on the peak of heatmap, we sample several guided points and train coordinate refinement in stage2. Concretely, we equally sample three kinds of guided points for training: (a) points which are close to the ground truth keypoint, (b) points which are far away from the ground truth keypoint, and (c) points which have high heat response. And the $k$th ground truth keypoint is denoted as $\mathbf{t}_{k}$. As exhibited in Fig. 3, (a) and (b) are randomly sampled within the red region and blue region, respectively, and the red region which centered at ground truth has a radius of $3\sigma$, where $\sigma$ is same with the standard deviation for generating gaussian heatmap target. (c) is randomly sampled from the top $N$ highest response points at the heatmap.

Due to the different characterization of different keypoints, we sample guided points for each keypoint individually, and the total amount of the three kinds of guided points for each keypoint is set equally to $N$.

After the $N$ guided points $\{\mathbf{s}^{1}_{k},\mathbf{s}^{2}_{k},...,\mathbf{s}^{N}_{k}\}$ are sampled, we divide them into two sets, positive set and negative set, denoted as

and $N_{k}^{+}=|\mathcal{S}_{k}^{+}|$, $N_{k}^{-}=|\mathcal{S}_{k}^{-}|$. Then all of the corresponding guided feature extracted from $\mathbf{F}$ by means of bilinear interpolation are feeded into stage2 for refinement while only the guided points from positive set contributed to the coordinate regression.

According to the above label assignment manner, (a) and (b) are definite positive and negative samples, and the influence of proportion between them will be explored in Section 4.3. While (c) is almost negative samples during the beginning stage of training and turns to positive samples as the training schedule goes on. We suppose that (c) can not only accelerate the feature learning at the beginning of training, because (c) are hard negative samples for classification at this stage, but also contribute to the learning of regression when the classification status of feature is relatively stable, as (c) are almost positive samples at this period. Further more, (c) is not necessarily positive when the model converges roughly because of some prediction error caused by hard situation. In this circumstances, (c) can also be regarded as hard negative samples for helping the model to be trained better.

In most of previous works, many fields have been well studied for human pose estimation, such as network structure, data preprocessing and postprocessing, post refinement, etc. However, in these works, the localization of human keypoints are conducted independently for each keypoint while the relationship between different keypoints is ignored all along. Intuitively, the human keypoints construct a salient graph structure base on the pattern of human body, and they have clear adjacent relation with each other. So we consider that the localization of keypoints can be infered better with the help of the information hinted by this relationship. For instance, in our framework, if we know that a guided point is left elbow, then the positive guided points of left wrist should tend to have higher response on left wrist a priori, as left wrist is adjacent to left elbow. So that more supervision can be imposed upon the feature of these keypoints than treating them independently.

To take advantage of the information implicit in the graph structure mentioned above, we propose a graph pose refinement module to model it, and then refine the feature of these keypoints. As shown in Fig. 4, we build a graph and conduct graph convolution for each keypoint. The output embedding feature can be computed by

where $\mathcal{N}(k)$ represents for a point set containing the guided point $\mathbf{s}_{k}$ and its neighbours, $\mathcal{T}_{k^{{}^{\prime}}k}$ for the linear transformation from guided point $\mathbf{s}_{k^{{}^{\prime}}}$ to $\mathbf{s}_{k}$, and $\mathbbm{1}$ for the indicator function. $Z_{k}=\sum_{\mathbf{s}_{k^{{}^{\prime}}}\in\mathcal{N}(k)}\omega_{k^{{}^{\prime}}}$ is used for normalization. $R_{k^{{}^{\prime}}}$ is a boolean type parameter encoding the reliability of a guided point which works for filtering out points of low quality, and its definition will be explained in detail in section 3.4.

Specially, as defined in (7), this graph convolution is an extension of traditional graph convolution, it is designed by considering the characteristic of pose estimation problem. Firstly, we add a weight for each message generated from $\mathbf{s}_{k}^{{}^{\prime}}$ to $\mathbf{s}_{k}$, which can control the contribution of each message according to the intensity and reliability of $\mathbf{s}_{k}^{{}^{\prime}}$. With the constraint of these weights, the graph convolution can be trained more stably. Further more, we set $\omega_{k^{{}^{\prime}}}=1$ when $k^{{}^{\prime}}=k$. And this can make the graph convolution degrading to a traditional linear transformation for $\mathbf{s}_{k}$ when $\mathbbm{1}(R_{k^{{}^{\prime}}})=0$ for all $\mathbf{s}_{k^{{}^{\prime}}}\in\mathcal{N}(k)$ where $k^{{}^{\prime}}\neq k$, without being affected by the intensity and reliability of $\mathbf{s}_{k}$ itself.

After the refinement module above, the embedded feature is sent to a module containing several fully connected layers and batch norm layers, as illustrated in Fig. 4. Finally two predictions are outputed, denoted as $\mathbf{c}_{k}$ and $\mathbf{r}_{k}$, for classification and regression respectively. Giving ground truth keypoint location $\mathbf{t}_{k}$, the losses for these two branches are defined as

and

where $\mathcal{L}^{cls}$ and $\mathcal{L}^{reg}$ are softmax cross-entropy loss and L1 loss. The total loss of the stage2, can be expressed as

where $\gamma_{k}$ is the target weight of keypoint $k$. And $\lambda$ is a loss weight which is set to 16 constantly. And the total loss of Graph-PCNN is

where $L^{s_{1}}$ is the traditional heatmap regression loss for stage1.

In this paper, we use the most popular human pose estimation dataset, COCO. The COCO keypoint dataset [24] presents challenging images with multi-person pose of various body scales and occlusion patterns in unconstrained environments. It contains 200,000 images and 250,000 person samples. Each person instance is labelled with 17 joints. We train our models on train2017(includes 57K images and 150K person instances) with no extra data, and conduct ablation study on val2017. Then we test our models on test-dev for comparison with the state-of-the-art methods. In evaluation, we use the metric of Object Keypoint Similarity (OKS) for COCO to report the model performance.

For fair comparison, we follow the same training configuration as [42] and [33] for ResNet and HRNet respectively. To construct the localization subnet, we copy the conv5 stage, which spatial size is 1/32 to the input size, and the last three deconvolution layers for ResNet series networks, while copying the stage4, which has three high resolution modules, for HRNet series networks. For ablation study, we also add 128x96 input size in our experiment following [49]. And we set $N$ as 48, 192 and 432 corresponding to the three input sizes of 128x96, 256x192 and 384x288 during all of our experiment except the ablation study of $N$. During inference, we use person detectors of AP 56.4 and 60.9 for COCO val2017 and test-dev respectively, while for pose estimation, we evaluate single model and only use flipping test strategy for testing argumentation.

We use ResNet-50 backbone to perform ablation study on COCO val2017.

DARK[49] is a state-of-the-art method which improved traditional decoding by a more precise refinement based on Taylor-expansion. We follow the training settings of DARK and compare our refinement results with it. From Table 3 we can observe that our Graph-PCNN generally outperforms DARK over different network architecture and input size. This suggests that regression based refinement predicts coordinate more precise than analyzing the distribution of response signal from heatmap, as the response signal itself may not satisfy gaussian distribution strictly because of complex human pose and image context while regression is regardless of these drawback.

PoseFix[26] is a model-agnostic method which refines a existing pose result from any other method by a independent model. A coarse-to-fine coordinate estimation schedule ended by coordinate calculation following integral loss[35] is used to enhance the precision. We conduct comparison with PoseFix by using same backbone and input size with its model from refinement stage and the performance of human detectors for these two methods are comparable, AP 55.3 vs 56.4 for PoseFix(using CPN) and our Graph-PCNN respectively. As illustraed in Table 4, we achieve a competable result with PoseFix, but PoseFix included input from CPN which need an extra R50 network while our method only need an extra R50 conv5 stage as refinement branch.

We compare our Graph-PCNN with other top-performed methods on COCO test-dev. As Table 5 reports, our method with HR48 backbone at the input size of 384x288 achieves the best AP(76.8), and improves HR48 with the same input size(75.5) by a large margin(+1.3). Mean while, It also outperforms other competitors with same backbone and input size settings, such as DARK(76.2), UDP(76.5) and PoseFix(76.7), which illustrates the advantages of our method.