MetaFuse: A Pre-trained Fusion Model for Human Pose Estimation

In the first step, we train the backbone network (i.e. the layers before the fusion model) to speed up the subsequent meta-training process. All images from the training dataset are used for training the backbone. The backbone parameters are directly optimized by minimizing the MSE loss between the initial heatmaps and the ground truth heatmaps. Note that the backbone network is only trained in this step, and will be fixed in the subsequent meta-training step to notably reduce the training time.

In this step, as shown in Figure 5, we learn the generic fusion model $\bm{\omega^{\text{base}}}$ and the initializations of $\theta$ by a meta-learning style algorithm. Generally speaking, the two parameters are sequentially updated by computing gradients over pairs of cameras (sampled from the dataset) which are referred to as tasks.

Task is an important concept in meta-training. In particular, every task $\mathcal{T}_{i}$ is associated with a small dataset $\mathcal{D}_{i}$ which consists of a few images and ground truth $2$D pose heatmaps sampled from the same camera pair. For example, the camera pair $(\text{Cam}_{1},\text{Cam}_{2})$ is used in task $\mathcal{T}_{1}$ while the camera pair $(\text{Cam}_{3},\text{Cam}_{4})$ is used in in task $\mathcal{T}_{2}$. We learn the fusion model from many of such different tasks so that it can get good results when adapted to a new task by only a few gradient updates. Let $\{\mathcal{T}_{1},\mathcal{T}_{2},\cdots,\mathcal{T}_{N}\}$ be a number of tasks. Each $\mathcal{T}_{i}$ is associated with a dataset $\mathcal{D}_{i}$ consisting of data from a particular camera pair. Specifically, each $\mathcal{D}_{i}$ consists of two subsets: $\mathcal{D}_{i}^{train}$ and $\mathcal{D}_{i}^{test}$. As will be clarified later, both subsets are used for training.

We follow the model-agnostic meta-learning framework [10] to learn the optimal initializations for $\bm{\omega^{\text{base}}}$ and $\theta$. In the meta-training process, when adapted to a new task $\mathcal{T}_{i}$, the model parameters $\bm{\omega^{\text{base}}}$ and $\theta$ will become $\bm{\omega^{\text{base}}{{}^{\prime}}}$ and $\theta^{\prime}$, respectively. The core of meta-training is that we learn the optimal $\bm{\omega^{\text{base}}}$ and $\theta$ which will get a small loss on this task if it is updated based on the small dataset of the task. Specifically, $\bm{\omega^{\text{base}}{{}^{\prime}}}$ and $\theta^{\prime}$ can be computed by performing gradient descent on task $\mathcal{T}_{i}$

The learning rate $\alpha$ is a hyper-parameter. It is worth noting that we do not actually update the model parameters according to the above equations. $\bm{\omega^{\text{base}\prime}}$ and $\theta^{\prime}$ are the intermediate variables as will be clarified later. The core idea of meta learning is to learn $\bm{\omega^{\text{base}}}$ and $\theta$ such that after applying the above gradient update, the loss for the current task (evaluated on $\mathcal{D}_{i}^{test}$) is minimized. The model parameters are trained by optimizing for the performance of $\mathcal{L}_{\mathcal{D}^{\text{test}}_{i}}(\bm{\omega^{\text{base}\prime}},\theta^{\prime})$ with respect to $\bm{\omega_{\text{base}}}$ and $\theta$, respectively, across all tasks. Note that, $\bm{\omega^{\text{base}}{{}^{\prime}}}$ and $\theta^{\prime}$ are related to the initial parameters $\bm{\omega_{\text{base}}}$ and $\theta$ because of Eq.(6) and Eq.(6). More formally, the meta-objective is as follows:

The optimization is performed over the parameters $\bm{\omega_{\text{base}}}$ and $\theta$, whereas the objective is computed using the updated model parameters $\bm{\omega^{\text{base}\prime}}$ and $\theta^{\prime}$. In effect, our method aims to optimize the model parameters such that one or a small number of gradient steps on a new task will produce maximally effective behavior on that task. We repeat the above steps iteratively on each task $\mathcal{D}_{i}\in\{\mathcal{D}_{1},\mathcal{D}_{2},\cdots,\mathcal{D}_{N}\}$. Recall that each $\mathcal{D}_{i}$ corresponds to a different camera configuration. So it actually learns a generic $\bm{\omega^{\text{base}}}$ and $\theta$ which can be adapted to many camera configurations with the gradients computed on small data. The $\bm{\omega^{\text{base}}}$ will be fixed after this meta-training stage.

This dataset [17] provides images captured by a large number of synchronized cameras. We follow the convention in [37] to split the training and testing data. We select $20$ cameras (i.e. $380$ ordered camera pairs) from the training set to pre-train MetaFuse. Note that we only perform pre-training on this large dataset, and directly finetune the learned model on each target dataset to get a customized multi-view fusion based $2$D pose estimator. For the sake of evaluation on this dataset, we select six from the rest of the cameras. We run multiple trials and report average results to reduce the randomness caused by camera selections. In each trial, four cameras are chosen from the six for multi-view fusion.

This dataset [14] provides synchronized four-view images. We use subjects $1,5,6,7,8$ for finetuning the pre-trained model, and use subjects $9,11$ for testing purpose. It is worth noting that the camera placement is slightly different for each of the seven subjects.

In this dataset [34], there are five subjects performing four actions including Roaming(R), Walking(W), Acting(A) and Freestyle(FS) with each repeating $3$ times. We use Roaming 1,2,3, Walking 1,3, Freestyle 1,2 and Acting 1,2 of Subjects 1,2,3 for finetuning the pre-trained model. We test on Walking 2, Freestyle 3 and Acting 3 of all subjects.

The $2$D pose accuracy is measured by Joint Detection Rate (JDR). If the distance between the estimated and the ground-truth joint location is smaller than a threshold, we regard this joint as successfully detected. The threshold is set to be half of the head size as in [2]. JDR is computed as the percentage of the successfully detected joints. The $3$D pose estimation accuracy is measured by the Mean Per Joint Position Error (MPJPE) between the ground-truth $3$D pose and the estimation. We do not align the estimated $3$D poses to the ground truth as in [22, 32].

The warming up step takes about $30$ hours on a single 2080Ti GPU. The meta-training stage takes about $5$ hours. This stage is fast because we use the pre-computed heatmaps. The meta-testing (finetuning) stage takes about $7$ minutes. Note that, in real deployment, only meta-testing needs to be performed for a new environment which is very fast. In testing, it takes about $0.015$ seconds to estimate a $2$D pose from a single image.

We use a recent $2$D pose estimator [38] as the basic network to estimate the initial heatmaps. The ResNet50 [13] is used as its backbone. The input image size is $256\times 256$ and the resolution of the heatmap is 64 $\times$ 64. In general, using stronger 2D pose estimators can further improve the final 2D and 3D estimation results but that is beyond the scope of this work. We apply softmax with temperature $T=0.2$ to every channel of the fused heatmap to highlight the maximum response.

The Adam [18] optimizer is used in all phases. We train the backbone network for $30$ epochs on the target dataset in the warming up stage. Note that we do not train the fusion model in this step. The learning rate is initially set to be $1e^{-3}$, and drops to $1e^{-4}$ at 15 epochs and $1e^{-5}$ at 25 epochs, respectively. In meta-training and meta-testing, the learning rates are set to be $1e^{-3}$ and $5e^{-3}$, respectively. We evaluate our approach by comparing it to the five related baselines, which are detailed in Table 1.

The joint detection rates (JDR) of the baselines and our approach are shown in Figure 7. We present the average JDR over all joints, as well as the JDRs of several typical joints. We can see that the JDR of No-Fusion (the grey dashed line) is lower than our MetaFuse model regardless of the number of images used for finetuning the fusion model. This validates the importance of multi-view fusion. The improvement is most significant for the wrist and elbow joints because they are frequently occluded by human body in this dataset.

NaiveFuse^full (the grey solid line) gets the highest JDR because it uses all training data from the H36M dataset. However, when we use fewer data, the performance drops significantly (the green line). In particular, NaiveFuse^$50$ even gets worse results than No-Fusion. This is because small training data usually leads to over-fitting for large models. We attempted to use several regularization methods including $l_{2}$, $l_{1}$ and $L_{2,1}$ (group sparsity) on $\bm{\omega}$ to alleviate the over-fitting problem of NaiveFuse. But none of them gets better performance than vanilla NaiveFuse. It means that the use of geometric priors in MetaFuse is more effective than the regularization techniques.

Our proposed AffineFuse^$K$, which has fewer parameters than NaiveFuse^$K$, also gets better result when the number of training data is small (the blue line). However, it is still worse than MetaFuse^$K$. This is because the model is not pre-trained on many cameras to improve its adaptation performance by our meta-learning-style algorithm which limits its performance on the H36M dataset.

Our approach MetaFuse^$K$ outperforms all baselines. In particular, it outperforms No-Fusion when only $50$ training examples from the H36M dataset are used. Increasing this number consistently improves the performance. The result of MetaFuse^$500$ is already similar to that of NaiveFuse^full which is trained on more than $80$K images.

We also evaluate a variant of NaiveFuse which is learned by the meta-learning algorithm. The average JDR are $87.7\%$ and $89.3\%$ when 50 and 100 examples are used, which are much worse than MetaFuse. The results validate the importance of the geometry inspired decomposition.

Figure 6 explains how MetaFuse improves the $2$D pose estimation accuracy. The target joint is the left knee in this example. But the estimated heatmap (before fusion) has the highest response at the incorrect location (near right knee). By leveraging the heatmaps from the other three views, it accurately localizes the left knee joint. The last three images show the warped heatmaps from the other three views. We can see the high response pixels approximately form a line (the epipolar line) in each view.

We visualize some typical poses estimated by the baselines and our approach in Figure 8. First, we can see that when occlusion occurs, No-Fusion usually gets inaccurate $2$D locations for the occluded joints. For instance, in the first example, the left wrist joint is localized at a wrong location. NaiveFuse and MetaFuse both help to localize the wrist joint in this example, and MetaFuse is more accurate. However, in some cases, NaiveFuse may get surprisingly bad results as shown in the third example. The left ankle joint is localized at a weird location even though it is visible. The main reason for this abnormal phenomenon is that the NaiveFuse model learned from few data lacks generalization capability. MetaFuse approach gets consistently better results than the two baseline methods.

We estimate $3$D pose from multi-view $2$D poses by a pictorial structure model [25]. The results on the H36M dataset are shown in Table 2. Our MetaFuse trained on only $50$ examples decreases the error to $32.7$mm. Adding more training data consistently decreases the error. Note that some approaches in the table which use the full H36M dataset for training are not comparable to our approach.