A Unified Framework for Domain Adaptive Pose Estimation

Pose estimation has become an active research topic for years. In this paper, we focus on 2D pose estimation. Hourglass [30] is one of the dominant approaches for human pose estimation which applies an encoder-decoder style network with residual modules and finally generate heatmaps. A mean-squared error loss is applied between the predicted heatmap and ground-truth heatmap consisting of a 2D Gaussian centered on the annotated joint location [38]. Xiao et al. [42] propose a simple baseline model that combines upsampling and deconvolutional layers without using residual modules. HRNet [36] is proposed to maintain high-resolution in the model and achieves promising results. In this paper, we adopt the architecture of the Simple baseline model [42] following [17] to fairly compare our method with prior domain adaptation algorithms.

Unsupervised Domain Adaptation (UDA) aims to bridge the domain gap between a labeled source domain and unlabeled target domain. Existing domain adaptation methods utilize adversarial learning [9, 28], minimize feature distances using MMD [11], optimal transport [2], pixel-level adaptation [13], or maximum classifier discrepancy [34] for classification. In addition several other UDA methods have been proposed for dense prediction tasks including semantic segmentation [14, 25, 39, 44] and depth estimation [21, 22, 32]. Compared to other visual tasks, domain adaptation for regression tasks are still not well explored.

There are two categories in domain adaptation pose estimation: (1) For human pose estimation, RegDA [17] made changes in MDD [45] for human and hand pose estimation tasks, which measures discrepancy by estimating false predictions on the target domain. (2) For animal pose estimation, pseudo-labeling based approaches have been proposed in [23, 29]. Mu et al. [29] proposed invariance and equivariance consistency learning with respect to transformations as well as temporal consistency learning with a video. Li et al. [23] proposed a refinement module and a self-feedback loop to obtain reliable pseudo labels. Besides, WS-CDA [3] leverages human pose data and a partially annotated animal pose dataset to perform semi-supervised domain adaptation. In our experiments, we observed that (1) and (2) do not work well on the other tasks. A likely cause could be that each estimation task has different types of domain shifts, as shown in Fig 2(a). To address this, we propose a unified framework that generalizes well on diverse tasks by utilizing both input-level and out-level cues.

Given a labeled pose dataset $\mathcal{S}=\{(x_{s}^{i},y_{s}^{i})\}_{i=1}^{N}$ in source domain consisting of $N$ pairs of images $x_{s}\in\mathbb{R}^{H\times W\times 3}$ and corresponding annotation heatmap $y_{s}\in\mathbb{R}^{K\times 2}$ representing the coordinates of $K$ keypoints, as well as an unlabeled pose dataset $\mathcal{T}=\{x_{t}^{i}\}_{i=1}^{M}$ in target domain consisting of $M$ images $x_{t}\in\mathbb{R}^{H\times W\times 3}$, we aim to learn a 2D pose estimation model $h$ and optimize the performance on the target domain. Typically, the pose estimation model $h$ is pre-trained on the source domain dataset in a supervised manner to learn pose estimation from heatmaps $H_{s}=L(y_{s})$, where $H\in\mathbb{R}^{K\times H^{\prime}\times W^{\prime}}$ with the output heatmap size $H^{\prime}$ and $W^{\prime}$, generated through the heatmap generating function $L:\mathbb{R}^{K\times 2}\rightarrow\mathbb{R}^{K\times H^{\prime}\times W^{\prime}}$, with classic MSE loss: $L_{sup}=\frac{1}{N}\sum_{x_{s}\in\mathcal{S}}||h(x_{s})-H_{s}||_{2}$.

Different from prior works [13, 14, 40] that adopt adversarial learning, we propose to perform input-level alignments via style transfer for the sake of efficiency and simplicity. We borrow notations from AdaIN [15] and follow its settings and training procedure to extract content features from a content image $c$ and style feature from a style image $s$ through a pre-trained VGG [35] model $f$. Formally, style transfer is performed with a generator $g$ pre-trained as in AdaIN:

where $t=\text{AdaIN}(f(c),f(s))$ is the combination of content and style feature through adaptive instance normalization and $\alpha$ is the content-style trade-off parameter. Exemplar results are illustrated in the appendix. With a fixed AdaIN model, we transform source domain images with styles from target domain $x_{s\rightarrow t}=T(x_{s},x_{t},\alpha)$ and revise the supervised loss above:

To better exploit information from the unlabeled target domain, we adopt the paradigm of Mean Teacher that trains a student pose estimation model $h^{s}$ by the guidance produced by its self-ensemble, i.e., the teacher pose estimation model $h^{t}$ in an unsupervised learning branch. The input image for each model is augmented by $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ stochastically sampled from data augmentation $\mathcal{A}$. While the student $h^{s}$ is updated according to the supervised loss in Eq. 2 and self-guidance from the teacher $h^{t}$, the weights of the latter are updated as the estimated moving average of the former.

On the opposite direction to the supervised learning branch that transforms the source image to the target domain, we also propose to transform the target domain image back to the direction of the source domain where supervised learning happens and bridge the domain gap when generating guidance from the teacher model. Formally, we take a source domain image as the style reference and generate $x_{t\rightarrow s}=T(\mathcal{A}_{1}(x_{t}),x_{s},\alpha)$. After that, we pass the transformed image through the teacher model and get corresponding heatmap $H_{t}=h^{t}(x_{t\rightarrow s})$.

With the generated guidance heatmap from the teacher model, we still need to address the drifting effect that brings in instability in the unsupervised learning, as illustrated in Fig. 4. Technically, we generate pseudo-labels $\hat{H_{t}}=L(\hat{y_{t}})$ with the positions that produce maximum activation $\hat{y_{t}}=\arg\max_{p}H_{t}^{:,p}$ from each keypoints of the guidance heatmap to normalize the heatmap. We also revise the typical thresholding mechanism using a fixed value in Mean Teacher and determine the confidence threshold $\tau_{conf}$ with the top $p\%$-th values among maximum activation from each keypoint to exclude noises and further improve the quality of the self-guidance.

In addition to improving the quality of the teacher’s prediction, we also seek to challenge the student model by adaptively occluding the input to the student model according to feedback from the teacher. To be more specific, we mask the regions where the teacher model makes confident prediction of a keypoint with activation greater than $\tau_{occ}$ via an occlusion operation: $\hat{x}_{t}=O(\mathcal{A}_{2}(x_{t}),\tau_{occ})$, and let the student to learn robust prediction based on its contextual correlation with other keypoints from teacher’s pseudo-label after reversing augmentations $\tilde{\mathcal{A}}_{1}$ and $\tilde{\mathcal{A}}_{2}$. Overall, the student model $h^{s}$ will be guided by the normalized heatmap $\hat{H_{t}}$ via an unsupervised learning loss on keypoints $k$ producing maximum activation $H_{t}^{k,\hat{y}_{t}}$ greater than or euqal to threshold $\tau_{conf}$:

Combining our supervised learning loss from Eq. 2 and unsupervised learning loss from Eq. 3, we present the illustration for the overall pipeline in Fig. 3 and the final learning objectives:

We adopted the architecture of Simple Baseline [42] as our pose estimation model for both $h^{s}$ and $h^{t}$, with backbone of pre-trained ResNet101 [12]. Following Simple Baseline and RegDA, we adopted Adam [20] as the optimizer and set the base learning rate as 1e-4. It decreased to 1e-5 at 45 epochs and 1e-6 at 60 epochs, while the whole training procedure consisted of 70 epochs. The batch size was set to 32 and there are in total 500 iterations for each epoch. The confidence thresholding ratio $p$ is 0.5, while the occlusion thresholding value $\tau_{occ}$ is 0.9. The momentum $\eta$ for the update of the teacher model is 0.999 and the unsupervised learning weight was set to 1 to balance the supervised and unsupervised loss to a similar level. Also, the model was only trained by the supervised loss on the source domain for the first 40 epochs. On the basis of augmentation in RegDA, we added rotation (-30°, 30°) and random 2D translation (-5%, 5%) for the input source and target domain images. Finally, it should be noted that we used the same hyper-parameters for all experiments, did not tune the number of training epochs on test sets, and always report the accuracy of models from the last epoch. As for the architecture and optimization procedure of the style transfer model, we follow settings in AdaIN, except that we pre-train the model bidirectionally, i.e., both source and target domain image can be a content or a style image. Additional details can be found in the appendix.

Rendered Hand Pose Dataset [47] (RHD) provides $44k$ synthetic hand images including $41.2k$ training images and $2.7k$ test images along with corresponding 21 hand keypoints annotations. Hand-3D-Studio [46] (H3D) is a real-world multi-view indoor hand pose images dataset with $22k$ frames. We follow RegDA’s policy to split $3.2k$ frames as the test set. FreiHAND [48] includes $44k$ frames of real-world multi-vew hand pose images with more varied pose and view points. It contains $130k$ training image, and we still follow settings in RegDA to select $32k$ test images. SURREAL [41] provides more than 6 million synthetic human body pose images with annotations. Human3.6M [16] contains 3.6 million frames of real-world indoor human body pose images captured from videos. We follow protocols in [24] and split 5 subjects (S1, S5, S6, S7, S8) as the training set and 2 subjects (S9, S11) as test set. Leeds Sports Pose [18] (LSP) is a real-world outdoor human body pose dataset containing $2k$ images. Synthetic Animal Dataset [29] is a synthetic animal pose dataset rendered from CAD models. The dataset contains 5 animal classes, horse, tiger, sheep, hound, and elephant, each with $10k$ images. TigDog Dataset [31] includes $30k$ frames from real-world videos of horses and tigers. Animal-Pose Dataset [3] provides $6.1k$ real-world images from 5 animals including dog, cat, cow, sheep, and horse.

Among the baseline methods, UDA-Animal achieves the best performance in estimating a horse’s pose and approaches the oracle performance from a model trained jointly by the annotated source and target domains. Our method achieves slightly lower performance in the horse set that is close to the oracle level but slightly outperforms UDA-Animal in the tiger set.

In despite of the promising results in SynAnimal$\rightarrow$TigDog, we observe that UDA-Animal significantly underperforms than RegDA and ours in the AnimalPose dataset from Table 4. This is because SynAnimal$\rightarrow$AnimalPose is more challenging than SynAnimal$\rightarrow$TigDog by comparing the accuracy of source only models (32.2% vs. 71.4%). Even though we can still see improvements from the source only with augmentations, CCSSL and UDA-Animal face more noisy pseudo-labels during self-training possibly due to their hyper-parameter sensitivity, so that improvements are marginal. On the contrary, RegDA shows noticeable improvement compared to source only. Our method can handle these challenging settings via heatmap normalization in pseudo-labeling and obtain the best performance in these experiments in both categories.

So far, we have focused on accuracy in a given target domain, but we may face other types of unseen domains during training in real-world applications. Thus, we compare the generalization capacity of our method with baselines in a domain generalization setting where we test models on unseen domains and objects.

To further validate the robustness and generalization capacity of our method, we conducted sensitivity analysis regarding three major hyper-parameters in our framework, including the confidence thresholding ratio $p$, occlusion thresholding value $\tau_{occ}$, the momentum $\eta$ in Mean Teacher on RHD$\rightarrow$H3D. Additionally, we randomly split a separate validation set with the same size as the test set from the target domain training data to simulate the hyper-parameter tuning process and avoid directly tuning the test accuracy. Based on the results presented in Fig. 6, we find that our framework works stably under various settings. Meanwhile, we also find that the performance gradually decreases when we have a higher thresholding ratio for pseudo-labels, presumably because it brings in lower confident predictions as pseudo-labels and that deteriorates the unsupervised learning process. Also, we find that a greater teacher momentum is more likely to limit the framework to learn actively and harm the performance. More importantly, we can also learn that the validation accuracy in all experiments is highly correlated with that on the test sets, which also indicates the generalization capacity of our method and the reliability to give indicative clues when tuning hyper-parameters on a separate validation set.

We perform ablation studies in our framework to test their effectiveness and interaction with the rest of the framework. This also justify our other motivations regarding the task and the framework. Experiments are conducted under our major benchmarks including RHD$\rightarrow$H3D and SynAnimal$\rightarrow$TigDog. Additional ablation studies can be found in the appendix.

Based on Table 7, our framework can benefit from the heatmap normalization (denoted by Norm) that stabilizes the drifting effect and enables effective unsupervised learning from pseudo-labels via output-level domain alignment. Nevertheless, experiments on animal adaptation tasks show that such alignment might not be sufficiently helpful. Instead, more improvements are brought by the style transfer module, which confirms our reasoning that input-level variance is the major challenge in this task and can be mitigated by input-level alignments.

Adaptive occlusion can also provide extra focus on learning to detect occluded keypoints, as we can observe from RHD$\rightarrow$H3D. However such improvements are not reflected in SynAnimal$\rightarrow$TigDog. Considering the qualitative results in Figs. 2, we conjecture that it is because the improvements in detecting occluded keypoints are not verifiable as their annotations are not available in the real animal dataset and therefore these predictions are not included in the [email protected] evaluation protocol. More ablation studies are available in the appendix.

We follow settings from AdaIN to train the generator $g$ from Eq. 2 with a content loss and a style loss balanced by style loss weight $\lambda=0.1$, on images with a resolution of $256\times 256$. Exemplar results are illustrated in Fig. 7. During the training process of our framework, the pre-trained style transfer module will be fixed and perform bidirectional style transfer with a probability of $0.5$ in both our supervised and unsupervised learning branch with the content-style trade-off parameter $\alpha$ uniformly sampled from 0 to 1.

Our pose estimation model $h$ is trained with input images with a resolution of $256\times 256$ and output heatmaps with a size of $64\times 64$, with the batch size of $32$ in each iteration, following our baselines [17].

As for our adaptive keypoint occlusion, we randomly select keypoints with maximum activation greater than the occlusion threshold $\tau_{occ}$ and occlude it with a probability of $0.5$. The keypoints will be occluded by a patch from a random position in the same image with the size of $20\times 20$.

In addition to RHD$\rightarrow$H3D and SynAnimal$\rightarrow$TigDog, we also present ablation studies on another major benchmark, SURREAL$\rightarrow$Human3.6M in Table 8. Based on the results we can observe a greater improvement after applying heatmap normalization (the first and the second row), showing the necessity of addressing the drift effect under this scenario. On the other hand, we can also observe fewer improvements (the third and the fourth row) brought by the style transfer module, which coincide with our conclusion from the ablation studies on RHD$\rightarrow$H3D that the major challenge in human pose estimation tasks comes from the output-level discrepancy instead of the input-level. On that basis, our adaptive keypoint occlusion mechanism further boosts the performance by 2.2 percent points (the last row) and achieves the state-of-the-art performance, which shows the effectiveness of the occlusion mechanism specialized in this task.

Tab. 9 presents ablation studies of data augmentation methods on RHD$\rightarrow$H3D. We compare the performance of our method with different compositions of augmentations commonly used in pose estimation tasks, and we observe that rotation provides the most significant gain.

We provide additional qualitative results in Figures. 8,  9, 10, 11, and 12.