Semi-supervised Dense Keypoints
Using Unlabeled Multiview Images

Given a pair of corresponding images from two different views, a pair of corresponding points $\mathbf{x}\leftrightarrow\mathbf{x}^{\prime}$, are related by a fundamental matrix given the calibrated cameras, i.e. $\widetilde{\mathbf{x}}^{\prime\mathsf{T}}\mathbf{F}\widetilde{\mathbf{x}}=0$, where $\mathbf{F}\in\mathbb{R}^{3\times 3}$ is the fundamental matrix, and $\widetilde{\mathbf{x}}\in\mathds{P}^{2}$ is a homogeneous representation of $\mathbf{x}$. The measure of geometric consistency between the two images can be written as [9]:

$\displaystyle d(\mathbf{x},\mathbf{x}^{\prime};\mathbf{F})=\frac{|\widetilde{\mathbf{x}}^{\prime\mathsf{T}}\mathbf{F}\widetilde{\mathbf{x}}|}{\sqrt{(\mathbf{F}\widetilde{\mathbf{x}})_{1}^{2}+(\mathbf{F}\widetilde{\mathbf{x}})_{2}^{2}}},~{}~{}~{}~{}\mathbf{x}\leftrightarrow\mathbf{x}^{\prime},$

(1)

where $(\mathbf{Fx})_{i}$ is the $i^{\rm th}$ entry of $\mathbf{F}\mathbf{x}$. This measures epipolar error that is the Euclidean distance between $\mathbf{x}^{\prime}$ and the epipolar line $\mathbf{F}\widetilde{\mathbf{x}}$.

Consider an injective dense keypoint mapping $\phi:\mathds{R}^{2}\rightarrow\mathds{R}^{2}$ that maps a pixel coordinate to a canonical 2D body surface coordinate, i.e., $\mathbf{u}=\phi(\mathbf{x};\mathcal{I})$ where $\mathbf{x}\in\Theta(\mathcal{I})$ is the set of the foreground pixels in the image $\mathcal{I}$, and $\mathbf{u}\in\mathds{R}^{2}$ is the 2D coordinate in the body surface as shown in Figure 2. This dense keypoint mapping can be learned from annotations, e.g., DensePose [8] for humans where it maps to the texture coordinate of 3D human body surfaces. This mapping is incomplete because there exists missing correspondences in an image due to occlusion. One can find a correspondence between the two images through $\mathbf{u}$, i.e., $\phi^{-1}(\mathbf{u};\mathcal{I})\leftrightarrow\phi^{-1}(\mathbf{u};\mathcal{I}^{\prime})$ where $\mathcal{I}$ and $\mathcal{I}^{\prime}$ the two view images. However, $\phi$ is an injective mapping where the analytic inverse does not exist in general.

Given a point $\mathbf{x}$ in the image $\mathcal{I}$, one can measure the expectation of geometric error by a nearest neighbor search on the body surface space:

$\displaystyle\mathds{E}(\mathbf{x})=d(\mathbf{x},\mathbf{x}^{\prime};\mathbf{F}),~{}~{}~{}\mathbf{x}^{\prime}=\underset{\mathbf{x}^{\prime}\in\Theta(\mathcal{I}^{\prime})}{\operatorname{argmin}}~{}\|\phi(\mathbf{x};\mathcal{I})-\phi(\mathbf{x}^{\prime};\mathcal{I}^{\prime})\|,$

(2)

where $\mathds{E}(\mathbf{x})$ is the expectation of geometric error at $\mathbf{x}$. The expectation of the geometric error measures the epipolar error over all possible matches in the other view image, i.e., $\forall\mathbf{x}^{\prime}\in\Theta(\mathcal{I}^{\prime})$.

There are two limitations in the nearest neighbor search: (1) The correspondences are not exact, leading to a biased estimate of the geometric error expectation. (2) The argmin operation is not differentiable, which cannot be used to learn a dense keypoint detector in an end-to-end fashion.

Instead, we address these limitations by making use of a soft correspondence, or matchability—a likelihood of a point being matched to another point as shown in Figure 2:

$\displaystyle M(\mathbf{x},\mathbf{x}^{\prime};\phi,\mathcal{I},\mathcal{I}^{\prime})=P\left(\phi(\mathbf{x};\mathcal{I}),\phi(\mathbf{x}^{\prime};\mathcal{I}^{\prime});\{\phi(\mathbf{y};\mathcal{I}^{\prime})\}_{\mathbf{y}\in\Theta(\mathcal{I}^{\prime})}\right),$

(3)

where $P(\mathbf{u},\mathbf{u}^{\prime};\Omega)$ is the probability of matching between $\mathbf{u}$ and $\mathbf{u}^{\prime}$:

$\displaystyle P(\mathbf{u},\mathbf{u}^{\prime};\Omega)=\exp\left(-\frac{\|\mathbf{u}-\mathbf{u}^{\prime}\|^{2}}{2\sigma^{2}}\right)/\sum_{\mathbf{v}\in\Omega}\exp\left(-\frac{\|\mathbf{u}-\mathbf{v}\|^{2}}{2\sigma^{2}}\right).$

(4)

$\Omega$ is the domain of the canonical coordinates, $\sigma$ is the standard deviation that controls the smoothness of matching, e.g., when $\sigma\rightarrow 0$, it approximates the nearest neighbor search.

We use the matchability to form a probabilistic epipolar error:

$\displaystyle\mathds{E}(\mathbf{x},\mathbf{x}^{\prime})=M(\mathbf{x},\mathbf{x}^{\prime};\phi,\mathcal{I},\mathcal{I}^{\prime})d(\mathbf{x},\mathbf{x}^{\prime};\mathbf{F}),$

(5)

where $\mathds{E}(\mathbf{x},\mathbf{x}^{\prime})$ is the epipolar error between x and x’ weighted by the matchability.

The error expectation of $\mathbf{x}$ can be computed by marginalizing over all $\mathbf{x}^{\prime}$:

$\displaystyle\mathds{E}(\mathbf{x})=\sum_{\mathbf{x}^{\prime}\in\Theta(\mathcal{I}^{\prime})}\mathds{E}(\mathbf{x},\mathbf{x}^{\prime}).$

(6)

That is, the expectation of the geometric error of $\mathbf{x}$ measures a weighted average of epipolar errors for all possible correspondences. The higher matchability, the more contribution to the error expectation. Unlike Equation (2), Equation (6) does not include the non-differentiable argmin operation. Further, it takes into account all possible pairs of correspondences between two images, which eliminates the bias introduced by the nearest neighbor search.

Given the error expectations over the dense keypoint field, we derive a symmetric dense epipolar error that measures geometric consistency between two images:

$\displaystyle\mathds{E}(\mathcal{I},\mathcal{I}^{\prime})=\frac{1}{V}\sum_{\mathbf{x}\in\Theta(\mathcal{I})}v(\mathbf{x};\phi,\mathcal{I},\mathcal{I}^{\prime})\mathds{E}(\mathbf{x})+\frac{1}{V^{\prime}}\sum_{\mathbf{x}^{\prime}\in\Theta(\mathcal{I^{\prime}})}v(\mathbf{x}^{\prime};\phi,\mathcal{I},\mathcal{I}^{\prime})\mathds{E}(\mathbf{x}^{\prime}),$

(7)

where $v(\mathbf{x};\phi,\mathcal{I},\mathcal{I}^{\prime})\in\{0,1\}$ is a visibility indicator:

$\displaystyle v(\mathbf{x};\phi,\mathcal{I},\mathcal{I}^{\prime})=\begin{cases}1,&\exists\mathbf{x}^{\prime}\in\Theta(\mathcal{I^{\prime}}),\|\phi(\mathbf{x};\mathcal{I})-\phi(\mathbf{x}^{\prime};\mathcal{I}^{\prime})\|<\epsilon\\ 0,&\text{otherwise}.\end{cases}$

(8)

$V=\sum_{\mathbf{x}\in\Theta(\mathcal{I})}v(\mathbf{x};\phi,\mathcal{I},\mathcal{I}^{\prime})$ and $V^{\prime}=\sum_{\mathbf{x}^{\prime}\in\Theta(\mathcal{I}^{\prime})}v(\mathbf{x}^{\prime};\phi,\mathcal{I},\mathcal{I}^{\prime})$ are the numbers of foreground pixels visible in $\mathcal{I}^{\prime}$ and $\mathcal{I}$, respectively. $\epsilon$ is a correspondence tolerance defined in the canonical surface coordinate.

In fact, this dense epipolar error is a generalization of Sampson distance [9] that defines epipolar error between explicit point correspondences (point-to-point). We generalize it to model probabilistic correspondence between two set of points (field-to-field).

Existing approaches such as bootstrapping [22] establish the matching through 3D reconstruction of mesh and enforce the geometric error in an alternating fashion due to the non-differentiability of matching. Our differentiable formulation allows learning the dense keypoint detector in an end-to-end manner, which is flexible and shows superior performance.

We learn the dense keypoint detector $\phi$ by minimizing the following error:

$\displaystyle\mathcal{L}$

$\displaystyle=\lambda_{\rm L}\mathcal{L}_{\rm L}+\lambda_{\rm M}\mathcal{L}_{\rm M}+\lambda_{\rm R}\mathcal{L}_{\rm R}+\lambda_{\rm T}\mathcal{L}_{\rm T},$

(9)

where $\mathcal{L}_{\rm L}$ is the labeled data loss, $\mathcal{L}_{\rm M}$ is the multiview geometric consistency loss, $\mathcal{L}_{\rm R}$ is the regularization loss, and $\mathcal{L}_{\rm T}$ is the multiview photometric consistency loss. $\lambda_{\rm L}$, $\lambda_{\rm M}$, $\lambda_{\rm R}$ and $\lambda_{\rm T}$ are the weights to control their relative importance.

Then we compute multiview photometric consistency loss as:

$\displaystyle\mathcal{L}_{\rm T}=\sum_{\{\mathcal{I},\mathcal{I}^{\prime}\}\in\mathcal{D}_{\rm U}}\mathds{T}(\mathcal{I},\mathcal{I}^{\prime}),$

(15)

We design a new network architecture composed of twin networks to learn the dense keypoint detector by enforcing multiview consistency over dense keypoint fields as shown in Figure 4. Each network is made of a fully convolutional network that outputs the dense keypoint field per body part.

Given two dense keypoint fields, we compute the probabilistic epipolar error by constructing two affinity matrices: matchability matrix and epipolar matrix, $\mathbf{M},\mathbf{E}\in\mathds{R}^{|\Theta(\mathcal{I})|\times|\Theta(\mathcal{I}^{\prime})|}$ where $|\Theta(\mathcal{I})|$ is the cardinality of the range of foreground pixels. These two matrices are defined by:

$\displaystyle\mathbf{M}_{ij}$

$\displaystyle=M(\mathbf{x}_{i},\mathbf{x}_{j};\phi,\mathcal{I},\mathcal{I}^{\prime}),~{}~{}~{}~{}~{}\mathbf{E}_{ij}=d(\mathbf{x}_{i},\mathbf{x}_{j};\mathbf{F})$

(16)

where $\mathbf{M}_{ij}$ is the $i,j$ entry of the matrix $\mathbf{M}$. $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are the $i^{\rm th}$ and $j^{\rm th}$ foreground pixels from the images $\mathcal{I}$ and $\mathcal{I}^{\prime}$, respectively. We also compute the visibility maps $\mathbf{V}_{i}=v(\mathbf{x};\phi,\mathcal{I},\mathcal{I}^{\prime})$ and $\mathbf{V}^{\prime}_{i}=v(\mathbf{x}^{\prime};\phi,\mathcal{I},\mathcal{I}^{\prime})$.

We design a new operation to measure the dense epipolar error in Equation (7):

$\displaystyle\mathds{E}(\mathcal{I},\mathcal{I}^{\prime})=\frac{1}{V}\mathbf{V}^{\mathsf{T}}(\mathbf{M}\odot\mathbf{E})\mathbf{1}_{|\Theta(\mathcal{I}^{\prime})|}+\frac{1}{V^{\prime}}{\mathbf{V}^{\prime}}^{\mathsf{T}}(\mathbf{M}\odot\mathbf{E})^{\mathsf{T}}\mathbf{1}_{|\Theta(\mathcal{I})|},$

(17)

where $\mathbf{1}_{n}$ is the $n$-dimensional vector of which entries are all one. $\odot$ is the element-wise multiplication of matrices. Note that dense epipolar error $\mathds{T}(\mathcal{I},\mathcal{I}^{\prime})$ can be computed following the same operations.

We use HRNet [42] as the backbone network followed by four head networks made up of convolutional layers to predict foreground mask, body part index, and UV coordinates on the canonical body surface, respectively. Each network takes as an input a 224$\times$224 image and outputs 15-channel (for foreground mask head only) or 25-channel 56$\times$56 feature maps [8].

We train the network in two stages. In the first stage, we train an initial model using the labeled data by the labeled data loss $\mathcal{L}_{\rm L}$. Specifically, for the human dense keypoints, we use 48K human instances in DensePose-COCO [8] training set to train the initial model. For the monkey dense keypoints, we directly use the pretrained model for chimpanzees [36] as the initial model since we do not have an access to the labeled data. In the second stage, we train our model on unlabeled multiview data leveraging all losses. The initial model is used for two purposes: (1) the pre-trained model $\phi_{0}$ with weights fixed for distillation-based regularization and (2) the initial model to be refined via multiview supervision.

In our experiments, we consider four baselines: (1) A model fully-supervised by labeled data, i.e. DensePose-COCO [8], which is also our initial model for semi-supervised learning. We refer to this as Supervised in the following. (2) A model trained using multiview bootstrap strategy [38, 22], where multiview triangulation results from previous stage are used as the pseudo ground truth. (3) A learnable 3D mesh estimation method where dense keypoint estimation can be acquired by reprojecting 3D mesh to image domain, e.g. HMR [17]. (4) A 3D mesh estimation framework similar to HMR but additionally incorporating Model-fitting in the Loop, e.g. SPIN [20]. Supervised is also used as an initial model for our approach.

We evaluate the performance of our dense keypoint model using metrics from three aspects: (1) geometric consistency, (2) accuracy of dense keypoints, and (3) accuracy of 3D reconstruction from multiple views.

We use Human3.6M dataset to perform (1) comprehensive cross-method evaluations, (2) study on model’s generalizability towards new views, and (3) ablation study on losses used for training. Results are summarized in Table 1 with all metrics reported.

Cross-method Evaluation We evaluate the performance of our model against other methods as shown in the comparison block in Table 1 and show qualitative results (1st column in Figure 5). For the metrics of keypoint accuracy and geometry consistency, ours outperforms all other methods (except for $\rm{AUC_{10}}$ only second to HMR [17]). Note that HMR and SPIN are trained with the 3D ground truth (pre-estimated mesh) of Human3.6M while other algorithms including ours predict dense keypoints without the 3D ground truth. Having the 3D ground truth is a significant advantage while it requires a substantial additional amount of 3D annotation effort. It is expected to outperform the ones without it, in particular, on reconstruction accuracy metrics. We included these baselines to provide the upper bound of our semi-supervised method as a reference. Note that despite the lack of the 3D ground truth, ours still outperforms HMR and SPIN by a margin of up to 26.2% and 67.8% on keypoint accuracy metrics on three keypoint accuracy metrics. Compared to “Supervised” and “Bootstrapping” which are close to our setup, ours outperform on all metrics by a margin of up to 9.2% and 7.0% on keypoint accuracy metrics.

Generalizability towards new views We evaluate generalization by testing on different views: two views are used for training and other two views are used for testing. The results are summarized in the generalization block in Table 1. As can be seen, although a model trained purely on one camera pair does not use any sample captured by the other pair, its performance still improves on all metrics on top of Supervised model by a margin of 0.6%-12.8% on keypoint accuracy, 38.3% - 47.9% on geometric consistency, and 6.5% - 13.7% on reconstruction accuracy. This shows that model trained by our semi-supervised approach can be generalized to new views.

Ablation Study We conduct an ablation study to evaluate the impact of each loss. The results are reported in the ablation block in Table 1. Note that we propose semi-supervised learning thus $\mathcal{L}_{\rm L}$ is always used. The inferior performance of model trained with $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm M}$ or $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm T}$ or $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm M}$ + $\mathcal{L}_{\rm T}$ proves that refining initial model by multiview loss alone suffers from degeneration (Section 3). This limitation can be addressed by adding the regularization $\mathcal{L}_{\rm R}$. Note that regularization itself does not add any value: it is identical to the supervised model in theory. This is empirically verified since $\mathcal{L}_{\rm L}+\mathcal{L}_{\rm R}$ performs similarly to $\mathcal{L}_{\rm L}$. Both multiview consistency losses show effectiveness: $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm R}$ + $\mathcal{L}_{\rm M}$ and $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm R}$ + $\mathcal{L}_{\rm T}$ outperform $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm R}$ on all metrics by a margin up to 6.5% / 2.7% for keypoint accuracy metrics and 8.7% / 9.8% for reconstruction accuracy metrics respectively. $\mathcal{L}_{\rm L}$ + $\mathcal{L}_{\rm R}$ + $\mathcal{L}_{\rm M}$ + $\mathcal{L}_{\rm T}$ further achieves better results on all metrics. Multiview geometric consistency loss mainly contributes to the improvement on keypoint accuracy and geometric consistency metrics, while multiview photometric consistency loss mainly contributes to reconstruction accruacy metrics.

To further validate the usefulness of our method in terms of dense keypoint accuracy, in addition to detection accuracy on multiview image data, we conduct another cross-method evaluation on 3DPW dataset (single-view dense keypoint detection) on the keypoint accuracy metrics, i.e., training on DensePose-COCO with Human3.6M multiview supervision and testing on 3DPW without an adaptation, as shown in Table 2. The results show that our trained model with geometric consistency is more generalizable than the baselines.

We evaluate our method on multiview in-the-wild datasets: Ski-pose and OpenMonkeyPose. Since no ground truth is available, we evaluate geometric consistency summarized in Table 3 and show qualitative results (2nd and 3rd columns in Figure 5). The results show that our model outperforms other baselines by a large margin: 37.2%-54.1% on Ski-Pose and 31.9%-47.5% on OpenMonkeyPose, which can be visually identified by qualitative results.