DGGAN: Depth-image Guided Generative Adversarial Networks for Disentangling RGB and Depth Images in 3D Hand Pose Estimation

$3$D HPE from depth mapshas been extensively studied. Existing approaches in this field make noticeable advances [29, 33, 8, 31, 9, 11, 20]. Wan et al. [29] propose a dense regression approach to fit the parameters of a deformed hand model. Ge et al. [9, 11] present PointNet[24] to extract hand features and regress hand joint locations by referring to the extracted features. Wu et al. [31] adopt the intermediate dense guidance map supervision to generate hand heatmaps. Although the existing methods achieve very accurate estimation results, they typically rely on the hand data captured by high-precision depth sensors, which are still expensive to have in practice and usually require data collection in a lab environment. Different from the models in the aforementioned methods, our model performs inference on RGB data without the need of depth maps.

Due to the wide availability of RGB cameras, $3$D HPE from monocular RGB images is becoming increasingly popular in computer vision applications. Many recent methods aim at estimating hand joint locations directly from a single RGB image [4, 16, 10, 38, 22, 6, 32, 3, 36, 28]. Zimmermann et al. [38] use $2$D convolutional neural networks (CNN) to extract features from an RGB image, and regress the $3$D hand joint locations. However, their method suffers from depth ambiguity due to the absence of depth information. Developing the methods upon the work by Zimmermann et al., Iqbal et al. [16] and Cai et al. [4] inherit and adopt a similar $2$D CNN architecture for extracting image features. Iqbal et al. use depth maps as intermediate guidance while Cai et al. treat depth maps as a regularizer in a weakly supervised manner. Though these two methods make substantial progress in terms of estimation accuracy, there currently exist few datasets that fulfill their requirement of paired depth maps and RGB images. Ge et al. [10] take one step further by predicting the hand mesh from an RGB image and then the 3D hand joint locations based on the mesh. However, their method requires paired mesh information which is even rarer among all existing datasets.

Compared with these methods, our method also uses depth information during training, but it does not require any paired RGB images and depth maps. Thus, it is much more flexible since it can consume RGB images and depth maps from different datasets or sources.

To further enhance $3$D HPE [2, 3, 10, 18], hand mesh estimation can be included. Namely, the model estimates not only the hand joints but also the hand surface mesh. However these methods such as [10] have a common drawback: They require additional mesh annotations which are even more expensive to obtain than joint locations. Thus, they are typically trained on synthetic datasets due to this limitation. Seungryul et al. [3] introduce an iterative learning method to refine mesh shapes and achieve very good performance. However, like $3$D hand joint locations, hand meshes highly rely on additional supervision from hand segment maps which are typically not available in nowadays hand pose datasets. The method by boukhayma et al. [3] is the only extra-data-free method, but its performance is limited.

Generating images using generative adversarial networks (GAN) [13] has gained remarkable progress. Many approaches explore how to better manipulate images by applying GAN models [14, 17, 37, 7]. Isola et al. [17] propose the Pix2Pix network which translates label or edges maps to synthesized photos, reconstructs objects from edge maps, or colorizes images. Zhu et al. [37] introduce the cycle-consistent generated adversarial network (CycleGAN). CycleGAN uses the cycle consistency loss to disentangle the input and output pair and therefore does not need paired input. Hoffman et al. [14] propose cycle-consistent adversarial domain adaptation (CyCADA). Compared to CycleGAN, CyCADA contains a segmentation loss. As a result, CyCADA not only translates images from one modality to another but also deals with a specific visual task.

Applying the generative adversarial model to RGB hand images for hand pose estimation is also gaining popularity. Muller et al. [22] introduce the geometry consistent GAN (GeoConGAN) to generate synthetic image data for training. Chen et al. [6] propose the tonality-alignment generative adversarial networks (TAGAN) for producing more realistic images from synthetic images for hand pose estimator training. However, these methods only focus on generating RGB images. None of them generates depth maps for assisting hand pose estimator training.

The proposed DGGAN consists of two major modules, a depth-map reconstruction module and a hand pose estimation module. Its network architecture is shown in Figure 2.

Given an RGB hand image $\mathbf{I}$, we want to estimate the $K$ $3$D hand joint locations $\mathbf{J}^{xyz}\in\mathbb{R}^{3\times K}$. Each column in the $3\times K$ matrix is a vector of size $3$ and represents the $(x,y,z)$ coordinates of a joint, i.e., $\mathbf{J}^{xyz}=[J^{xyz}_{1},J^{xyz}_{2},\ldots,J^{xyz}_{K}]$.

The two modules in the proposed DGGAN $G$ are trained by using the GAN loss $\mathcal{L}_{GAN}$ and the task loss $\mathcal{L}_{task}$, respectively. The objective of learning $G$ is formulated as a min-max game:

where $\lambda_{t}$ and $\lambda_{g}$ control the relative importance of these two loss terms.

Given an RGB hand image, our depth-map reconstruction module tries to generate its corresponding depth map. A set of unpaired training depth images is adopted to train the depth-map reconstruction module so that its inferred depth maps are similar to real ones. To achieve that, the discriminator in this module works on distinguishing real depth maps from fake (generated) ones. Section 3.2 describes the details of depth-map reconstruction. The depth map inferred from the depth-map reconstruction module together with the input RGB image are fed to the hand pose estimation module for estimating the $3$D hand pose. In the hand pose estimation module, the input RGB image is used to regress the $3$D hand pose. The inferred depth-map is adopted to regularize the predicted $3$D hand pose. The loss for hand pose estimation $\mathcal{L}_{task}$ is adopted for optimization. Section 3.3 describes the details.

The depth-map reconstruction module aims at relaxing the requirement of paired RGB and depth images during training. This module is constructed via an adversarial network that infers the depth map according to an input RGB image. Figure 3 shows the network architecture of this module. In the training phase, our network requires both depth and RGB training images. Nevertheless, the RGB and depth images do not need to be paired. We consider the process of inferring depth map from its corresponding RGB image as an unsupervised adaptation problem, where the RGB modality $S$ and depth modality $T$ are both provided. We are given a set of RGB images $X_{S}$ and a set of real depth maps $X_{T}$. To translate from $S$ to $T$, we adopt an encoder-decoder architecture $G_{S\rightarrow T}$. The generator $G_{S\rightarrow T}$ is trained to generate a realistic depth map to fool the discriminator $D$ while $D$ id derived to distinguish the real data $x_{t}$ and generated fake data $G_{S\rightarrow T}(x_{s})$. The loss for the depth-reconstruction modules is as follows:

This loss also provides semantic constraints to force the generator to produce more realistic depth maps. By taking as input unpaired RGB and depth images, our depth-map reconstruction module becomes applicable to vastly more hand pose datasets. Furthermore, we can train the network with a large amount of unpaired RGB and depth images.

Given an inferred depth map computed by the depth-map reconstruction module, we combine it with the input RGB image and feed both to the hand pose estimation module. The network architecture of the hand pose estimation module is shown in Figure 4. The hand pose estimation module calculates the task loss $\mathcal{L}_{task}$, which is composed of two terms $\mathcal{L}_{task}=\mathcal{L}_{2D}+\mathcal{L}_{z}$. The $3$D hand regression loss $\mathcal{L}_{2D}$ and depth regularization loss $\mathcal{L}_{z}$ are described in section 3.3.1 and 3.3.2, respectively.

We conduct the experiments on three benchmark datasets, including the stereo hand tracking benchmark (STB) [35], the render hand pose dataset (RHD) [38], and the multi-view hand pose (MHP) dataset[12].

The STB dataset is a dataset of real hands. It contains two different subsets called SK and BB. The images in SK are captured by Point Grey Bumblebee2 stereo camera while images in BB are from a depth sensor. In our experiments, we use the BB subset for DGGAN training, and leverage the SK subset for unpaired testing.

RHD is a synthetic dataset. Zhang et al. [35] use a $3$D simulator, Maya, to render the images from $20$ different characters doing $39$ actions. Each data entry consists of an RGB image and the corresponding depth image, and both $2$D/$3$D annotations. This dataset is challenging since its images are captured with various view points and of many different hand shapes.

The MHP dataset provides color hand images as well as the bounding boxes of hands and the $2$D and $3$D location of each joint. It consists of hand imaegs of $21$ people with different hand movements. For each frame, it provides the images from four different angles of view. The $2$D and $3$D annotations are obtained by Leap Motion Controller.

Before training, we first crop the hand regions from the original canvas to make sure that hand parts have dominating proportion in the frame. Notice that the STB and MHP datasets use the center of a palm rather than a wrist as one of its hand keypoints. Hence, we revise the annotation to move the center of the palm to the wrist in the same way performed in [4].

Following the previous works [4, 6, 38], we evaluate the results of hand pose estimation by using 1) the area under the curve (AUC) on percentage of correct keypoints (PCK) between threshold $20$mm and $50$mm (AUC $20$_$50$) and 2) the end-point-error (EPE): the distance between predicted $3$D joint locations and the ground truth. In Table 1, we report the AUC $20$_$50$ as well as the mean and the median of EPE over all hand keypoints.

During training, we first initialize the weights of the depth-map reconstruction and hand pose estimation modules in the proposed DGGAN. Both modules are initialized by fitting the STB dataset (see 4.1) but trained separately. Then, we connect the two modules and fine-tune the whole network in an end-to-end manner. For training with the RHD and STB dataset, the discriminator is derived to distinguish the $G_{S\rightarrow T}(x_{s})$ and $x_{t}$, a randomly chosen depth-map from the respective dataset. For the MHP dataset, we simply randomly assign a depth-map from RHD dataset as $x_{t}$ because the MHP dataset does not contain any dense depth maps.

For analyzing the effectiveness of the proposed DGGAN, we conduct ablation studies for DGGAN on three different datasets. The detailed results are summarized in Table 1. Specifically, we conduct the experiments for the following three different settings:

1. 1.

   Regression: It represents training the regression network only on RGB images and without any depth regularizer.

2. 2.

   Regression + DR + DGGAN: We learne the depth-regularized regression network using RGB images with the depth maps generated by DGGAN.

3. 3.

   Regression + DR + true depth map: We derive the depth-regularized regression network using RGB images with their paired true depth maps.

To measure the effectiveness of the generated depth maps, we compare settings Regression and Regression + DR + DGGAN. As illustrated in Table 1, using the generated depth map significantly boosts the performance of the model in Regression. The AUC $20$_$50$ is improved by 0.043, 0.024, 0.011 on the RHD, STB, and MHP datasets, respectively. The EPE mean is also considerable reduced by 13.2% and 19.7% and 7.3% on the RHD, STB and MHP datasets respectively.

To compare the generated depth map with the real depth maps, we conduct two more experiments. Comparing results of Regression + DR + true depth map and Regression + DR + DGGAN shows that the generated depth maps are a key factor of performance boosting. On the RHD dataset, training with the generated depth maps is only slightly worse than the true RHD depth maps by $0.02$ in AUC $20$_$50$ and $1$ mm in EPE mean. However, on the STB dataset, the results of training with generated depth maps are even better than training with the real depth maps (by $0.006$ in AUC $20$_$50$ and $0.94$ mms in EPE mean). This result is probable due to the fact that the depth maps collected from depth sensors are less stable and noisier than the depth maps collected from a $3$D simulator. By training the DGGAN with unpaired high-quality depth maps from RHD, our generator can potentially reduce the noise, and further benefit the training in the hand pose estimation module. It is worth noting that Regression + DR + true depth map requires the paired depth and RGB image.

In addition to the quantitative analysis, Figure 7 and Figure 8 provide some examples for visual comparison between the generated and true depth maps on the RHD and STB datasets, respectively. We can see that the generated depth maps are visually very similar to the ground-truth ones.

We select the state-of-the-art approaches [3, 4, 23, 26, 35, 22, 38] for comparison. The comparison results are reported in Figure 6 and Table LABEL:table:stoa. As shown in Figure 6 and Table LABEL:table:stoa, our approach outperforms all existing state-of-the-art methods. Although the results of the method by Cai et al. [4] come close to ours, we emphasize that our DGGAN has an crucial advantage of not requiring any paired RGB and depth images.