Camera-Space Hand Mesh Recovery via Semantic Aggregation
and Adaptive 2D-1D Registration

The first phase of our pipeline extracts 2D pose and silhouette. Both 2D pose and silhouette are represented by heatmaps. To refine the 2D cues gradually, we use a multi-stack hourglass network [29].

The 3D mesh is defined by its shape and pose [35]. The silhouette is a holistic 3D-to-2D projection so it captures important shape cues. However, it can hardly describe the pose accurately. On the other hand, joint landmark locations are very informative for the pose. Given their different roles, how to better combine them becomes an interesting question. Therefore, we have an insight that to improve accuracy of the subsequent 3D tasks, it is essential not only to improve the accuracy of the 2D tasks respectively, but also to better aggregate them according to their semantic relations. Specifically, we propose to aggregate a series of 2D cues denoted as follows:

• •

  $\mathbf{H}_{p}$: $N$ heatmaps of 2D poses. Each heatmap corresponds to a joint landmark.

• •

  $\mathbf{H}_{s}$: a single heatmap of the silhouette.

• •

  cat($\mathbf{H}_{p},\mathbf{H}_{s}$): concatenating heatmaps of $\mathbf{H}_{p}$ and $\mathbf{H}_{s}$ to aggregate 2D pose and silhouette.

• •

  sum($\mathbf{H}_{p}$): combining all the joint landmarks as a single heatmap to aggregate joint locations.

• •

  group($\mathbf{H}_{p}$): concatenating $\mathbf{H}_{p}$ and tip-, part- , or level-grouped landmarks to aggregate joint semantics for high-level semantic relations.

2D silhouette and joint landmarks represent pixel-level locations in different aspects. A straightforward way of combining them is cat($\mathbf{H}_{p},\mathbf{H}_{s}$). Another simple baseline, sum($\mathbf{H}_{p}$), would discard semantics of individual joints by encoding their locations in a single heatmap. Thus, comparing $\mathbf{H}_{p}$ and sum($\mathbf{H}_{p}$) would reveal the effect of joint semantics. These simple baselines, as we will show, are inferior to more semantically meaningful ways of aggregation: group($\mathbf{H}_{p}$). That is, we sum the joint heatmaps in groups. As shown in Figure 3, three ways of grouping are introduced, i.e., by part, by level, and by tip grouping. Part grouping integrates joint landmarks on a finger, leg, arm, or torso, while level grouping integrates joint landmarks at the kinematic level [44]. Tip grouping integrates pairwise part tips. As a result, group($\mathbf{H}_{p}$) forms sub-poses that exploit high-level semantic relation of 2D joints.

The second phase of our pipeline generates the root-relative 3D mesh from the aggregated 2D cues using an improved spiral convolution decoder.

A 3D mesh $\mathcal{M}$ contains vertices $\mathcal{V}=\{\mathbf{v}_{i}=(x_{i},y_{i},z_{i})\}_{i=1}^{M}$ and faces $\mathcal{F}$. Convolution methods for $\mathcal{M}$ essentially process vertex features $f(\mathbf{v})$. SpiralConv [24] is a graph-based convolution operator, which processes vertex features in the spatial domain. By explicitly formulating the order of aggregating neighboring vertices, SpiralConv++ [10] presents an efficient version of SpiralConv. SpiralConv++ depends on a spiral manner of neighbor selection and adopts a fully-connected layer for feature fusion:

where $\mathbb{N}$ represents vertex neighborhood, $W$ and $b$ are weights and bias shared for $\forall\mathbf{v}\in\mathcal{V}$.

As shown in Figure 4(left), SpiralConv++ collects vertex neighbors (black dots) of the cell (red star) in a spiral manner. Then these neighbors are treated indiscriminately by a fully connected layer. Inspired by Inception and residual models [13, 38], we design an Inception Spiral Module (ISM) to enhance the receptive field of SpiralConv. Specifically, as shown in Figure 4(right), we distinguish neighbors according to the spiral hierarchy (red, orange, and green dots) and adopt parallel layers with diverse receptive field for 3D decoding. The ISM can be described as

where $[\cdot]$ denotes concatenating. In ISM we keep the number of parameters manageable by controlling the channel size of $o$. Note that the $i\text{-disk}$ for each vertex may contain different number of elements. Similar to SpiralConv++, we truncate it to obtain a fixed-length sequence so that $W_{i}$ and $b_{i}$ can be shared for all the vertices.

The overall architecture of our 3D mesh decoder is shown in Figure 5. Our design improves the spiral decoder in three aspects: (1) we replace SpiralConv with ISM; (2) we leverage multi-scale prediction and coarse-to-fine fusion; and (3) we introduce a self-regression mechanism by concatenating scale-level predictions with the same-scale feature. Meanwhile, we use a convolutional decoder which runs in parallel with the spiral decoder to refine the estimation of 2D pose and silhouette.

The silhouette reflects holistic 3D-to-2D projection and contains strong geometric information for root recovery. Given the intrinsic matrix $K$ of the camera, predicted 3D vertices $\mathcal{V}$ can be projected into the 2D space by $K\mathcal{V}$, resulting in a 2D mesh which consists of 2D vertices $\mathcal{V}^{2D}=\{\mathbf{v}^{2D}_{i}=(x^{2D}_{i},y^{2D}_{i})\}_{i=1}^{M}$ with the original connectivity. Ideally, the 2D mesh should align well with the silhouette. However, they cannot be directly aligned because (1) the large number of 2D points leads to prohibitive computational cost, i.e., the silhouette contains thousands of pixels while the 2D mesh has 778 (for hand) or 6,890 (for human body) vertices; (2) 2D mesh vertices have no explicit correspondence with respect to the silhouette.

To overcome these difficulties, we design a group of 1D projections to align the silhouette with the 2D mesh. First, the silhouette is converted into contours $\mathcal{C}=\{\mathbf{c}=(x^{c}_{i},y^{c}_{i})\}_{i=1}^{C}$ by edge detection [5]. We define a set of 1D axes $\mathcal{A}=\{\mathbf{a}_{j}=(x^{a}_{j},y^{a}_{j})\}_{j=1}^{A}$. These axes have unit length and are uniformly distributed, i.e., the angle difference between neighboring axes is $\pi/A$. As shown in Figure 6, we project contours and 2D vertices onto $\mathbf{a}_{j}$, resulting in their 1D span, i.e., $\mathcal{S}^{\mathcal{C}}_{j}=\{\mathbf{c}_{i}\cdot\mathbf{a}_{j}\}_{i=1}^{C}$ and $\mathcal{S}^{\mathcal{V}^{2D}}_{j}=\{\mathbf{v}^{2D}_{i}\cdot\mathbf{a}_{j}\}_{i=1}^{M}$.

We denote the camera-space root by $\mathbf{t}=(x^{r},y^{r},z^{r})$ and 2D joint landmark predictions by $\mathcal{P}=\{\mathbf{p}_{i}=(x^{p}_{i},y^{p}_{i})\}_{i=1}^{N}$. Given $K$ and joint regressor $J$ defined by MANO [35] or SMPL [25], the camera-space 3D vertices $\mathcal{V}+\mathbf{t}$ can be converted into 2D joints by $\mathcal{Q}=KJ\mathcal{(}\mathcal{V}+\mathbf{t})=\{\mathbf{q}_{i}=(x^{q}_{i},y^{q}_{i})\}_{i=1}^{N}$. Since $\mathcal{P}$ and $\mathcal{Q}$ have intrinsic correspondence, we define the energy function for 2D matching as

whose solution is denoted as $\mathbf{t}^{2D}=\mathop{\arg\min}_{\mathbf{t}}E_{2D}(\mathbf{t})$. Further, we use $\mathcal{V}+\mathbf{t}$ to implement the aforementioned 3D-2D-1D projection. Camera-space $\mathcal{S}^{\mathcal{V}^{2D}}$ is thereby produced. Then the 1D correspondence is obtained using two endpoints of the 1D span. The energy function for alignment of 1D spans is defined as

In 1D space, $\mathbf{t}^{1D}=\mathop{\arg\min}_{\mathbf{t}}E_{1D}(\mathbf{t})$. Both 2D and 1D optimizations are based on quadratic programming [2]. With $\mathbf{t}^{1D}$ and $\mathbf{t}^{2D}$, we develop an adaptive method for the final root $\mathbf{t}^{*}$ with their distance $d=||\mathbf{t}^{2D}-\mathbf{t}^{1D}||_{2}$:

where $\delta_{1}>\delta_{2}$, both of which are robust hyper-parameters according to 3D scale. Without wide search, we empirically use $0.06,0.02$ for the hand and $1.0,0.5$ for the body. This design attempts to sufficiently leverage the merits of 2D-1D registration. It is known that 2D pose is more fragile than silhouette. Hence, 2D pose is prone to be erroneous when there is a huge 2D-1D discrepancy, and silhouette is more dependable if $d>\delta_{1}$. From another perspective, joint correspondence is more explicit than that of 1D projection. Thereby, the 2D process is more reliable if 2D and 1D results are similar ($d<\delta_{2}$). The whole process of adaptive 2D-1D registration is presented in Figure 7, from which we can see that geometrical information is sufficiently exploited from 3D to 1D spaces for root recovery.

Our backbone is based on ResNet [13] and we use the Adam optimizer [19] to train the network with a mini-batch size of $32$. Serving as network inputs, image patches are cropped and resized to resolutions of $\text{224}\times\text{224}$ (for the hand) or $\text{256}\times\text{256}$ (for the body). Because of different data amounts, the total number of iterations is set as 38 and 25 epochs for tasks on hand and body. The initial learning rate is $10^{-4}$, which is divided by $10$ at the 20th or 30th epoch. Data augmentation includes random box scaling/rotation, color jitter, etc.

Our ablation studies are based on ResNet18. YoutubeHand [22] serves as the baseline, but its code and models are inaccessible. Our re-implemented model obtains $9.06$mm PA-MPVPE (see Table 1). With our spiral decoder, $8.54$mm PA-MPVPE is achieved. Thus, our designs can strengthen the robustness of 2D-to-3D decoding.

This paper explores cues of 2D pose and silhouette for 3D mesh recovery, and a two-stack network is leveraged. At first, discarding the second stack, we only use one encoder-decoder structure for exposing details of 2D-cue effects. As shown in Table 2, $\mathbf{H}_{s}$ leads to $8.10$mm PA-MPVPE while sum($\mathbf{H}_{p}$) reduces PA-MPVPE to $7.94$mm. Both $\mathbf{H}_{s}$ and sum($\mathbf{H}_{p}$) are heatmaps that contain location information, so the locations of joint landmarks are more instructive than that of a holistic shape. This phenomenon is evident, because pose is more difficult to estimate than shape, and joint locations can directly provide cues on poses. Furthermore, $\mathbf{H}_{p}$ leads to an improved PA-MPVPE of $7.77$mm by providing both joint locations and semantics. Compared to sum($\mathbf{H}_{p}$), it is seen that joint semantics is also important. In addition, the effect of cat($\mathbf{H}_{p},\mathbf{H}_{s}$) is worse than that of $\mathbf{H}_{p}$. Therefore, there is no complementary benefit from 2D pose and silhouette.

To reveal potential reasons for different effects of 2D cues, we dissect feature representation acted by them. It is known that channel-specific feature usually embeds semantics, so we focus on typical channels in the first encoding block in the 3D mesh recovery phase. As shown in Figure 8, this block tends to describe trivial properties such as edges (see “none”). If $\mathbf{H}_{s}$ is employed, holistic 2D shapes emerge, but pose cues are ignored. With sum($\mathbf{H}_{p}$), holistic 2D shapes still can be learned based on joint landmark locations. Thus, this phenomenon can be the reason why there exists no complementary benefit in joint landmarks and silhouette. Moreover, holistic joint locations can also be captured with sum($\mathbf{H}_{p}$). As for $\mathbf{H}_{p}$, although joint information is provided in separate heatmaps, the features after $\mathbf{H}_{p}$ have a tendency that multiple joints are simultaneously activated. Features shown in $\mathbf{H}_{p}$ of Figure 8 imply semantic relation of joints, which can have significant impact on 3D tasks. However, relation representation invited by $\mathbf{H}_{p}$ is not exhaustive enough. Specifically, the left two representations have similar patterns while both the 3rd and 4th representations mainly focus on the tips of thumb and middle finger.