Distribution and Depth-Aware Transformers for 3D Human Mesh Recovery

Feature Encoding. The initial step involves passing the input image and depth map through a CNN backbone to extract pertinent features. Subsequently, to explicitly model the structure of the features, position embedding is applied to these extracted features.

Specifically, we implement a hybrid positional encoding ($P_{e}$) illustrated in Equation (1), for the image and depth tokens. This hybrid approach capitalizes on the strengths of both learnable position embeddings ($P_{l}$) and sinusoidal position embeddings ($P_{s}$). $P_{l}$ adapts to task-specific positional patterns, proving highly effective in capturing intricate spatial relationships. Meanwhile, $P_{s}$ contributes to the globally consistent positional understanding, capturing more information about the position. This combination optimally balances adaptability and global context, yielding fine-grained spatial patterns and general positional relationships.

where $\omega_{1}$ and $\omega_{2}$ are learnable parameters controlling the position embedding contribution of both types.

Transformer Encoder. The utilization of the transformer encoder in D2A-HMR is driven by the overarching goal of effectively learning pseudo-depth cues from the input data. Using self-attention mechanisms on the encoded features derived from both modalities (image and pseudo-depth map), namely $z_{img}$ and $z_{depth}$, the transformer encoder facilitates understanding of spatial relationships within each domain. Furthermore, we propose to use a cross-attention mechanism to establish intricate connections between the image and pseudo-depth information. The resulting fused representation, denoted as $z$, encapsulates rich depth cues, crucial for the subsequent regression of human vertices.

The embedded features, denoted as $z_{img}$ and $z_{depth}$, serve as input tokens to the transformer encoder, embodying our pursuit of learning pseudo-depth cues. Using self-attention mechanisms, the encoder refines $z_{img}$ and $z_{depth}$ by capturing spatial relationships within each modality, producing updated features $z_{img}^{\prime}$ and $z_{depth}^{\prime}$, respectively. Subsequently, the introduction of a cross-attention mechanism facilitates connections between image and pseudo-depth features. The resulting cross-attended tokens denoted as $z_{c}$, are then fused with $z_{img}^{\prime}$ and $z_{depth}^{\prime}$ from their respective attention heads, yielding a final fused representation denoted as $z$, as illustrated in Equation (2). To facilitate this fusion, learnable fusion gates are employed, similar to the position encoding methodology. These gates adaptively emphasize the importance of each source, enhancing the model’s capacity to capture meaningful relationships between the image and pseudo-depth features.

Here, in Equation 2, $\omega_{3}$ and $\omega_{4}$ are the learnable parameters. Once the fusion is done, $z$ is normalized and fed as input to an MLP to get the output tokens. This holistic approach enables our model to effectively capture intricate patterns and dependencies within the input image and the 3D information of the scene. A visual illustration of the transformer encoder is shown in Figure 2.

The refinement module in the D2A-HMR framework encompasses three key components, each designed to enhance the model’s capabilities in capturing different aspects of human pose and shape. First, the distribution matching component aids in refining the model’s representation by aligning the output mesh distribution to the ground truth mesh distribution. This adaptation enables the model to capture and adapt to inherent variations in the distribution of training data, promoting a more generalized performance that extends beyond the specific characteristics of the training data. The second component, the silhouette decoder, focuses on optimizing the model’s capacity to align the shape with the input image by adeptly capturing the outlines of the human subject. This component contributes significantly to the model’s ability to refine and improve its representation based on the visual cues present in the input data. Lastly, the masked modeling component serves to empower the model by learning from available information, thereby enhancing its ability to capture long-range relationships among features in the image. This integration ensures that the model can leverage relationships across the entire input, contributing to a more comprehensive understanding of the underlying human pose and shape.

Distribution Matching. To align the model with the underlying data distribution, we incorporate the RealNVP [29] normalizing flow mechanism within the D2A-HMR framework. This aims to refine the model by minimizing the discrepancy between predicted and ground truth mesh distributions. The transformer encoder’s output tokens z are passed through a MLP regressor (R), which utilize linear layers to predict the mean $\mu$ and standard deviation $\sigma$, controlling the position and scale of the initially assumed Gaussian distribution. The flow-modeled distribution ($P_{\phi}(\hat{x})$, where $\hat{x}$ is the predicted mesh) is deconstructed into three essential terms, as expressed in the equation:

The first term, $\log Q(\hat{x})$, quantifies the logarithmic probability of the data under the simple distribution. The second term, $\log\dfrac{P(\hat{x})}{c\cdot Q(\hat{x})}$, represents the residual log-likelihood, serving as the distinction between the log-probability of the data under the optimal underlying distribution and the log-probability under the tractable initial density function. The third term, $\log c$, functions as a normalization constant.

Silhouette Decoder. To optimize shape alignment, we used a specialized decoder to reconstruct silhouettes. Leveraging features from the transformer encoder, this decoder employs a sequence of deconvolution layers with ReLU activation and dropout, culminating in a fully connected layer. This reconstruction process significantly augments the model’s capability to generate high-quality silhouette representations. To acquire the pseudo-ground truth silhouette of human subjects, we utilize an existing segmentation technique [30].

Masked Modeling. Prior works including [31], [8], and [32] have demonstrated the efficacy of masked modeling in elucidating diverse relationships within training datasets, spanning textual, vertex, and image domains respectively. In alignment with these established works, we adopt random masking of the embedded features to recover the vertex of the human body. By deliberately obscuring a percentage of embedded features during training, our model is forced to rely solely on the unmasked features extracted from the image. This enables a comprehensive understanding of both short and long-range relationships among the features, contributing to the overall performance of D2A-HMR framework.

In this sub-section, we present the comprehensive training objectives employed to recover the human mesh in our model. These objectives consist of a weighted combination of various loss components, each serving a specific role in refining the model’s output.

The loss function $\mathcal{L}_{v}$ is computed using the loss metric $L_{1}$, with the aim of minimizing the disparities between the model’s output vertices with the ground truth vertice representation. Simultaneously, ${\mathcal{L}}_{3D}=|J_{3D}-J^{g}_{3D}|$ leverages the same loss metric to optimize the 3D pose by regression ($J_{3D}$) of the output mesh vertices following [8], seeking alignment with the ground truth pose coordinates ($J^{g}_{3D}$). To enhance the alignment between image and mesh representations, camera parameters are employed to reproject and infer the 2D human pose coordinates ($J_{2D}$) represented with $\mathcal{L}_{2D}=|J_{2D}-J^{g}_{2D}|$, where $J^{g}_{2D}$ is the 2D pose ground truth. This reprojected output is refined by applying loss optimization using $L_{1}$.

As mentioned in Section III-B, a distribution matching regularizer is used to penalize the model for predicting outputs that are unlikely under the underlying ground truth distribution. Equation (4) shows the distribution regularizer ($\mathcal{L}_{RLE}$) used in the D2A-HMR architecture.

Here, in Equation (4), $G_{\phi}(\bar{\mu_{g}})$ is the learned residual distribution of the predicted value $\bar{\mu_{g}}$ where $\bar{\mu_{g}}$ = $(\mu_{g}-\mu)/\sigma$. Here, $\mu_{g}$ is the ground truth distribution and $\phi$ is the flow model parameter. Additionally, we incorporate silhouette loss, denoted $\mathcal{L}_{silh}$, which regularizes the model by controlling the shape of the reconstructed mesh. The overall objective function is shown in Equation (5), which represents a combination of these individual losses.

where $\lambda_{d}$, $\lambda_{v}$, $\lambda_{3D}$, $\lambda_{2D}$ and $\lambda_{s}$ denote the weights attributed to the training objectives concerning the distribution, vertices, 3D pose coordinates, 2D pose coordinates, and silhouettes, respectively.

Training Details. Training was carried out on an infrastructure comprising three NVIDIA A6000 GPUs. The network was trained for 500 epochs, with a batch size of 48, and 24 parallel workers. Adam optimizer, configured with a learning rate of $10^{-4}$ and beta values of $0.9$ and $0.99$, was used for optimization. The network was designed to output a coarse mesh representation containing $431$ vertices. This output was subsequently upsampled [19] to the original mesh’s $6890$ vertices, utilizing learnable MLP layers, resulting in the model’s ability to capture fine-grained spatial details.

Datasets. Following previous work, we used two prominent 3D human pose estimation datasets, namely 3D Poses in the Wild (3DPW) [38] and Human3.6M [39] to train our D2A-HMR model. For the 3DPW dataset, we follow the standard practice of splitting the dataset into a training set of 22,000 images and a test set of 35,000 images. In the case of Human3.6M, we trained our D2A-HMR model on subjects S1, S5, S6, S7, and S8 and conducted testing on subjects S9 and S11. These data configurations were aligned with the common training and evaluation settings within the domain [8, 5]. The qualitative evaluation of the model was done in Leeds Sports Pose (LSP) [40], and various dedicated sports datasets including the MLBPitchDB dataset [41] and HARPE dataset [42].

Evaluation Metrics. In line with established practices from previous research [11, 8, 7], we subjected our model to a comprehensive evaluation using key metrics: mean per joint position error (mPJPE), procrustes-aligned mean per joint position error (PA-mPJPE) and per vertex error (mPVE) in both the 3DPW and Human3.6M datasets. mPVE metric is ignored if the ground truth mesh is not available. All metrics were measured in millimeters (mm), providing a precise assessment of our model’s performance.

We assess the performance of the proposed D2A-HMR framework by comparing it with established state-of-the-art techniques for HMR. The results, presented in Table I, highlight the competitive performance of our method across various metrics on the Human3.6M and 3DPW datasets. The comparative results demonstrate that the meshes generated by the D2A-HMR framework exhibit superior alignment with the input image. Our method’s adept understanding of pseudo-depth cues and the distribution contributes significantly to improved alignment, particularly in handling challenging input scenarios characterized by depth ambiguities and extreme poses.

Table II presents a comprehensive comparison between our proposed method and established state-of-the-art HMR techniques, utilizing the baseball dataset [41]. Notably, D2A-HMR demonstrates superior performance in terms of accuracy and mPJPE on this dataset, which is characterized by high player motion blur and instances of self-occlusion.

Figure 3 visualizes the effectiveness of our approach in handling these complexities, highlighting its robustness to unseen poses. To further emphasize the efficacy of our proposed approach, we conducted a qualitative comparison against several state-of-the-art techniques on unseen poses, as depicted in Figure 4. This comparative analysis underscores D2A-HMR’s potential for tackling challenging real-world scenarios.

To verify the individual impact of each module on the proposed D2A-HMR model, comprehensive studies were conducted, as detailed in this sub-section. For consistency across all studies, the 3DPW dataset was utilized as the common benchmark.

Integration of multi-modal data. Experimentation to assess the impact of depth and distribution matching components within the D2A-HMR are detailed in Table III.

Incorporation of both the pseudo-depth and distribution modeling modules in the D2A-HMR framework is observed to lead to a substantial improvement in the overall performance of mesh recovery. This observation serves as confirmation that the underlying motivation behind the proposed framework is valid and aids in enhancing the model’s capabilities.

Depth on mPJPE(z). Experimentation on exclusively capturing the depth component of the regressed 3D joints in order to demonstrate its impact on the human pose is conducted in Table IV.

A notable enhancement in the z-axis of the reconstructed mesh is evident, as highlighted in Table IV. We computed mPJPE along the z-axis denoted as mPJPE(z), disregarding the components $x$ and $y$ of the reconstructed mesh. This experimentation validates that the incorporation of scene-depth information contributes to an improvement in HMR.

Silhouette and Masked Modeling. Table V illustrates the impact of the silhouette decoder and masked modeling used within the D2A-HMR framework.

The observations drawn from Table V highlight the beneficial impact of incorporating both the silhouette decoder and masked modeling modules in enhancing the model’s ability to disentangle the appearance and part-relationship of the person. While prior studies, such as [35], employ methods like explicit iterative optimization for mesh-to-image alignment, our silhouette decoder yields improved alignment outcomes compared to scenarios without the decoder. Thus, these modules are utilized during the training process of the D2A-HMR framework, contributing to its improved performance.

Backbones. A comprehensive analysis of D2A-HMR’s performance by investigating its behavior with various backbone architectures was conducted. To establish a strong baseline, we first trained two ResNet variants for 1000 epochs on the ImageNet dataset [43] for an image classification task. We also explored HRNet variants trained for 1000 epochs using the COCO dataset [44] for the classification task.

We observe that HRNet-w64 gives the most positive impact on feature extraction from both the image and depth maps compared to the ResNet backbones. This can be attributed to HRNet-w64’s effectiveness in capturing both local and global contexts through its multiresolution fusion representations, thereby enhancing the model’s ability to extract rich and informative features.