Monocular Human-Object Reconstruction in the Wild

Human-Object Keypoints. The raw images captured from the real world contain rich color and geometry information about the arrangement of the human and the object on the 2D image plane. There are several ways to encode the geometry information of the human and the object from unstructured images, such as masks, sparse keypoints and dense coordinates. To make a good balance between computation efficiency and geometry informality, we choose to use the sparse keypoints to represent the human and the object. For the human, we use the joints in the SMPL model as the keypoints. In the SMPL model coordinate system, the keypoints of human $\mathbf{X}_{\text{3D}}^{\text{SMPL}}\in\mathbb{R}^{22\times 3}$ can be computed as follows:

where $\mathcal{M}$ is the blending function which maps the shape parameter $\bm{\beta}\in\mathbb{R}^{10}$ and the pose parameter $\bm{\theta}\in\mathbb{R}^{63}$ to the $6890$ vertices of the SMPL model and $\mathcal{J}\in\mathbb{R}^{22\times 6890}$ is the joints weighting matrix. For the object, the keypoints are manually selected from the vertex on the surface of the object mesh model, which is based on the geometric characteristic of the object shape. Denote the localization of the 3D object keypoints under the object local coordinate system as $\hat{\mathbf{X}}_{\text{3D}}^{\text{object}}\in\mathbb{R}^{t\times 3}$, where $t$ is the number of the object keypoints. Under the SMPL local coordinate system, the localization of the object keypoints is computed by

where $s$ is the size of the object and $\mathbf{R},\mathbf{t}$ is the 6D pose of the object under the SMPL coordinate system. The coordinates of joints of SMPL and the selected object keypoints are concatenated together to get the representation for the 3D human-object spatial arrangement

In this keypoint-based represetation, the parameters $\{\bm{\beta},\bm{\theta},\mathbf{R},\mathbf{t},s\}$ are transformed into sparse keypoints $\mathbf{X}_{\text{3D}}$ under the SMPL local coordinate system.

The Bridge between 2D and 3D. Under the perspective camera model, any point $\mathbf{x}_{\text{3D}}\in\mathbf{X}_{\text{3D}}$ is projected onto the image plane obtaining the 2D centered coordinates $(u,v)$. The projection relationship between the two is determined by the following equation.

where $f$ is the focal length of the camera, $\mathbf{R}_{\text{SMPL}}$ and $\mathbf{t}_{\text{SMPL}}$ is the pose of the SMPL model under the camera coordinate system, $\lambda$ is the depth scale. To smplify the notation in following equation, let $\mathbf{x}_{\text{2D}}=(u,v,f)^{\rm{T}},\mathbf{R}_{\text{cam}}=\mathbf{R}_{\text{SMPL}}^{-1},\mathbf{t}_{\text{cam}}=-\mathbf{R}_{\text{SMPL}}^{-1}\mathbf{t}_{\text{SMPL}}$. Rearrange the terms and normalize both sides, we get

Equation (6) builds the relation between the 3D keypoints $\mathbf{x}_{\text{3D}}$ and the coordinate $(u,v)$ on the image plane. Given the observation point $(u,v)$ in the image plane and the pose of the camera $\{\mathbf{R}_{\text{cam}},\mathbf{t}_{\text{cam}}\}$, the corresponding 3D point $\mathbf{x}_{\text{3D}}$ lies on th ray with the direction $\mathbf{d}=\frac{\mathbf{R}_{\text{cam}}\mathbf{x}_{\text{2D}}}{\|\mathbf{R}_{\text{cam}}\mathbf{x}_{\text{2D}}\|}$ staring from $\mathbf{t}_{\text{cam}}$. Compute the direction vector $\mathbf{d}$ for each point in 3D human-object keypoint $\mathbf{X}_{\text{3D}}$ according to the right side of equation (6) and concatenate them with $\mathbf{t}_{\text{cam}}$ obtaining the intermediate representation

which is the bridge between the 3D keypoints $\mathbf{X}_{\text{3D}}$ and the projected coordinates $\Pi_{\rho}(\mathbf{X}_{\text{3D}})$ with the camera pose $\rho=\{\mathbf{R}_{\text{cam}},\mathbf{t}_{\text{cam}}\}$.

The Structure of the Normalizing Flow. The nomalizing flow transform the measurements $\mathbf{X}_{\text{2.5D}}$ to a sample $\mathbf{z}$ in the gaussian distribution with $\mathbf{f}$ as the condition, i.e.

where $\mathbf{f}$ is the visual feature extracted from the input image $\mathbf{I}$. The structure of the normalizing flow $\mathcal{F}$ is constructed using actnorm layer, invertible $1\times 1$ convolution layer, and the affine coupling layer shown in (Kingma and Dhariwal, 2018). The density distribution of $\mathbf{X}_{\text{2.5D}}$ for the image $\mathbf{I}$ is given by

where $q$ is the density function of the Gaussian distribution.

The Training Process of the Normalizing Flow. The training objective of the normalizing flow is minimizing the negative log-likelihood, i.e.

However, the camera distribution $\varrho$ and the 2D projection $\Pi_{\rho}(\mathbf{X}_{\text{3D}})$ under each viewport is unknown for each image. In the 2D image dataset $\mathcal{D}=\{\mathbf{I}_{1},\mathbf{I}_{2},\dots\}$ collected from the Internet, each image has one viewport $\rho$ and one 2D projection $\Pi_{\rho}(\mathbf{X}_{\text{3D}})$ naturally, which is apparently insufficient to train the normalizing flow. To get other views and the corresponding 2D projections for each image, we group these images using the k-nearest neighbor grouping algorithm (Dong et al., 2011) based on the distance metric defined following

where $\mathbf{X}_{\text{2.5D}}=\left(\mathbf{d}_{1}^{\rm{T}},\mathbf{d}_{2}^{\rm{T}},\dots,\mathbf{d}_{n}^{\rm{T}},\mathbf{t}_{\text{cam}}^{\rm{T}}\right)^{\rm{T}}$ is the intermediate representation calculated from the 2D keypoints $\Pi_{\rho}(\mathbf{X}_{\text{3D}})$ and the camera pose $\rho=\{\mathbf{R}_{\text{cam}},\mathbf{t}_{\text{cam}}\}$ for image $\mathbf{I}$ according to the left side of equation (6) and $\mathbf{X}_{2.5D}^{\prime}=\left(\mathbf{d}_{1}^{\prime\rm{T}},\mathbf{d}_{2}^{\prime\rm{T}},\dots,\mathbf{d}_{n}^{\prime\rm{T}},\mathbf{t}_{\text{cam}}^{\prime\rm{T}}\right)^{\rm{T}}$ is the intermediate representation for the image $\mathbf{I}^{\prime}$. Equation (11) calculates the average distance between two rays which starting from $\mathbf{t}_{\text{cam}}$ and $\mathbf{t}_{\text{cam}}^{\prime}$ with the direction $\mathbf{d}$ and $\mathbf{d}^{\prime}$ respectively. The smaller the value of $d(\mathbf{I},\mathbf{I}^{\prime})$, the more likely that these rays intersect with each other in 3D space, which indicates that the image $\mathbf{I}$ and $\mathbf{I}^{\prime}$ capture the same interaction. The grouping process is totally free of human intervention and the algorithm outputs the top-k nearest neighbor for each image $\mathbf{I}$ in the form of cluster

where $d_{i}$ is the distance between the image $\mathbf{I}$ and the $i$-th item in the cluster $\mathcal{G}_{\mathbf{I}}$. We drop the neighbors whose distance exceeds a threshold. During training, we random select a batch of neighbors from $\mathcal{G}_{\mathbf{I}}$ for each image $\mathbf{I}$ to minimize the loss objective shown in equation (10).

Initialization. We first use the state-of-the-art 3D human mesh recovery model SMPLer-X (Cai et al., 2023) to predict the shape parameter $\bm{\beta}$, the pose parameter $\bm{\theta}$ and the gobal pose $\{\mathbf{R}_{SMPL},\mathbf{t}_{\text{SMPL}}\}$ for SMPL and the pre-trained CDPN (Li et al., 2019) to obtain the 6D pose $\{\mathbf{R},\mathbf{t}\}$ of the object. The scale $s$ of the object template is initialized empirically according to the size of the category in the real world. Besides we extract the keypoints for the human using ViTPose (Xu et al., 2022) and the 2D-3D corresponding maps using the pre-trained CDPN (Li et al., 2019).

Prior Loss. The prior learned from 2D images is flexible enough to be deployed to tune the parameter $\bm{\beta},\bm{\theta}$ of SMPL and the 6D pose $\mathbf{R},\mathbf{t}$ of the object during post-optimization. Given the input image $\mathbf{I}$, we draw $m$ samples from the distribution $p(\Pi_{\rho}(\mathbf{X}_{\text{3D}}),\rho|\mathbf{I})$ learned by the normalizing flow to initialize the translation poses $\{\hat{\mathbf{t}}_{\text{cam}}^{(1)},\dots,\hat{\mathbf{t}}_{\text{cam}}^{(m)}\}$ for each virtual camera. Then the 3D human-object keypoints $\mathbf{X}_{\text{3D}}$ computed from $\{\bm{\beta},\bm{\theta},\mathbf{R},\mathbf{t},s\}$ according to equation (2) and (3) are then projected onto the image planes of each virtual camera to get the intermediate representation $\{\hat{\mathbf{X}}_{\text{2.5D}}^{(1)},\dots,\hat{\mathbf{X}}_{\text{2.5D}}^{(m)}\}$ according to the right side of equation (6) and equation (7). The multi-view keypoints prior loss is defined as

The camera translation poses $\{\hat{\mathbf{t}}_{\text{cam}}^{(1)},\dots,\hat{\mathbf{t}}_{\text{cam}}^{(m)}\}$ are treated as the optimization parameters which are optimized together with $\{\bm{\beta},\bm{\theta},\mathbf{R},\mathbf{t},s\}$ during post-optimization process.

Contact Loss. In addition to constraining the 3D human-object keypoints, we also use the contact loss to generate more fine-grained interaction following PHOSA (Zhang et al., 2020). The contact map is also hard to be acquired in the wild. The contact between the human and the object in 3D space will result in occlusion in the 2D image plane, and inversely, the contact map can be approximated from occlusion in different views. Based on this idea, we approximate it using the average occlusion map. Denote the occlusion map for the human as the binary array $\mathbf{c}_{\text{h}}\in\mathbb{R}^{6890}$ where the element is set to 1 if the corresponding projected coordinate in the image plane falls within the occlusion region with the object. The occlusion map $\mathbf{c}_{\text{o}}$ for the object is defined similarly. For each image from the dataset, we calculate the occlusion maps for the human and the object and average them to get the mean occlusion map $\bar{\mathbf{c}}_{\text{h}}$ and $\bar{\mathbf{c}}_{\text{o}}$. During optimization, we compute the occlusion map for the human and the object from the target image and multiply them with the mean occlusion maps to get the candidate indices set of the points that are under contact. The contact index set for SMPL mesh is given by $\mathcal{C}_{\text{h}}=\{i|[\mathbf{c}_{\text{h}}]_{i}\cdot[\bar{\mathbf{c}}_{\text{h}}]_{i}>\eta\}$ and the contact index set for object mesh is given by $\mathcal{C}_{\text{o}}=\{i|[\mathbf{c}_{\text{o}}]_{i}\cdot[\bar{\mathbf{c}}_{\text{o}}]_{i}>\eta\}$, where $\eta$ is contact threshold. The contact loss is defined as the weighted chamfer distance between the two contact point clouds decided by the index set $\mathcal{C}_{\text{h}}$ and $\mathcal{C}_{\text{o}}$.

where $w_{ij}=[\mathbf{c}_{\text{h}}\cdot\bar{\mathbf{c}}_{\text{h}}]_{i}[\mathbf{c}_{\text{o}}\cdot\bar{\mathbf{c}}_{\text{o}}]_{j}$, $\mathbf{p}_{i}^{\text{h}}$ is the $i$-th point in SMPL mesh and $\mathbf{p}_{j}^{\text{o}}$ is the $j$-th point in the object mesh.

Optimization Objective. The overall optimization objective is defined as

where $\mathcal{L}_{\text{J}}$ is the reprojection loss for SMPL keypoints, $\mathcal{L}_{\text{coor}}$ is the reprojection loss for the object defined in (Huo et al., 2023) and $\mathcal{L}_{\text{norm}}$ is the regularization term for the pose of the human and the scale of the object. Our optimization process consists of two phases. In the first phase, we lock the parameters for SMPL and only optimize the 6D pose $\{\mathbf{R},\mathbf{t}\}$ and the scale $s$ of the object. In the second phase, we tuned all the parameters together by minimizing the optimization objective (3.3).

Dataset and Metrics. To compare our method with previous 3D supervised methods, we conduct experiments on the BEHAVE dataset (Bhatnagar et al., 2022). The BEHAVE dataset is a large-scale 3D full-body human-object interaction dataset that contains 8 subjects interacting with 20 common objects. This dataset is captured by four Kinect RGB-D cameras at 30 FPS. Each image is annotated with pseudo SMPL labels, 6D object pose labels, camera poses, and camera calibration parameters. In experiments, we follow the official splits, using 217 video sequences for training and 82 video sequences for testing. To speed up the test process, we only test on the key frames (Xie et al., 2023). In terms of evaluation metrics, we use the chamfer distance between the reconstructed 3D mesh with the ground truth. Before calculating the chamfer distance, 10,000 points are sampled from the surface of the mesh. The point clouds sampled from the surface of the reconstructed mesh and the ground-truth mesh are aligned using optimal Procrustes alignment and then the chamfer distance between these two point clouds is calculated. The chamfer distances for the human and the object are reported separately.

Implementation Details. Since our method is very sensitive to the accuracy of the 2D human-object keypoints, we use the 2D human-object keypoints rendered from the ground truth and the camera pose obtained from the 3D annotations to train our normalizing flow. Note that we only access the 2D human-object keypoints and the 6D camera pose without directly accessing the SMPL annotations and the object 6D pose labels during the training process of the normalizing flow which doesn’t violate the original 2D-supervised goal. The cluster size $k$ is set to 8 in the top-$k$ nearest neighbor grouping. The normalizing flow is trained objectwisely for 30 epochs with the learning rate set to $1e-4$. In the process of inference, we initialize the SMPL parameters and the object 6D pose using ProHMR (Kolotouros et al., 2021) and Epro-PnP (Chen et al., 2022) respectively. In post-optimization stage, $\lambda_{\text{J}},\lambda_{\text{coor}},\lambda_{\text{prior}}$ is set to 0.1, 0.1, 1 respectively. The number of the virtual camera $m$ is set to 8 by default. We didn’t use the contact loss $\mathcal{L}_{\text{contact}}$ in the post-optimization process on the BEHAVE dataset.

Baselines. We compare our method with previous 3D supervised methods CHORE (Xie et al., 2022) and StackFLOW (Huo et al., 2023). Since our method is supervised with 2D keypoints, it is unfair to compare our method with these methods trained with 3D annotations. We compare our method with theirs to see the marge between the 3D supervised methods and the 2D supervised methods.

Main Results. The comparison in terms of reconstruction accuracy between the 3D supervised methods and the 2D supervised methods is shown in table 1. Compared with the 3D supervised methods CHORE and StackFLOW, our method achieves almost comparable performance even if we don’t access the 3D annotation directly, which indicates that we find a lighter way to learn the human-object spatial relation prior without using the 3D annotations generated by the calibrated multi-view capture systems.

Ablation on Different Supervisions. In table 2, we show the impact of different supervisions on reconstruction accuracy. We train the normalizing flow in three ways: (1) directly using 3D human-object keypoints to train the normalizing flow, (2) training the normalizing flow using 2D human-object keypoints but with the ground truth grouping, (3) training the normalizing flow using 2D human-object keypoints and the grouping is constructed by the KNN algorithm. As shown in table 2, as we expected, the 3D supervision using 3D human-object keypoints (the second line in table 2) achieves the highest reconstruction accuracy. However. using 2D keypoints with ground truth (the third line in table 2) also yields favorable results, closely following the performance of direct 3D supervision. Finally, using 2D keypoints with grouping from the KNN algorithm (the last line in table 2) results in slightly lower accuracy but still performs reasonably well. This indicates that even without direct access to 3D annotations, it is possible to achieve comparable performance with a minimal drop in accuracy by utilizing 2D keypoints and appropriate grouping strategies.

Dataset and Metrics. The WildHOI dataset is used to evaluate the performance of our method on in-the-wild images. We evaluate our method using the chamfer distance but with a slight difference from the evaluation metric on the BEHAVE dataset. The reconstruction meshes and the ground-truth meshes are placed on the local coordinate system of SMPL where the pelvis joint of SMPL is rooted at the origin. The chamfer distance between the reconstruction mesh and the ground-truth mesh is calculated without using the optimal Procrustes alignment. Besides, we also use the rotation error and the translation error to evaluate the relative pose between the object and the human. In addition to the numerical results, we also conduct human evaluation. More details about human evaluation can be found in the supplementary materials.

Implementation Details. We employ the normalizing flow with a depth of 8 and a width of 512. The training time of the normalizing flow on each object category varies depending on the convergence of its loss. The cluster size $k$ is set to 16 in the top-$k$ nearest neighbor grouping algorithm. During post-optimization, $\lambda_{\text{J}}$ is set to 0.01, while $\lambda_{\text{prior}}$ and $\lambda_{\text{coor}}$ are set to 0.1, $\lambda_{\text{contact}}$ is set to 1. We use the corresponding maps generated using 6D pose labels to calculate the loss $\mathcal{L}_{\text{coor}}$ in equation (3.3).

Baselines. Most recent works are learning-based methods that require 3D annotations for training. Compared with them our 3D-free method will be unfair since we don’t have large-scale 3D annotations for in-the-wild images. We have to select the annotation-free method PHOSA (Zhang et al., 2020) as our main baseline. To make the comparison fairer, we adapt the PHOSA into our method by substituting the loss $\mathcal{L}_{\text{prior}}$ and $\mathcal{L}_{\text{contact}}$ in equation (3.3) with the coarse interaction loss $\mathcal{L}_{\text{coarse inter}}$, the fine interaction loss $\mathcal{L}_{\text{fine inter}}$ and the ordinal depth loss $\mathcal{L}_{\text{depth}}$ proposed in PHOSA with all the other settings the same with our methods.

Main Results. As shown in table 3, our method outperforms PHOSA in terms of all evaluation metrics, especially in terms of the translation error of the object. Because PHOSA and our method all used the same SMPL parameters predicted by SMPLer-X and the same correspondence maps of the object rendered from the object 6D pose labels, there is only a slight difference in chamfer distance of SMPL and the rotation error of the object. The improved performance of our method can be contributed by the human-object prior loss $\mathcal{L}_{\text{prior}}$. With this strong prior learned from vast 2D images, our method leads to lower translation error on objects and better overall performance compared to PHOSA.

Qualitative Results. As shown in figure 3, we compare the qualitative results of our method with PHOSA. From the qualitative results, we can see that our method can accurately reconstruct the spatial relation between the human and the object in different scenarios. Although the reconstruction results of PHOSA can align well with the image, the reconstruction is not coherent when observed from the side view. Moreover, our method can deal with non-contact interaction types, whereas PHOSA, which relies on the contact map to constrain the relative pose between the human and the object, fails in such cases of non-contact interaction.

Ablation on the Optimization Loss. Table 4 shows the results of ablation experiments on different optimization losses. From the results, we can see that including both the prior and contact losses leads to the best performance, achieving the lowest errors in all metrics except the chamfer distance for SMPL. This indicates that both the prior and contact losses play important roles in the optimization process. By comparing the third line and the last line, we can see that excluding the prior loss leads to a significant drop, which indicates that the prior loss has a more significant impact on reconstruction accuracy.