InterCap: Joint Markerless 3D Tracking
of Humans and Objects in Interaction

A long-standing goal of Computer Vision is to understand human actions from videos. Given a video people effortlessly figure out what objects exist in it, the spatial layout of objects, and the pose of humans. Moreover, they deeply understand the depicted action. What is the subject doing? Why are they doing this? What is their goal? How do they achieve this? To empower computers with the ability to infer such abstract concepts from pixels, we need to capture rich datasets and to devise appropriate algorithms.

Since humans live in a 3D world, their physical actions involve interacting with objects. Think of how many times per day one goes to the kitchen, grabs a cup of water, and drinks from it. This involves contacting the floor with the feet, contacting the cup with the hand, moving the hand and cup together while maintaining contact, and drinking while the mouth contacts the cup. Thus, to understand human actions, it is necessary to reason in 3D about humans and objects jointly.

There is significant prior work on estimating 3D humans without taking into account objects [4] and estimating 3D objects without taking into account humans [68]. There is even recent work on inserting bodies into 3D scenes such that their interactions appear realistic [65, 30, 16]. But there is little work on estimating 3D humans interacting with scenes and moving objects, in which the human-scene/object contact is explicitly modeled and exploited. To study this problem, we need a dataset of videos with rich human-object interactions and reliable 3D ground truth.

PROX [15] takes a step in this direction by estimating the 3D body in a known 3D scene. The scene mesh provides information that helps resolve pose ambiguities commonly encountered when a single camera is used. However, PROX involves only coarse interactions of bodies, static scenes with no moving objects, and no dexterous fingers. The recent BEHAVE dataset [3] uses multi-view RGB-D data to capture humans interacting with objects but does not include detailed hand pose or fine hand-object contact. Finally, the GRAB dataset [54] captures the kind of detailed hand-object and whole-body-object interaction that we seek but is captured using marker-based MoCap and, hence, lacks images paired with the ground-truth scene.

We argue that what is needed is a new dataset of RGB videos containing natural human-object interaction in which the whole body is tracked reliably, the hand pose is captured, objects are also tracked, and the hand-object contact is realistic; see Fig. 1. This is challenging and requires technical innovation to create. To that end, we design a system that uses multiple RGB-D sensors that are spatially calibrated and temporally synchronized. To this data we fit the SMPL-X body model, which has articulated hands, by extending the PROX [15] method to use multi-view data and grasping hand-pose priors. We also track the 3D objects with which the person interacts. The objects used in this work are representative of items one finds in daily life. We obtain accurate 3D models for each object  with a handheld Artec scanner. Altogether we collect 223 sequences (67,357 multi-view frames), with 10 subjects interacting with 10 objects.

The problem, however, is that separately estimating the body and objects is not sufficient to ensure accurate 3D body-object contact. Consequently, a key innovation of this work is to estimate these jointly, while exploiting information about contact. Objects do not move independently, so, when they move, it means the body is in contact. We define likely contact regions on objects and on the body. Then, given frames with known likely contacts, we enforce contact between the body and the object when estimating the body and object poses. The resulting method produces natural body poses, hand poses, and object poses. Uniquely, it provides detailed pseudo ground-truth contact information between the whole body and objects in RGB video.

In summary, our major contributions are as follows: (1) We develop a novel Motion Capture method utilizing multiple RGB-D cameras. It is relatively lightweight and flexible, yet accurate enough, thus suitable for data capture of daily scenarios. (2) We extend previous work on fitting SMPL-X to images to fit it to multi-view RGB-D data while taking into account body-object contact. (3) We capture a novel dataset that contains whole-body human motions and interaction with objects, as well as multi-view RGB-D imagery. Our data and code are available at https://intercap.is.tue.mpg.de.

There is a large literature on estimating 3D human pose and shape from images or videos [4, 44, 7, 25, 29, 57, 37, 41]. Here we focus on the work most closely related to ours, particularly as it concerns, or enables, capturing human-object interaction.

MoCap from Multi-view Videos and IMUs. Markerless MoCap from multi-view videos [31, 8, 22] is widely studied and commercial solutions exist (e.g., Theia Markerless). Compared with traditional marker-based MoCap, markerless offers advantages of convenience, applicability in outdoor environments, non-intrusiveness, and greater flexibility. However, traditional MoCap methods, both marker-based and markerless, focus on extracting a 3D skeleton. This is useful for biomechanics but our goal is to reason about body-scene contact. To enable that, we need to capture the body surface.

Various 3D human representations have been proposed, with recent work focused on learning a parametric mesh-based model of body shape from large-scale collections of 3D scans [2, 33, 50, 44, 43, 59, 42]. Here we use the SMPL-X model [44] because it contains fully articulated hands, which are critical for reasoning about object manipulation. The body parameters are often estimated by fitting the 3D generative model to various 2D cues like landmarks detected by Convolutional Neural Networks [6, 58, 39] or silhouettes [47, 60, 1]. Though effective, these monocular video-based methods suffer from depth ambiguity and occlusions. To address this issue, researchers have proposed to combine IMUs with videos to obtain better and more robust results [36, 45].

Many methods estimate 3D bodies from multi-view images but focus on skeletons and not 3D bodies [19, 24, 46, 55, 10, 9, 66]. Recent work addresses 3D body shape estimation from multiple views [22, 11, 67]. Most related to our work are two recent datasets. The RICH dataset [21], fits SMPL-X bodies to multi-view RGB videos taken both indoors and outdoors. The method uses a detailed 3D scan of the scene and models the contact between the body and the world. RICH does not include any object motion; the scenes are completely rigid. In contrast, BEHAVE [3] contains SMPL bodies interacting with 3D objects that move. We go beyond that work, however, to integrate novel contact constraints and to capture hand pose, which is critical for human-object interaction. Additionally, BEHAVE focuses on large objects like boxes and chairs, whereas we have a wider range of object sizes, including smaller objects like cups.

Human-Object Interaction. There has been a lot of work on modeling or analyzing human-object interactions [61, 13, 40, 48, 56, 14, 18, 26, 3]. A detailed discussion is out of the scope of this work. Here, we focus on modeling and analyzing human-object interaction in 3D space. Most existing work, however, only focuses on estimating hand pose [18, 14, 17, 49], ignoring the strong relationship between body motion, hand motion, and object motion. Recent work considers whole-body motion. For example, the GRAB [54] dataset provides detailed object motion and whole-body motion in a parametric body format (SMPL-X). Unfortunately, it is based on MoCap and does not include video. Here our focus is on tracking the whole-body motion, object motion, and the detailed hand-object contact to provide ground-truth 3D information in RGB video.

Joint Modeling of Humans and Scenes. There is some prior work addressing human-object contact in both static images and video. For example, PHOSA estimates a 3D body and a 3D object with plausible interaction from a single RGB image [63]. Our focus here, however, is on dynamic scenes. Motivated by the observation that natural human motions always happen inside 3D scenes, researchers have proposed to model human motion jointly with the surrounding environment [15, 51, 5, 62]. In PROX  [15] the contact between humans and scenes is explicitly used to resolve ambiguities in pose estimation. The approach avoids bodies interpenetrating scenes while encouraging contact between the scene and nearby body parts. Prior work also tries to infer the most plausible position and pose of humans given the 3D scene [65, 30, 16]. Most recently, MOVER [62] estimates the 3D scene and the 3D human directly from a static monocular video in which a person interacts with the scene. While the 3D scene is ambiguous and the human motion is ambiguous, by exploiting contact, the method resolves many ambiguities, improving the estimates of both the scene and the person. Unfortunately, this assumes a static scene and does not model hand-object manipulation.

Datasets. Traditionally, MoCap is performed using marker-based systems inside lab environments. To capture object interaction and contact, one approach uses MoSh [32] to fit a SMPL or SMPL-X body to the markers [35]. An advanced version of this is used for GRAB [54]. Such approaches lack synchronized RGB video. The HumanEva [52] and Human3.6M [23] datasets combine multi-camera RGB video capture with synchronized ground-truth 3D skeletons from marker-based MoCap. These datasets lack ground-truth 3D body meshes, are captured in a lab setting, and do not contain human-object manipulation. 3DPW [36] is the first in-the-wild dataset that jointly features natural human appearance in video and accurate 3D pose. This dataset does not track objects or label human-object interaction. PiGraphs [51] and PROX [15] provide both 3D scenes and human motions but are relatively inaccurate, relying on a single RGB-D camera. This makes these datasets ill-suited as evaluation benchmarks. The recent RICH dataset [21] addresses many of these issues with indoor and outdoor scenes, accurate multi-view capture of SMPL-X, 3D scene scans, and human-scene contact. It is not appropriate for our task, however, as it does not include object manipulation.

An alternative approach is the one of GTA-IM [5] and SAIL-VOS [20], which generate human-scene interaction data using either 3D graphics or 2D videos. They feature high-accuracy ground truth but lack visual realism. In summary, we believe that a 3D human-object interaction dataset needs to have accurate hand poses to be useful, since hands are how people most often interact with objects. We compare our InterCap dataset with other ones in Tab. 1.

Our core goal is to accurately estimate the human and object motion throughout a video. Our markerless motion capture method is built on top of the PROX-D method of Hassan et al. [15]. To improve the body tracking accuracy we extend this method to use multiple RGB-D cameras; here we use the latest Azure Kinect cameras. The motivation is that multiple cameras observing the body from different angles give more information about the human and object motion. Moreover, commodity RGB-D cameras are much more flexible to deploy out of controlled lab scenarios than more specialized devices.

The key technical challenge lies in accurately estimating the 3D pose and translation of the objects while a person interacts with them. In this work we focus on $10$ variously sized rigid objects common in daily life, such as cups and chairs. Being rigid does not make the tracking of the objects trivial because of the occlusion by the body and hands. While there is a rich literature on 6 DoF object pose estimation, much of it ignores hand-object interaction. Recent work in this direction is promising but still focuses on scenarios that are significantly simpler than ours, cf. [53].

Similar to previous work on hand and object pose estimation [14] from RGB-D videos, in this work we assume that the 3D meshes of the objects are known in advance. To this end, we first gather the 3D models of these objects from the Internet whenever possible and scan the remaining objects ourselves. To fit the known object models to image data, we first preform semantic segmentation, find the corresponding object regions in all camera views, and fit the 3D mesh to the segmented object contours via differentiable rendering. Since heavy occlusion between humans and objects in some views may make the segmentation results unreliable, aggregating segmentation from all views boosts the object tracking performance.

In the steps above, both the subject and object are treated separately and processing is per frame, with no temporal smoothness or contact constraint applied. This produces jittery motions and heavy penetration between objects and the body. Making matters worse, our human pose estimation exploits OpenPose for 2D keypoint detection, which struggles when the object occludes the body or the hands interact with it. To mitigate this issue and still get reasonable body, hand and object pose in these challenging cases, we manually annotate the frames where the body or the hand is in contact with the object, as well as the body, hand and object vertices that are most likely to be in contact. This manual annotation can be tedious; automatic detection of contact is an open problem and is left for future work. We then explicitly encourage the labeled body and hand vertices to be in contact with the labeled object vertices. We find that this straightforward idea works well in practice. More details are described in the following.

We use the proposed InterCap algorithm (Sec. 3) to capture the InterCap dataset, which uniquely features whole-body interactions with objects in multi-view RGB-D videos.

Data-capture Protocol. We use 10 everyday objects, shown in Fig. 2-left, that vary in size and “afford” different interactions with the body, hands or feet; we focus mainly on hand-object interactions. We recruit 10 subjects (5 males and 5 females) that are between 25-40 years old. The subjects are recorded while interacting with $7$ or more objects, according to their time availability. Subjects are instructed to interact with objects as naturally as possible. However, they are asked to avoid very fast interactions that cause severe motion blur (Azure Kinect supports only up to $30$ FPS), or misalignment between the RGB and depth images for each Kinect (due to technicalities of RGB-D sensors). We capture up to 3 sequences per object depending on object shape and functionality, and by picking an interaction intent from the list below, as in GRAB [54]:

• •

  ”Pass”: The subject passes the object on to another imaginary person standing on their left/right side; a graspable area needs to be free for the other person to grasp.

• •

  ”Check”: The subject inspects visually the object from several viewpoints by first picking it up and then manipulating it with their hands to see several sides of it.

• •

  ”Use”: The subject uses the object in a natural way that “agrees” with the object’s affordances and functionality for everyday tasks.

We also capture each subject performing a freestyle interaction of their choice. All subjects gave informed written consent to publicly share their data for research.

4D Reconstruction. Our InterCap method (Sec. 3) takes as input multi-view RGB-D videos and outputs 4D meshes for the human and object, i.e., 3D meshes over time. Humans are represented as SMPL-X meshes [44], while object meshes are acquired with an Artec hand-held scanner. Some dataset frames along with the reconstructed meshes are shown in Fig. 1 and Fig. 4; see also the video on our website. Reconstructions look natural, with plausible contact between the human and the object.

Dataset Statistics. InterCap has 223 RGB-D videos with a total of 67,357 multi-view frames ($6$ RGB-D images each). For a comparison with other datasets, see Tab. 1.

Contact Heatmaps. Figure 5-left shows contact heatmaps on each object, across all subjects. We follow the protocol of GRAB [54], which uses a proximity metric on reconstructed human and object meshes. First, we compute per-frame binary contact maps by thresholding (at 4.5mm) the distances from each body vertex to the closest object surface point. Then, we integrate these maps over time (and subjects) to get “heatmaps” encoding contact likelihood. InterCap reconstructs human and object meshes accurately enough so that contact heatmaps agree with object affordances, e.g., the handle of the suitcase, umbrella and tennis racquet are likely to be grasped, the upper skateboard surface is likely to be contacted by the foot, and the upper stool surface by the buttocks.

Figure 5-right shows heatmaps on the body, computed across all subjects and objects. Heatmaps show that most of InterCap’s interactions involve mainly the right hand. Contact on the palm looks realistic, and is concentrated on the fingers and MCP joints. The “false” contact on the dorsal side is attributed to our challenging camera setup and interaction scenarios, as well as some reconstruction jitter.

Penetration. We evaluate the penetration between human and object meshes for all sequences of our dataset. We follow the protocol of GRAB et al. [54]; we first find the “contact frames” for which there is at least minimal human-object contact, and then report statistics for these. In Fig. 6-left we show the distribution of penetrations, i.e., the number of “contact frames” (Y axis) with a certain mesh penetration depth (X axis). In Fig. 6-right we show the cumulative distribution of penetration, i.e., the percentage of “contact frames” (Y axis) for which mesh penetration is below a threshold (X axis). Roughly $60\%$ of “contact frames” have $\leq 5mm$, $80\%$ $\leq 7$ mm, and $98\%$ $\leq 20$ mm mean penetration. The average penetration depth over all “contact frames” is $7.2$ mm.

Fitting Accuracy. For every frame, we compute the distance from each mesh vertex to the closest point-cloud (PCL) point; for each human or object mesh we take into account only the respective PCL area obtained with PointRend [28] segmentation. The mean vertex-to-PCL distance is 20.29 mm for the body, and 18.50 mm for objects. In comparison, PROX-D [15], our base method, achieves an error of 13.02 mm for the body. This is expected since PROX-D is free to change the body shape to fit each individual frame, while our method estimates a single body shape for the whole sequence. SMPLify-X [44] achieves an mean error of 79.54 mm, for VIBE the mean error is 55.59 mm, while ExPose gets an mean error of 71.78 mm. These numbers validate the effectiveness of our method for body tracking. Note that these methods are based on monocular RGB images only, so there is not enough information for them to accurately estimate the global position of the 3D body meshes. Thus we first align the output meshes with the point clouds, then compute the error. Note that the error is bounded from below for two reasons: (1) it is influenced by factory-design imperfections in the synchronization of Azure Kinects, and (2) some vertices reflect body/object areas that are occluded during interaction and their closest PCL point is a wrong correspondence. Despite this, InterCap empirically estimates reasonable bodies, hands and objects in interaction, as reflected in the contact heatmaps and penetration metrics discussed above.

Ablation of Contact Term. Figure 7-left shows results with-/out our term that encourages body-object contact; visualization“zooms” into hand-object grasps. We see that encouraging contact yields more natural hand poses and fewer interpenetrations. This is backed up by the contact heatmaps and penetration metrics discussed above.

Ablation of Temporal Smoothing Term. Figure 7-right shows results with-/out our temporal smoothing term. Each solid line shows the acceleration of a randomly chosen vertex without the temporal smoothness term; we show 3 different motions. The dashed lines of the same color show the same motions with the smoothness term; these are clearly smoother. We empirically find that the learned motion prior of Zhang et al. [64] produces a more natural motion than handcrafted ones [22].

Discussion on Jitter. Despite the smoothing, some jitter is still inevitable. We attribute this to two factors: (1) OpenPose and Mask-RCNN are empirically relatively sensitive to occlusions and illumination (e.g., reflections, shadows, poor lighting); the data terms for fitting 3D models depend on these. (2) Azure Kinects have a reasonable synchronization, yet, there is still a small delay among cameras to avoid depth-camera interference; the point cloud “gathered” across views is a bit “patchy” as information pieces have a small time difference. The jitter is more intense for hands relatively to the body, due to their low image resolution, motion blur, and coarse point clouds. Despite these challenges, InterCap is a good step towards capturing everyday whole-body interactions with commodity hardware. Future work will study advanced motion priors.

Here we focus on whole-body human interaction with everyday rigid objects. We present a novel method, called InterCap, that reconstructs such interactions from multi-view full-body videos, including natural hand poses and contact with objects. With this method, we capture the novel InterCap dataset, with a variety of people interacting with several common objects. The dataset contains reconstructed 3D meshes for the whole body and the object over time (i.e., 4D meshes), as well as plausible contacts between them. In contrast to most previous work, our method uses no special devices like optical markers or IMUs, but only several consumer-level RGB-D cameras. Our setup is lightweight and has the potential to be used in daily scenarios. Our method estimates reasonable hand poses even when there is heavy occlusion between hands and the object. In future work, we plan to study interactions with smaller objects and dexterous manipulation. Our data and code are available at https://intercap.is.tue.mpg.de.

We thank Chun-Hao P. Huang, Hongwei Yi, Jiaxiang Shang, and Mohamed Hassan for helpful discussion of technical details. We thank Yuliang Xiu, Jinlong Yang, Victoria F. Abrevaya, Taylor McConnell, Galina Henz, Marku Höschle, Senya Polikovsky, Matvey Safroshkin and Tsvetelina Alexiadis for data collection and cleaning. We thank all the participants of our experiments, and Benjamin Pellkofer for IT and website support.