6-DoF Pose Estimation of Household Objects for Robotic Manipulation:
An Accessible Dataset and Benchmark

Because our goal is to support robotic manipulation research, it is imperative that the set of objects be: 1) somewhat realistic, 2) of the proper size and shape for grasping by a variety of robotic end effectors, and 3) accessible to researchers throughout the world. An emerging area in robotic manipulation is that of household robots for automating daily chores in a laundry room or kitchen. Such applications are crucial for facilitating aging-in-place, assisted living, and similar healthcare-related challenges. While we initially intended to scan real items from a grocery store, we quickly realized that such an approach comes with several fundamental limitations: 1) such objects are not widely available to researchers in other parts of the world, 2) the reflective metallic properties of many of these objects make for difficult scanning and rendering, and 3) the surface textures of real-world objects frequently change due to seasonal marketing campaigns. Again, it is not enough for us to provide a dataset for perception researchers to design and test their algorithms—we also want to support robotics researchers who need to leverage the latest perception research in their own labs. As a result, the trained networks from the former should be immediately usable by the latter.

To this end, we chose a set of 28 toy grocery objects, depicted in Fig. 1. These toys are widely available online and can be purchased for less than 60 USD total. Because they are not real grocery items, there are no issues with perishability or transporting of the contents. Moreover, these objects are of the proper size and shape for typical robotic grippers, with all objects having at least one dimension between 2.4 and 7.2 cm.

Each object was scanned with a low-cost EinScan-SE desktop 3D scanner to generate textured 3D object meshes. The scanning process introduced another limitation on both the size and materials of the objects, due to the inability of the scanner to handle objects larger than about 20 cm on any side. The software tends to produce fragmented texture maps, so we used Maya to further refine the textures. Fig. 2 shows a synthetic rendering of the 3D textured object models, rendered using NViSII, a Python-interfaced ray tracing library [17]. Textured object models are useful for generating training data, as in DOPE [4] or BlenderProc [18].

To acquire images for the dataset, we placed the real objects in ten different environments, with five object arrangements / camera poses per environment. The environments are shown in Fig. 3, and the object arrangements for one environment are depicted in Fig. 4. These 50 different scenes exhibit a wide variety of backgrounds, clutter, poses, and lighting. With some arrangements, objects are placed inside other containers (e.g., boxes, bags, or drawers) to provide additional clutter and partial occlusion. Both RGB and depth images were collected for each scene using an Intel RealSense D415 RGBD camera at full HD resolution ($1920\times 1080$). Images were captured from distances of 0.5 to 1.0 m, which are typical of robotic grasping.

Once the objects were arranged in an environment and the camera was set, we acquired multiple images with various lighting conditions by turning on/off the lights, opening window blinds, and so forth. Since the scene was static, the annotations (described below) do not change and thus required no additional work. In this manner we were able to collect a large variety of lighting conditions, see Fig. 5.

In total, the dataset contains 50 unique scenes, 238 images (an average of 4.8 lighting variations per scene), and 914 object poses. Images from two environments (“chair”) and (“window 1”) are made available with ground truth for a validation set, while the remaining scenes are reserved as the test set. Fig. 6 summarizes the object sizes, as well as the variety of object pose angles, estimated visibility, and distance to the camera across all pose instances in the entire dataset. Minimum and maximum object dimensions are in the ranges 2.4–7.2 cm and 6.8–25.0 cm, respectively. Average object visibility is 83%, with only 39% of objects in unoccluded poses ($>$95% visible). Objects are set in a wide variety of orientations, and object distances range from 39.0–141.0 cm with an average distance of 73.8 cm.

We annotated the dataset with ground truth object poses by manually identifying point correspondences between images and 3D textured object models. In the P$n$P version of our annotation tool, the annotator selects corresponding points in the 2D texture map of the object model and the image in which the object appears, after which the 3D model is aligned to the image by P$n$P with RANSAC. Alternatively, in the RGBD version, correspondences are made between the 3D textured model and the 2.5D RGBD depth map, with alignment made by Procrustes. The RGBD annotation tool is typically faster to operate but may suffer from noise or bias in depth measurements, while P$n$P is more error-prone along the projection ray from the camera to the object. Most annotations were performed with the RGBD tool, resorting to P$n$P only when the depth-based annotation was not satisfactory. All annotations were automatically refined by SimTrack [19]. Then, if necessary, the alignment was refined manually until the reprojected model was visually aligned with both the image and the depth map.

Following Hodaň et al. [20], we checked the depth calibration of the RealSense camera and found a small systematic error. We captured over 50 images of a checkerboard at distances between 0.5 and 2 m using the RealSense left-infrared camera. Checkerboard corners were automatically detected, then unprojected to 3D using P$n$P and the camera intrinsics. The resulting coordinates were compared with the corresponding 3D coordinates obtained from the 2.5D depth map. Fitting a scale factor of 0.9804 reduced the mean absolute difference between measurements from 19.3 to 7.6 mm. The depth images included in the dataset were scaled by this factor before registering to RGB.

Perhaps the most common and easily interpretable metric for evaluating pose estimates is the average distance (ADD) metric [21], which computes the mean pairwise distance between corresponding vertices in the 3D object model under the ground truth ($\bar{\mathbf{P}}$) and predicted ($\hat{\mathbf{P}}$) poses:

where $\mathbf{x}\in O$ are the positions of model vertices in the object coordinate frame, and $\|\cdot\|$ is the $\ell_{2}$-norm.

Because ADD makes no allowances for object symmetries, it provides only limited insight into the graspability of detected objects based on the predicted pose, as translation errors are penalized similarly to rotation errors. Moreover, in the case of symmetric objects, ADD penalizes predictions even when the true pose cannot be determined from the input image. To overcome these limitations, some researchers adopt a mean closest point distance (ADD-S) [22]:

where each vertex in the ground truth model is paired with the nearest vertex in the prediction model, irrespective of consistency with other assignments. As shown later in this section, this metric can significantly underestimate pose error due to unrealistic pairings.

Another way to handle symmetries is to directly model the set $\mathbf{S}_{O}$ of symmetry transformations corresponding to the object. Starting from ADD in Eq. (1), this leads to the Mean Symmetry-Aware Surface Distance (MeanSSD):

MeanSSD requires calculating or manually identifying all valid (discrete and/or continuous) symmetries for a given object, a potentially burdensome task for a large and varied set of objects. If the mean operator in Eq. (3) is replaced with max, MeanSSD becomes the Maximum Symmetry-Aware Surface Distance (MSSD) proposed by Hodaň et al. [23]:

As a compromise between the inaccurate ADD-S metric and the labor-intensive MeanSSD/MSSD metrics, we propose a variant of the average distance metric in which the vertex correspondences between ground truth and predicted models are made by solving a linear sum assignment problem. The most famous solution to this assignment problem is the Hungarian algorithm [24], hence we denote this metric ADD-H. The assignment $f_{A}:\mathbf{x}_{2}\mapsto\mathbf{x}_{1}$ produces a bijective mapping from the vertices of the model in the predicted pose to those in the ground truth pose. If we let $A=\{(f_{A}(\mathbf{x}_{2}),\mathbf{x}_{2})\}$ be the set of such correspondences, then the set that minimizes the sum of distances between paired vertices is calculated as:

We compute Eq. (5) using a modern variation of the Hungarian algorithm.³³3Implemented as scipy.optimize.linear_sum_assignment. Note that versions of SciPy before 0.17.0 use a much slower algorithm. The average distance is then computed between assigned pairs $(\mathbf{x}_{1},\mathbf{x}_{2})\in\tilde{A}$:

Fig. 7 visualizes the vertex correspondences used by each metric when computing the error between the ground truth and predicted pose for a bottle. The inset rendering on the left depicts the scene, where the predicted pose (in translucent gray) is offset from the ground truth pose (with colored vertices) by a 150° rotation about the vertical axis and a horizontal translation of about half the object width (3 cm).

The right side of the figure depicts the vertex assignments for each metric: vertices in each model ($\mathbf{x}_{2}\in O$) are colored according to the paired vertices in the ground truth model ($\mathbf{x}_{1}\in O$). Since ADD uses a fixed, bijective assignment based on the original vertex ordering, the error is large. In contrast, ADD-S assigns the nearest ground truth vertex to each target, resulting in a non-injective mapping, in which only points from the right side of the ground truth are used, as reflected in the green and purple color across the entire target mesh. MeanSSD uses the same vertex assignment as ADD but explicitly selects the symmetry-preserving rotation that minimizes error, resulting in a rotational alignment that is more representative of the grasp-relevant geometry of the prediction. Finally, ADD-H optimizes a bijective mapping between poses to minimize error—achieving a vertex assignment that reflects the pose symmetry, similar to MeanSSD.

To further compare the metrics, Fig. 8 shows the error, averaged over 5 trials for each of the 28 HOPE objects, as each object is translated from its initial position in increments of 1 cm. In the left plot, each object is first randomly rotated by one of the symmetry-preserving transformations, whereas in the right plot, each object is first randomly rotated by an arbitrary transformation. ADD-H closely matches the MeanSSD error without any explicit enumeration of object symmetries, especially when the rotation is symmetry-preserving. In contrast, ADD and ADD-S significantly over- and under-estimate the error, respectively.

Although ADD-H is computationally more expensive than related methods, by using an efficient algorithm to solve the assignment problem it can be applied to meshes with several hundred vertices. In our experiments, we use 500 vertices, which yields consistent results with sufficiently fast computation.

To estimate the error in our ground truth pose annotations, we captured orthogonal views ($\sim\!90$ degrees between camera axes) of three static scenes. Objects were annotated independently in both views, as shown in Fig. 9. Using a robust Procrustes alignment of all but one annotated object pose, we estimated the extrinsics between the two camera views. The pose of the held-out object was projected from the first view to the second, and ADD error was computed between the transferred pose and the annotation made directly in the second view. By repeating the process while holding out each object in turn, we estimated the annotation error for all objects in the scene. Fig. 5 shows a histogram of ADD for the 64 object instances contained in the three scenes. The mean and median ADD are 5.7 and 4.3 mm, respectively. We conclude, therefore, that ground truth is accurate to several millimeters.

We trained baseline pose prediction models for each object class using the Deep Object Pose Estimation (DOPE) [4] and CosyPose [3] methods for detecting and predicting object poses in RGB images. For DOPE, models were trained from synthetic images using domain randomization [25]. Unlike the original DOPE paper, we did not train using any photorealistic rendered images, which no doubt degrades performance. Accordingly, we denote the baseline as DOPE-DR. Two architecture variants are considered with $50\times 50$ and $400\times 400$ output maps, denoted DOPE-DR-50 and DOPE-DR-400, respectively. For CosyPose, we trained using photorealistic images rendered by the BlenderProc tool [18].

As both DOPE and CosyPose are RGB-only methods, both can suffer from errors along the projection ray from the camera to the object. We consider a simple method for refinement using the RGB-D point clouds: given a predicted pose, visible model vertices are aligned with the depth map by adjusting the predicted translation by a scalar factor determined by a line search. We denote this method with “-LS”. CosyPose has a distinct advantage in that the object mask from the Mask R-CNN detector can be used to filter both the object mesh and RGB-D point cloud, thereby limiting the effect of occluding objects.

Table I shows the average recall (AR) under three metrics as computed by the BOP benchmark [14]: Maximum Symmetry-Aware Surface Distance (MSSD), Visible Surface Discrepancy (VSD), and Maximum Symmetry-Aware Projection Distance (MSPD). MSSD, which measures 3D alignment of model vertices, is the most relevant to robotics applications. VSD measures the discrepancy between 2D distance maps, rendered for GT and predicted poses, respectively. MSPD measures the discrepancy between 2D projections of model vertices, thereby capturing visual alignment with an eye toward augmented reality applications.

CosyPose-LS yields a nearly 2x improvement in $\text{AR}_{\text{MSSD}}$ over DOPE-400-LS. Furthermore, mistakes made along the projection ray to the object are shown to be a significant source of error, as evidenced by the dramatic improvement made by our simple depth refinement (-LS). The refinement improves MSSD and VSD scores dramatically for all three methods, including a 25% improvement in CosyPose. Because MSPD is sensitive to subtle changes in visual alignment, it suffers slightly under depth refinement despite overall better 3D alignment.

Two aspects of the BOP metrics are worth noting: 1) as mentioned previously, the BOP definition of symmetries requires both geometric and (nearly) exact visual similarity, meaning that none of the HOPE objects are considered symmetric in the BOP challenge; and 2) the recall error threshold is defined relative to object size, varying from 5% to 50% of the object diameter [23].

Because of our focus on graspability, we present additional results using absolute detection thresholds, varying from 0.5 to 10 cm. We also consider all of our objects to be geometrically symmetric, ignoring small details like tabs on can lids. For defining geometric symmetries, meshes were manually aligned to a consistent set of coordinate axes and grouped into three categories: cylinders (360° rotation around vertical axis in 1° increments, and 180° flip top to bottom), cuboids (180° flip along any axis), and bottles (180° flip front to back).

Fig. 11 presents the results of four experiments, depicting the detection rate of ground truth objects (recall) on the $y$-axis while varying the maximum error threshold for a positive detection on the $x$-axis. We show results on both validation and test sets. To facilitate interpretability by those familiar with existing metrics, we use the MeanSSD metric.⁶⁶6 For versions of Fig. 11 (a-c) and Table II using the ADD-H metric, see Fig. 12 and Table III, respectively, in the Appendix.

In the analysis that follows, we specifically highlight performance at two particular error thresholds: 2 cm and 10 cm. The smaller threshold (2 cm) approximately indicates errors sufficient for grasping, although this will of course vary by gripper. The larger threshold (10 cm) indicates errors sufficient for a system that is able to gather additional views of a detected object for refined results, e.g., when the camera is in the robot hand—although this threshold also is an approximation.

Fig. 11 (a) compares CosyPose and DOPE using the MeanSSD metric. CosyPose-LS, trained with photorealistic images, incorporating a visual refinement step, and augmented with our simple depth refinement, predicts over 70% of objects in the HOPE test set within 2 cm. Considering DOPE, the figure shows the orthogonal benefits of a larger output map (DOPE-DR-400-*)—which improves the precision of keypoint prediction in the DOPE method—and line search depth refinement (DOPE-DR-*-LS). With line search and depth refinement, more than 30% of objects in the HOPE test set are predicted within 2 cm.

Fig. 11 (b) shows the performance of CosyPose-LS using the MeanSSD metric across three object categories. Cuboids present a particular challenge since they are less resistant to small rotational errors. In this dataset, cuboids also tend to be larger objects that are placed further from the camera and behind other objects in some scene arrangements. Highest success is obtained with bottles, with almost 90% predicted within 2 cm.

Fig. 11 (c) depicts the effect of lighting variations by comparing detection rate (again using MeanSSD) between the most favorable image and most difficult image in each scene (determined subjectively). Particularly on the test set, which is much larger than the validation set, the difference in accuracy is small, indicating a promising level of robustness in these synthetically trained methods.

Finally, Fig. 11 (d) compares the various metrics using CosyPose-LS predictions, confirming previous observations that ADD-H is a reasonable compromise that matches the results of MeanSSD without requiring explicit enumeration of object symmetries. As mentioned earlier, ADD over-estimates, while ADD-S under-estimates, the error.

Finally, Table II presents detailed statistics of CosyPose-LS predictions for each object class and category, using the MeanSSD metric, along with some statistics of the ground truth (GT) object poses. The table includes the median of the MeanSSD error among objects detected within a 10 cm threshold, as well as precision and recall at both 2 cm and 10 cm thresholds. At the most generous 10 cm detection threshold, 83% of ground truth objects (in the test set) are detected by CosyPose-LS, and at the tighter 2 cm threshold, 72% are detected, meaning that grasping may be feasible. CosyPose-LS could benefit from an improved object detection step, as nearly 20% of objects are not detected. False positives appear to be less of a problem, with $>$98% precision at the 10 cm threshold. These results offer room for improvement, while showing that an existing method like CosyPose can be reasonably successful as an off-the-shelf predictor for these objects.

The 28 toy grocery objects come from the following products, with links for purchasing:

• •

  Melissa & Doug Let’s Play House! Grocery Shelf Boxes

• “Chocolate Pudding”, “Cookies”, “Granola Bars”, “Mac & Cheese”, “Popcorn”, “Raisins”, and “Spaghetti”

• Links: Melissa & Doug, Amazon, Google Shopping

• •

  Melissa & Doug Let’s Play House! Grocery Cans

• “Alphabet Soup”, “Cherries”, “Corn”, “Green Beans”, “Mushrooms”, “Peaches”, “Peas & Carrots”, “Pineapple”, “Tomato Sauce”, and “Tuna”

• Links: Melissa & Doug, Amazon, Google Shopping

• •

  Melissa & Doug Let’s Play House! Fridge Fillers

• “Butter”, “Cream Cheese”, “Milk”, “Orange Juice”, “Parmesan”, and “Yogurt”

• Links: Melissa & Doug, Amazon, Google Shopping

• •

  Melissa & Doug Favorite Condiments

• “BBQ Sauce”, “Ketchup”, “Mayo”, “Mustard”, and “Salad Dressing”

• Links: Melissa & Doug, Amazon, Google Shopping

Note that some items included in these products (viz., “Crackers”, “Crispy Crisps”, “Pancake Mix”, “Deli Cheese Slices”, and “Deli Slice Meats”) were omitted from our dataset because their sizes made scanning difficult.

Here we include versions of both Fig. 11 (a-c) and Table II using the ADD-H metric instead of MeanSSD.