ROBI: A Multi-View Dataset for Reflective Objects in Robotic Bin-Picking

Object detection and 6D pose estimation play an important role in many technological areas. With the increasing availability of RGB-D cameras, numerous datasets has appeared [2, 14, 15, 4, 16, 17, 9]. A summary of all these datasets is presented in [17], and in the benchmark for 6D object pose estimation (BOP) [11], the authors have performed a comprehensive evaluation of 15 diverse approaches on eight recent datasets.

The LINEMOD Dataset [2] is a widely used benchmark for texture-less objects. It contains $\sim$18000 RGB-D images of 15 objects, and has become a standard benchmark in most of recent works [2, 3, 4, 5]. The test images feature severe clutter but only mild occlusion. This work was augmented by [18] to consider a high degree of occlusion for evaluation. Datasets presented in [15, 4, 16] have similar properties. The objects have discriminative color, large size, and limited pose variability, making recognition relatively easy.

For industrial object detection, Drost et al. [9] introduced the ITODD dataset, which contains 28 industrial parts with minor reflective surfaces. The T-LESS Dataset [14] features 30 texture-less objects with no discriminative color and shape. The test images were captured from 20 scenes with various complexity: from several isolated objects on a clean background to multiple objects stacked with severe occlusions and clutter. Despite numerous advantages, these two datasets have the following limitations: 1): they lack pose and scene variations, hence cannot represent industrial warehouse scenarios; 2) all objects have low reflectivity and do not encounter adverse effects of noisy and missing depth measurements, which is common to industrial parts.

To better serve the bin-picking scenario, Doumanoglou et al. [7] provided the IC-BIN dataset with multiple objects stacked in a bin. It compromises three scenes of two objects from IC-MI dataset [15]. The same scene was recorded from different viewpoints for evaluating object pose estimation and active vision techniques. Recently, the Fraunhofer IPA dataset [17] has been introduced for object pose estimation and instance segmentation for robotic bin-picking. However, it contains only 520 real-world depth images of two industrial objects. In comparison, we captured a total of $\sim$8K images for 7 objects in the real-world.

Historically, 6D object pose estimation has been addressed from a static viewpoint. However, depth cameras often produce inaccurate depth measurements or fail to sense depths entirely from a single viewpoint due to reflective object materials, limited sensor resolution, and scene occlusions. To overcome these limitations, when setup permits, multi-view fusion approaches are able to provide higher levels of scene completion than single view acquisition.

There are several datasets of multi-view RGB-D scans, including ScanNet [19], Matterport3D [20], and others [21, 22, 23]. However, there is no ground-truth available for qualitative evaluation of depth fusion or scene reconstruction. In [24, 25], authors obtained the ground truth by fusing all frames of each scene using standard TSDF fusion [26] and manually removed outliers. The evaluation was then performed on only a subset of frames. Compared to these methods, we provide a novel acquisition method to acquire high-quality ground truth depth images for multi-view depth fusion and scene reconstruction.

Due to the requirements of high accuracy and short cycle time in robotic bin-picking, we used active stereo cameras for our data capture. Specifically, as shown in Fig. 3(a), a high-cost Ensenso N35 depth sensor and a low-cost Intel RealSense D415 RGB-D camera were used for our real-world data capture. We calibrated intrinsic parameters for both cameras with a precisely manufactured calibration board (shown in Fig. 1(a)) and calibration toolboxes provided by camera vendor. The root mean square re-projection errors calculated on circles of the calibration board are $0.22\;px$ for Ensenso, and $0.12\;px$ and $0.3\;px$ for RealSense’s RGB and depth frames, respectively.

The Ensenso N35 camera comes with an optical lens for short working distance. It has a minimum working distance of $240\;mm$, a maximum working distance of $520\;mm$, and an optimal working distance of $330\;mm$. The sensor is comprised of two visible-light cameras and a light projector, and produces depth maps with a resolution of $1280\times 1024$. Since RGB images are not available, we captured Ensenso data with the following two configurations:

• •

  Depth configuration. We turn the camera projector on and use a low exposure time to eliminate the impact of ambient light. The raw disparity maps and pattern projected stereo pairs are collected.

• •

  Monochrome configuration. The camera projector is turned off for this configuration to capture monochrome images. We use a higher exposure time to obtain optimal contrast for objects. Only stereo pair data is saved with this configuration.

Fig. 4(a) shows the sample camera data captured with Ensenso N35 for the "ZIGZAG" object . The depth map demonstrates that the Ensenso is able to capture fine geometric details, but there is a significant amount of missing data due to surface reflection.

The Intel RealSense D415 is a compact, low cost sensor. It comes with an RGB camera and a depth camera system, comprising two IR cameras and an IR projector. We used the RealSense camera to capture the synchronized RGB images, depth maps, and pattern projected stereo pairs at $1280\times 720$ resolution. The working distance is tuned to the range of $[243\,mm,638\,mm]$. As shown in Fig. 4(b), compared to Ensenso, the depth map captured with RealSense has a higher level of completeness, but with significant degradation of depth accuracy (e.g., loss of geometric details).

For each of the seven objects depicted in Fig 2, we captured four bin scenarios (2 full-bin and 2 low-bin). The two active stereo cameras are mounted to an EPSON 6-Axis C4L robot arm, illustrated in Fig. 3(b). We program the robot arm to move on a view sphere around the scene, from approximately $45^{\circ}$ to $90^{\circ}$ of elevation with different camera distances. The robot end-effector stays pointing towards the center of the workstation. For full-bin scenarios, we sampled a total of 106 views for Ensenso and 125 views for RealSense. Due to the occlusion of bin wall, in low-bin data capture, the sphere elevation is limited to the range of $[65^{\circ},90^{\circ}]$, and there are 68 and 94 views for Ensenso and RealSense, respectively. Fig. 5 demonstrates the sampled viewpoints for these two different scenes, captured with Ensenso camera.

To determine 6D camera poses at different viewpoints, we use a two-step calibration procedure: 1) initial camera pose estimation using robot forward kinematics and the calibrated poses between end-effector and cameras; 2) pose refinement by performing iterative closest point (ICP) on calibration spheres. To acquire the transformation between robot end-effector and two cameras, we leveraged the industrial-grade accuracy of our C4L robot arm, and move the cameras to look at the same calibration pattern at different viewpoints. The 6D poses for Ensenso and RealSense camera are then estimated by solving the classic "hand-eye" calibration problem [27]. For better refinement accuracy using ICP algorithm, we placed calibration spheres around the bin (illustrated in Fig. 1(a)). The spheres are manufactured with a matte surface, so that the cameras can achieve their optimal accuracy. The average closest-point residual error was successfully reduced from 0.33 mm to 0.26mm for Ensenso data, and 1.52 mm to 0.8 mm for RealSense data.

Active stereo cameras can provide reliable depth measurements when object surfaces have the ideal diffuse reflection, also known as Lambertian reflectance. However, most industrial objects do not have this property, resulting in inaccurate or missing depth measurements. This phenomenon is more obvious for objects with glossy surfaces. As demonstrated in Fig. 6(a), due to severe specular and inter-reflections in the bin, the high-cost Ensenso camera fails to sense a large amount of depth data. To this end, some techniques have been studied to recover missing depth, such as depth completion [12] and multi-view depth fusion [10] for the bin-picking problem. In this section, we provide a method to capture the ground truth depth maps in the real world.

We captured the ground truth depth maps with the Ensenso camera, and applied a scanning spray [28] on objects to better approach ideal Lambertian surfaces, so that the Ensenso camera can achieve its optimal accuracy ($\sim 0.2mm$). The scanning spray generates a homogeneous layer with only $8-15\,\mu m$ thickness, which is at least one order of magnitude less than expected depth accuracy. The spray is able to self-evaporate within a few hours. Therefore, we leveraged the high repeatability of our robot arm and captured the test and ground truth images with two separate scans. In the first scan, we applied the scanning spray and capture ground truth depth maps. After full evaporation, the test images were captured during the second scan. Fig. 6(b) shows the captured images after applying the scanning spray. Compared with images without the spray (Fig. 6(a)), the Ensenso sensor is able to capture more complete and less noisy depth maps on the same scene.

To evaluate depth fusion methods, for each scene, we constructed a ground truth mesh by applying TSDF fusion [29] on ground truth depth maps. To demonstrate that the reconstruction of the ground truth scene is not biased to any fusion method, we apply two methods, TSDF fusion and probabilistic fusion [10] to produce two sets of ground truth meshes, as shown in Fig. 6(a) and 6(b). We compute the mean point-to-point distance to quantitatively measure the difference between these two sets. The distance is less than 0.03 mm, indicating that our ground truth accuracy is not sensitive to different fusion methods.

For each object, we provide both a manually created CAD model and a semi-automatically reconstructed model. Both models are provided with the dataset in the form of 3D meshes.

The reconstructed model is created in a manner similar to the creation of the scene model. We capture the multi-view high-quality depth maps after applying scanning spray, and use TSDF fusion for model reconstruction. For each object, we first reconstruct 2-4 partial models (depending on its geometric properties), and manually remove unnecessary parts and minor artifacts for each partial model. The partial models were then registered into a global reference frame by manual alignment and ICP refinement. Finally, we used MeshLab [30] to inpaint minor holes in the model and smooth the mesh surface. The reconstructed models are shown in Fig. 7 (first column).

We provide ground truth object poses using the CAD object model and the high-quality scene model, described in Sec. . We perform the alignment process using MeshLab [30]. We first manually aligned object models to the scene model as an initial 6D object pose estimate. We then upsample the CAD model to a high resolution, and apply ICP to refine the object pose. Lastly, we adjust the pose manually whenever necessary to correct any in-plane translation and rotation. We repeat this process several times until a satisfactory alignment is achieved.

Lastly, for each ground truth pose, we generate a visibility score in the scene. This is done by rendering the segmentation mask of object pose via perspective projection. The visibility score, $\mathbf{v}\in[0,1]$, is the fraction of the number of visible pixels over the number of pixels without any occlusion.

For the performance of depth fusion, we evaluated two volumetric fusion approaches, the traditional TSDF fusion [26] and a probabilistic fusion approach from [10]. We use three metrics from [10] for the evaluation: mean point-to-point distance, outlier percentage, and scene completeness. These metrics are performed only on the surface of objects, which has a major impact on object pose estimation.

Fig. 8 shows the quantitative reconstruction results for each object with all viewpoints from different cameras. It can be seen that, for both Ensenso and RealSense data, the probabilistic fusion [10] has a much lower reconstruction error than TSDF method on all objects. Moreover, it provides a better trade-off between scene completeness and outlier percentage due to its explicit uncertainty handling. However, we can observe that the probabilistic fusion sacrifices more completeness of scene on the "Gear" object, which has both a high-gloss surface and a complex geometric shape. Fig. 8 also reveals that, when compared with Ensenso results, the reconstruction performance drops significantly on RealSense data, indicating that both methods are upper bounded by the quality of depth measurements.

To measure the 6D pose error of the object, we used the standard Average Distance of Model Points (ADD) for non-symmetric objects and ADD-S for symmetric objects [2]. A pose hypothesis is accepted as the correct detection if its ADD or ADD-S score is less than 10% of the object diameter.

For the performance evaluation, we chose three representative pose estimators: LINE-2D [2], PPF (point pair feature) [3] and AAE (augmented autoencoder) [8]. These methods only require the object model for training. The LINE-2D and AAE are RGB-based pipelines. LINE-2D is a template-based method that builds the multi-view templates from the 3D models, and exploits the gradient response on monochrome/RGB images for detection in run-time [31]. AAE is a learning-based pose estimator. The authors used the self-supervised Augmented Autoencoders for estimating 3D rotation, and object detector (e.g., YOLO [redmon2016you]) for translation estimation. Compared with these two approaches, the method proposed by Drost et al. [3] relies on 3D point cloud data as input, and solves object poses by coupling the idea of point pair features and a dense voting scheme. This method is still one of the best performings on the 6D object pose estimation leader board (BOP) [11]. We implemented these methods based on OpenCV [32], Point Cloud Library (PCL) [33] and open-source codes provided by authors [8]. The rendering of training images was based on a toolkit from BOP [11]. Lastly, we apply the iterative closest point (ICP) algorithm for the final refinement of the object pose estimates.

As in [14, 3, 6], the performance is measured by correct detection rate (CDR). A ground truth pose will be taken into consideration only if its visibility score is larger than 0.6. Fig. 9 presents the CDR for each object using single view data. It can be seen that, due to a large number of missing depth measurements, the PPF method has an overall poor performance on both Ensenso and RealSense data. In comparison, the LINE-2D and AAE methods have significantly higher detection rates since they are less dependent on depth data (used for ICP only).

To improve the performance of pose estimation, as illustrated in [10], we can apply the depth fusion and provide high-quality depth input data for the pose estimator. We demonstrate this strategy on the Ensenso camera data with PPF pose estimator, whose performance is highly correlated to depth data. Fig. 10 shows the results on two volumetric fusion approaches, evaluated in Sec. IV-A, as well as the ground truth scene model. It can be seen that, compared with the single view result, the performance of PPF pose estimator can be greatly improved with multi-view depth fusion. And as a result of the higher reconstruction accuracy and few outliers, the probabilistic approach outperforms TSDF fusion on almost all objects. When given more accurate 3D data in the form of the ground truth meshes, the PPF pose estimator can provide close to perfect detection rates.