Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation

The ability to recognize and interact with unseen objects is critical in robotics applications. Recent works have explored unseen object segmentation [19, 20]. However, the segmentation objective disregards the rich information within objects, e.g. curves and flat surfaces. Dense object descriptors by [7] is a promising direction for providing such ability.

Dense Object Nets (DONs) is a self-supervised method for generating dense object descriptors useful for robotics applications. DONs learns from point to point correspondence, and are able to provide rich information within objects. Recent works demonstrated the effectiveness of DONs in many challenging robotics applications. The work in [18] presented a method for manipulating ropes using dense object descriptors. Another work by [5] enforced an additional object-centric loss to enable multi-step pick-and-place from demonstration; while in [21] imposed an additional object loss to achieve goal-conditioned grasping. Cao at el. [4] proposed Cluttered Object Descriptors (CODs) to represent the objects in a cluttered for robot picking cluttered objects.

Training learning-based methods in the real-world are costly for robotics applications. Hence, bridging the gap between simulation and real-world is critical for ubiquitous learning-based robotics applications. Recent works mainly focus on domain randomization and domain adaptation.

In domain randomization, we randomize the simulation so that the policies are robust enough to handle real-world data [22]. Such methods include randomizing textures, rendering, and scene configurations [9]. However, domain randomization requires careful task-specific engineering in selecting the kind of randomization.

In domain adaptation, we map simulation and real-world data into a knowledge preserving unified space. We can directly map simulation images to real-world images, where GANs are commonly applied [2, 17, 9]. However, GAN-based methods do not generalize well to unseen objects and scenes [9], which is crucial for many robotics applications. Prior works have also explored learning domain invariant features [8, 16, 3]. The work in [11] employed the temporal nature of robot experience. Another approach by [8] separated task-specific and domain-specific knowledge via adversarial training.

While many of these prior works for dense object descriptors only used data in the same domain (only real data or simulation data) for training, we train the descriptor to match the pixels of the object between simulation and real-world images. Therefore, our SRDONs not only learn the useful object descriptors but also map the simulation and real-world images into a unified feature space with pixel consistency, addressing the sim-to-real problem.

Previous works [7, 5, 4], by using 3D TSDF (truncated signed distance function) reconstruction, can only generate matching pixels of static scenes in the same data domain. Here, we proposed an object-to-object matching method with object poses and 3D models, which can generate matching points for images from different scenes and data domains. Additionally, while other methods required collecting training data by running real robots, our method reduces time and cost by taking advantage of public datasets.

As illustrated in Figure 1, we compute the pixel coordinates corresponding to the 3D vertices on the same object model in each image to find matching points. Suppose an image contains a set of objects, $O=(O_{1},O_{2},O_{3},\ldots,O_{n})$, and pose annotations for each object, $\Phi=(\Phi_{1},\Phi_{2},\Phi_{3},\ldots,\Phi_{n})$. The 3D model $O_{i}$ is given by a set of vertices $\textbf{V}_{\textbf{i}}=(\textbf{X},\textbf{Y},\textbf{Z})^{T}$. To associate 3D model vertices with 2D pixel coordinates, we also need the projection matrix $\mathcal{P}_{i}$ for each object $O_{i}$, where $\mathcal{P}_{i}$ is computed from the intrinsic matrix, $K$, and extrinsic matrix, $E_{i}$. $K$ deals with the camera properties, and is known from the camera properties; $E_{i}$ represents the translation, $t_{i}$, and the orientation $R_{i}$ of object $O_{i}$ with respect to the camera. $E_{i}$ is computed from the object pose $\Phi_{i}$. Note the pose annotations are in the camera frame, so we do not need to consider camera transformations.

The projection matrix $\mathcal{P}_{i}$ is:

$$\centering\mathcal{P}_{i}=KE_{i}\quad\textrm{with}\quad E_{i}=[R_{i}|t_{i}]\@add@centering$$

(1)

We then project all 3D vertices $\textbf{V}_{\textbf{i}}$ of the object $O_{i}$ onto the image coordinate system to get their 2D corresponding pixel coordinates $(\textbf{u},\textbf{v})^{T}$.

$$\centering\begin{bmatrix}\textbf{u}\\ \textbf{v}\\ \textbf{1}\end{bmatrix}=\mathcal{P}_{i}\begin{bmatrix}\textbf{X}^{\prime}\\ \textbf{Y}^{\prime}\\ \textbf{Z}^{\prime}\\ \textbf{1}\end{bmatrix}\quad\textrm{with}\quad\begin{bmatrix}\textbf{X}^{\prime}\\ \textbf{Y}^{\prime}\\ \textbf{Z}^{\prime}\end{bmatrix}=\Phi_{i}\begin{bmatrix}\textbf{X}\\ \textbf{Y}\\ \textbf{Z}\end{bmatrix}\@add@centering$$

(2)

For a pair of images both containing some objects, we randomly sample from the object models a subset of their 3D vertices and calculate their corresponding 2D pixel coordinates in each image. The pixel coordinates in each image corresponding to a 3D model vertex are considered as matching pixel coordinates. To deal with occlusion, we assign the pixel coordinate to the vertex closest to the camera in Euclidean distance. Object-to-object matching enables generating matching in a variety of scenarios: different scene matching (dynamic scenes), sim-to-real matching, and multiple matching (finding matching between one object in an image with multiple of the same objects in another).

We employ the contrastive loss from [7] to enable self-supervised learning for SRDONs. Given an image $I\in\mathbb{R}^{W\times H\times d}$ where $d$ can be either $3,4$ depending on whether the input is RGB or RGBD, we map $I$ to a dense descriptor space $\mathbb{R}^{W\times H\times D}$. Each pixel in $I$ has a corresponding $D$-dimensional feature vector. Given a pair of images, the matching pixels coordinates, and the non-matching pixels coordinates, we optimize the dense descriptor network, $f$, to minimize the L2 distances between descriptors of matching pixels, and keep descriptors of non-matching pixels $M$ distance apart.

$$\mathcal{L}_{\textrm{\small{m}}}(I_{a},I_{b})=\frac{1}{N_{\textrm{\small{m}}}}\sum\limits_{N_{\textrm{\footnotesize{m}}}}\lVert f(I_{a})(u_{a})-f(I_{b})(u_{b})\rVert^{2}_{2}$$

(3)

$\mathcal{L}_{\textrm{\small{nm}}}(I_{a},I_{b})=\frac{1}{N_{\textrm{\small{strict\_nm}}}}\sum\limits_{N_{\textrm{\footnotesize{nm}}}}max(0,M-\lVert f(I_{a})(u_{a})-f(I_{b})(u_{b})\rVert_{2})^{2}$

(4)

$$\mathcal{L}(I_{a},I_{b})=\mathcal{L}_{\textrm{\small{m}}}(I_{a},I_{b})+\mathcal{L}_{\textrm{\small{nm}}}(I_{a},I_{b})$$

(5)

where ”m” is matching, and ”nm” is non-matching; $N_{\textrm{m}}$ is the number of matches, and $N_{\textrm{nm}}$ is the number of pairs of non-matching pixels ; $f(I)(u)$ is the descriptor of $I$ at pixel coordinate $u$; $N_{\textrm{strict\_nm}}$ is the number of pairs of non-matching descriptors within $M$ distance to each other, namely the number of non-zero terms in the summation term in Equation 4.

Previous works [7, 5] required the use of a real robot arm to collect data. In contrast, as described in Subsection III-A, our proposed matching method enables to use real data from public datasets and simulated data generated from the simulation. By this approach, we not only have easy access to diverse objects but also reduce the time and cost of real training data collection. Table I compares our dataset with other works.

Real Data. In this paper, we mainly use the real data from GraspNet [6]. The dataset provides 97,280 RGBD images of 88 objects over 190 cluttered scenes. Each scene contains 9-10 objects placed at random positions on a tabletop. They capture 256 images per scene with different view poses and recorded the camera pose, the 6D pose of each object corresponding to each image. However, to make the view of each scene sufficiently different, we downsampled the number of images to 50 images per scene.

Simulation Data. We use V-REP simulator to generate the simulation image. For each scene, we randomly drop 9 to 10 objects on a table. We then use a camera to capture the RGBD images and record the camera poses and object poses with the same format as GraspNet dataset. We also apply texture randomization and background randomization for generalization purpose.

Pairing Images. With the proposed object-to-object matching in subsection III-A, we can generate matching pixel coordinates for any pair of images independent of data domain (simulation or real-world). We have 3 different types of pairing: (a) Sim-Sim: a pair of images is sampled from simulated images. (b) Real-Real: a pair of images is sampled from real images. (c) Sim-Real: one image is sampled from simulated images, and the other comes from the real images. Figure 2 shows some examples of different pairing types.

Training. During each training step, we uniformly sample pairing types (Sim-Sim, Real-Real, and Sim-Real). Once a type has been sampled, we then choose whether the two images are from the same scene or different scenes (with the probability of 30% and 70%, respectively). For Sim-Sim and Real-Real matching, two images may come from the same or different scenes, while for Sim-Real matching, two images have to come from different scenes, since they come from different data domains. For each pair of images, we sample 1000 pairs of matching points, and 5000 pairs of non-matching points (object to object, object to background, background to background). More details about collecting data and training are provided in the accompanying video.

We want to use SRDONs to serves as a building block for robotic tasks. The work in [4] proposed a network structure that can use the intermediate layers of the descriptor network for training a reinforcement learning task. We adopt their method, and extend to supervised learning method.

To use SRDONs for training the reinforcement learning (RL) task, we simply apply the same structure as proposed in [4] (further details in the accompanying video), which is based on actor-critic RL, namely PPO. In the supervised learning task, we slightly modify the network structure. We first remove the critic head, and then, change the actor head which originally is fully convolutional layers to fully connected layers. Specifically, in our experiment, we use the grasping task to demonstrate the use of SRDONs with supervised learning. Grasping task requires the agent to predict the grasping position ($x$, $y$) and the top-down grasping pose ($\theta$) to grasp the object. Therefore, the output of the network in this task is the pose ($x$, $y$, $\theta$) $\in\mathbb{R}^{3}$. Figure 3 shows how we combine the intermediate layer of SRDONs and the depth stream in a U-Net structure.

We evaluate the performance of descriptors by finding matching interest points, as in [7]. We employ two evaluation metrics: the accuracy of matching correct objects, and the matching error distance normalized by image diagonal distance. Given a pair of source and target images and a point $p$ on the source image that belongs to an object $O$, let $p^{*}$ indicate the true match point in the target image (disregarding the case of occlusion as described above), and $p^{\prime}$ indicate the best match point by using a given descriptor. For the former, the object matching accuracy of matching the correct objects, i.e., $p^{\prime}$ and $p^{*}$ are on the same object in the target image. For the latter, the matching error distance is the average distance between $p^{*}$ and $p^{\prime}$.

In the experiments, we use the following methods as the baselines: (a) The original DONs [7]. (b) The Multi-Object DONs (MODONs) [21]. (c) The Cluttered Object Descriptors (CODs) [4]. When training the original Dense Object Nets, we use Real-Real in the same scenes; for the Multi-Object DONs, we use Real-Real pairing in both the same and different scenes; for the CODs, we use Sim-Sim in both the same and different scenes; and our proposed SRDONs uses Sim-Real paring only. We also evaluated the effects of randomizing object textures by randomizing 0%, 80%, and 100% of the object textures when training SRDONs, denote as Rd-0, Rd-80, and Rd-100, respectively. Additional training details and matching results are reported in the accompanying video.

Sim-to-Real Finding Matching Points. To evaluate the matching performance, we select 500 unseen image pairs for each type of pairing. For each pair of images, we sample 1000 matching points and evaluate the matching performances. Table II shows the experimental results of finding matching points of objects in the same domain (Sim-Sim Rd-100, Real-Real) and different domains (Sim-Real Rd-0 and Rd-100). (Note that Sim-Sim Rd-0 is less interesting so ignored in the table.) We can see that the original DONs method (in the second row), which trained with same scenes only, fails to represent multi-object scenes. In different domains like Sim-Real, our SRDONs shows the best performance in the rightmost two columns. In the same domains like Sim-Sim and Real-Real, the result of our SRDONs are close to other methods which are trained with these specific types of paring, while our method used Sim-Real pairing only. We visualize the descriptors of different methods in Figure 4, which shows that SRDONs can represent objects in simulation and real-world images with pixel consistency. Furthermore, texture randomization enables SRDONs to focus on the object geometry rather than color, as shown by the consistent object representation under texture randomization.

For training the descriptors, we leverage public datasets like GraspNet, however, our SRDONs also works with unseen objects. Figure 5 shows the result of our SRDONs when performs testing on unseen objects. We can see that DONs fails to represent and find the matching points with unseen objects in a multiple-object scene. In contrast, our SRDONs is able to represent objects in the images consistently in the representation space and perform better matching. Moreover, the matching performance of our method is improved by adding texture randomization and depth information, which are not considered in other DONs-based methods.

Sim-to-Real Multi-Object Consistent Evaluation. We conduct the experiment to verify that SRDONs are able to represent objects in simulation and real-world images with object consistency. We use 100 unseen images from both simulation and real images. Each image contains 9 to 10 objects. We feed these images through the SRDONs, and randomly select 1000 pixel-descriptors per image. Then, we use t-SNE to project the selected pixel-descriptors into two-dimensional for visualization, as shown in 6(a) and 6(b). In particular domain, the descriptors of the same objects are clearly distinguishable from the other objects. While, in different domains, the descriptors of the same object from the real-world and the simulation reside in similar regions.

We used SRDONs as a building block in a robotics two-fingered grasping task to grasp an object that placed at randomly pose on the table. We use the supervised learning (described in Subsection III-D) to predict the grasping pose, though applying reinforcement learning with continuous action spaces (DDPG [15]) or discretizing the continuous action space will also work in this task.

In the simulation, we use Mean Square Error (MSE) between the ground-truth and the predicted grasping poses angle in radiance as the evaluation metrics in training and testing. In the real-world experiments, we do a real grasping by the robot and measure the success grasp rate.

We use the following methods as baselines: (a) Domain randomization methods with RGB and RGBD as the inputs, denoted by RGB and RGBD respectively. The model is similar to Figure 3, however, we replace the SRDONs stream by RGB stream, which is a trainable ResNet34_8s. (b) The model is similar to in Figure 3, but we remove the SRDONs and using depth only, denoted by Depth. (c) The method proposed by [4] for picking cluttered objects in simulation, denoted by CODs. (d) Our supervised learning method proposed in Subsection III-D, denoted by SRDONs.

In the simulation, we generated the ground truth grasping orientations, e.g., we place objects on the table in such a way that the grasping orientation is zero, where the robot can grasp the objects with zero z-rotation. We captured observations from different camera poses, and calculated the grasping location and the z-rotation ($x$,$y$,$\theta$) labels via camera poses. For training data, we captured 50 RGBD images for each 28 objects from the GraspNet train slipt. We then test the grasping methods with 15 objects from the GraspNet test splits. In the real-world, we use a parallel gripper to grasp a single object that is placed on the table at random position and orientation. Each model performs 20 grasping trials (repeat twice with 10 objects collected from our lab).

Table III shows the results for grasping orientations prediction in both simulation and real-world environments. In the simulation, SRDONs achieve the minimal average error of 0.15 radiance (8.59 degrees) on unseen objects. Furthermore, the agent with SRDONs also outperformed others in the real-world by achieving 90% success grasp rate without any further training. We can see that the models with depth information performed better than the others that use RGB input only. Training details and real experiment videos are provided in the accompanying video.

Now, we used SRDONs as a building block for a more complex robotics picking task. Similarly as [4], we train an agent with reinforcement learning to pick cluttered objects with a suction pad. We have two metrics for evaluating the performance. The first is the rate of completion for all runs. A run is completion if all objects are picked before the episode terminates. The second is the average number of objects picked in all runs. In this picking cluttered objects task, we use the similar baselines to those in grasping task in Subsection IV-B, but with reinforcement learning version.

In the simulation, we train each method in with 10 random objects sampled from GraspNet train split. We then test with 20 and 30 objects from GraspNet test splits, and novel household objects. In the real-world, we directly use the trained policy in the simulation to pick 10 novel household objects without any fine-tuning.

The experimental results in the simulation are shown in Table IV and Table V. Our method with SRDONs clearly out-performed other methods on all of the metrics, and are also efficient to be generalized to more cluttered scenarios with unseen objects. When directly applying the trained policy in the simulation to the real-world testing, our method can successfully pick all 10 objects within 12.81 steps (78.1% success pick rate), which is better than other methods (as shown in Table VI). Training details and real experiment videos are provided in the accompanying video.