EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration

Given a robot arm with a known 3D model and a camera with known intrinsic parameters, the objective is to solve the eye-to-hand calibration problem, which is to estimate the transformation between the camera $c$ and the robot base $b$. In this paper, the transformation is represented as $T_{cb}\in\mathit{SE}$(3) that satisfies: $P^{c}=T_{cb}P^{b}$, where $P^{c}$ and $P^{b}$ are the 3D points in the camera coordinate system and the robot base coordinate system, respectively.

As shown in Fig. 2, we optimize the camera pose in an iterative manner. In each iteration, given the image of the robot arm at a specific joint pose, we first predict the observed mask using PointRend [20]. Following this, we utilize differentiable rendering (Sec. III-C) to refine the camera pose by minimizing the discrepancy between the rendered mask and the observed mask. Next, to optimize the $T_{cb}$ more efficiently, we utilize the consistency-based joint space exploration module (Sec. III-D) to select the most informative joint pose for the next iteration. The robot arm is then driven to this informative joint pose and the proposed method proceeds to the next iteration. Note that in later iterations, the camera pose is optimized with the masks at all the observed joint poses.

Different from the equation $AX=XB$ [3] that is widely used in marker-based hand-eye calibration, our approach employs a novel objective function that operates at the pixel level. Assuming that the 3D model of the robot arm is known, we can obtain the camera pose by minimizing the difference between the rendered robot arm and the camera observation, which is similar to the render-and-compare technique used in [21]. Unlike marker-based hand-eye calibration methods and regression-based methods, the pixel-wise objective function provides a direct connection between the camera pose and the observed image. This feature makes our approach considerably more diagnosable.

Specifically, we use the objective function to minimize the difference between the rendered mask and the observed mask:

where $\pi$ is the differentiable mask renderer, $l$ represents the 3D shape of a robot link, $M$ is the mask of the robot, $T_{cb}$ is the transformation between the robot base $b$ and the camera $c$, and $T_{bl}$ is the transformation between the link $l$ and the robot base $b$, which can be obtained through forward kinematics with joint poses. Since we generate masks for each individual link and combine them to form the mask for the entire robot arm, we utilize the $\min$ function to limit the sum of all the link masks to a maximum value of 1.

As the optimization target $T_{cb}$ belongs to the $\mathit{SE}(3)$ space, we transform the transformation matrix to the Lie algebra space [22] and then perform the optimization using the following objective function:

The above loss function is fully differentiable, we could use the gradient descent method with PyTorch [23] Autograd to perform the optimization.

In this subsection, a novel approach is presented that alternates between identifying the most informative supplementary joint pose for camera pose estimation and recalculating the camera pose by incorporating this new joint pose.

Recall that, our camera pose estimation is achieved by solving Eq. 2, which aims to align the mask of the projected 3D model of the arm with the mask of the observed arm. We can rewrite the objective function in a more simplified manner:

where $\pi$ is a mask renderer, $J$ is a joint pose, $t$ is a camera pose, $M_{J}$ is the observed robot arm mask at joint pose $J$. This 2D matching constraint is often insufficient in constraining the solution space of the 6D camera pose given limited joint pose observations. For example, accurately estimating the camera’s distance to the robot base can be challenging with only one joint pose observation. In other words, there might be a set of reasonable camera pose candidates $T$ that satisfies $L(t,J)<\epsilon$, where $\epsilon$ is a small value.

The goal is to shrink $T$ as much as possible by obtaining a new $T^{\prime}\subseteq T$ as the solution set of $L(t;J^{\prime})<\epsilon$ with a new joint pose $J^{\prime}$. The next question is, how to choose $J^{\prime}$ so that $T$ can be shrunk as soon as possible? Equivalently, how to choose $J^{\prime}$ so that $L(t,J^{\prime})>\epsilon$ for as many $t\in T$ as possible?

Our approach is to sample a representative set of $\left\{t_{i}\right\}_{i=1}^{K}\in T$, and based upon which, we use $\sum_{i}L(t_{i},J^{\prime})$ as the surrogate to help us to choose $J^{\prime}$. In fact, it can be proved that:

where $m_{i}=\pi(J^{\prime},t_{i})$. This inequality then tells us that, if we choose $J^{\prime}$ to maximize the variance of the masks, we can shrink $T$ as soon as possible.

Based on this observation, we develop a joint space exploration strategy that operates solely in simulation, allowing us to generate informative joint poses before executing motion commands on the robot. The algorithm is summarized in Alg.1, where we first sample a large number of joint poses in the simulator and then evaluate the variance of the rendered masks over the camera pose candidates. Then, we move the robot to the joint pose with the highest variance and perform the next iteration of DR-based optimization. Unlike traditional marker-based methods that necessitate the manual design of joint poses, our approach is fully automatic and requires no human intervention.

In the subsequent DR-based optimization iterations, we include all previously observed masks rather than using only the most recent observation. The final optimization objective is defined as follows:

where $K=\{1,2,\dots,N\}$ is the number of images, $T_{bl,i}$ is the transformation between the link $l$ and the robot base $b$ at the $i$-th joint pose, and $M_{i}$ is the mask of the robot in the $i$-th image.

In our implementation, we employ nvdiffrast [26] as the differentiable mask renderer. During the optimization process, we utilize the Adam optimizer with a learning rate of 3e-3 and perform 1000 steps. For the space exploration module, we randomly choose 50 camera poses from the range between step 200 and 1000 as camera pose candidates. In practice, the number of sampled camera pose candidates and the range may vary depending on the number of optimization iterations. Next, we randomly sample 2000 joint poses and filter out any invalid poses that result in self-intersection of the robot arm, links exceeding a pre-defined distance threshold from the robot arm base, or collisions with the surrounding environment. Subsequently, we identify the joint pose with the largest variance as the next joint pose to explore. The stopping criterion can be set based on the desired level of accuracy. Typically, the calibration error will converge after enough iterations. In real-world scenarios where the calibration error is unavailable, the change in variance between two iterations of space exploration can be used as a proxy to determine the stopping criterion.

Our method only required the image with the initial joint pose and the subsequent views are generated online. An example of the synthetic dataset is shown in Fig. 3, where we attach a chessboard marker on the robot arm for the marker-based method. The size of the marker is 4cm$\times$5cm, with each grid being 1cm$\times$1cm. We rendered 10,000 images for the PointRend [20] and another 10,000 images for the PVNet [25] training to obtain the observed segmentation mask and initial camera pose for our method.

The results are shown in Tab. I and Tab. II, where our method outperforms the marker-based method [3] by a large margin. Specifically, the average rotation and translation errors of the marker-based method are 0.9 degrees and 2.1 cm, respectively. For the proposed method, the evaluation is performed using different numbers of views ranging from 1 to 5. The errors consistently decrease with an increase in the number of views. Specifically, when there is only 1 view, the error of our method is 0.32 degrees and 0.5 cm, whereas, with 5 views, the error decreases to 0.08 degrees and 0.2 cm.

DREAM [12] uses a network to detect keypoints on the robot arm and uses the PnP algorithm to estimate the camera pose. Since the keypoints are not discriminative, the network often fails to accurately detect the keypoints. Moreover, detecting keypoints is not resilient to occlusion and truncation, thus the method produces a low-accuracy result, especially for the rotation error. Even if the performance of the DREAM method is enhanced by employing multiple views, it still produces worse performance compared to our method. In contrast, our method eliminates the need for any keypoints and demonstrates robustness in the presence of occlusion and truncation.

We report the 2D and 3D percentage of correct keypoints (PCK), which is the percentage of 2D/3D keypoints that are within a certain threshold distance from the ground truth. We also provide some visualizations of the results in Fig. 4. In addition to the thresholds employed in  [17, 16], we report results with smaller thresholds to further elucidate the accuracy of our method since our method nearly reaches 100% at the thresholds used in previous works. Following the conventions in [17, 16], we present the evaluation results on the training images in Tab. III and Tab. IV, while in Fig. 5, we present the results evaluated on the entire dataset since it is more representative to evaluate the performance at other joint poses.

To assess the effectiveness of our proposed hand-eye calibration method in real-world scenarios, we conduct a high-precision targeting experiment using an ArUco marker board and a pointer attached to the robot’s end effector.

We detect the corners of the marker board using OpenCV [27] and transform their positions to the robot arm base coordinate system via our hand-eye calibration results.

Next, we employ the pointer to tip the marker board corners and manually measure the distance between the pointer and the actual corner position. A representative illustration is visualized in Fig. 1.

The results and comparison with DREAM are presented in Tab. V. Since DREAM estimates 2D keypoints and uses the PnP algorithm to estimate the camera pose, it is not robust to occlusion and truncation and suffers from the sim-to-real domain gap. In contrast, our method uses dense pixel-wise error as the objective function, thus it is inherently robust to truncation and self-occlusion. Although the manual measurement is not highly reliable, it offers a preliminary indication of the system’s accuracy and convincingly demonstrates the effectiveness of our approach. To further improve the accuracy, we can use a more precise segmentation network such as SAM [28] to reach a higher accuracy as shown in Tab. V. In practice, the selection between these two segmentation methods depends on specific needs. PointRend involves a preliminary pre-training but offers a fully automated calibration process, whereas SAM mandates simple mask labeling but requires human intervention in the process.

In general, the total runtime of the system consists of three main components: DR-based optimization, space exploration, and the robot’s movement between joint poses in real-world applications. In practice, the runtime depends on the number of optimization steps, space exploration iterations, camera pose candidates, and sampled joint poses. In real experiments, the runtime also depends on factors such as the robot’s speed, the distance between the joint poses, and the planned trajectory. The runtime of our method is presented in Tab. VI. This runtime is considered acceptable when compared to time-consuming marker-based calibrations [3], which can take several hours to finetune. It’s important to note that once the calibration is completed, there is no need to repeat it unless the camera or the robot is replaced.

EasyHeC is useful in a variety of real-world applications [29, 30]. In this section, we present some real-world applications to exemplify the efficacy of our proposed approach. Suppose we intend to employ a state-based policy with solely visual inputs, then it becomes imperative to convert object states from the camera coordinate system to the robot base coordinate system. In this case, the proposed EasyHeC could be leveraged to perform this transformation. In our experiment, we choose CoTPC [29] as the state-based algorithm, which takes states as input and outputs the next robot action. We perform two experiments using the CoTPC algorithm: stacking cube and peg insertion. Specifically, we use PVNet [25] to estimate the object poses in the camera coordinate system and then use the hand-eye calibration results to transform the object poses from the camera coordinate system to the base coordinate system as the input states to the CoTPC network. Fig.7 shows the screenshots captured during the execution process, which demonstrate that the cubes are accurately aligned and the peg is successfully inserted, thereby affirming the effectiveness of our method in state-based algorithms.