Enhancing Grasping Performance of Novel Objects through an Improved Fine-Tuning Process

Given an object or a dense clutter scene, the goal of the grasping algorithm is to find and execute the optimal grasp at the moment. Current data-driven algorithms are mainly divided into two categories: planar depth grasping and point cloud-based 6D pose grasping. In planar depth grasping, leading algorithms in terms of grasping success rate include GGCNN[1], GQCNN[28], and others. The core idea of these algorithms is consistent. Taking GGCNN as an example, during the offline stage, a large number of RGB images of objects and their aligned planar depth maps are collected, and then these images are annotated with grasps (manually using rectangular bounding boxes). These data are then used to train the network. In the online stage, the input is the RGB image and its corresponding depth map, and the output is the grasp confidence, width, and angle for each pixel. The main drawback of these methods is the loss of the gripper’s degrees of freedom, which can only perform grasps from above. Therefore, flat objects such as pizza boxes cannot be grasped from above. Moreover, these algorithms have a lower grasping success rate when dealing with dense clutter scenes, so point cloud-based grasping algorithms are needed.

Liang et al. proposed the PointNetGPD[2] algorithm to address the problem of selecting grasp configurations from point clouds. This method mainly consists of two stages: offline and online. In the offline stage, grasp datasets are generated from object point clouds in open-source datasets, specifically by sampling grasps for each object point cloud based on the Antipod method[15], with each grasp corresponding to the closing region of the hand and its score annotation. Subsequently, a network model is designed based on PointNet[29], which takes the closing region of the hand as input and outputs the corresponding score. In the online stage, the scene point cloud (provided by a depth camera) is input, and GPD[14] is used to sample grasps from the scene point cloud, obtaining candidate closing regions of the hand contains. Then the grasp with the highest score is executed. Firstly, when grasping novel objects different from those in the open-source dataset, the algorithm’s grasping success rate is much lower than that for grasping objects similar to those in the open-source object dataset. Secondly, the algorithm’s offline stage of generating the closing region of the hand for an object requires ten hours on a mainstream home computer (Intel Core i7 13th), which is too time-consuming. Lastly, the score annotation is based on the force closure method, which does not take into account that the object may move and cause grasping failure when the gripper closes.

Fang et al. proposed GraspNet[3] as a benchmark for large-scale grasp generation. GraspNet takes the scene point cloud as input, specifically, generating grasps for objects in open-source datasets using the PointNetGPD[2] method during the offline stage. Then, GraspNet arranges and combines these objects in the scene, removing grasps that collide with other objects, thereby generating scene point cloud grasps different from previous single-object grasps. This method has a higher grasping success rate in dense clutter scenarios, as it fully considers the effective closing region of the hand contains for objects stacked in the scene. However, its shortcomings are the same as those of PointNetGPD, namely, the large time consumption for grasp generation and the lack of consideration for the scoring annotation method during the grasping process.

To evaluate the quality of grasps, many analytical methods have physically analyzed the geometric shape of the gripper configuration and the objects used for evaluating grasp quality. Force closure[13] and GWS[17] analysis are two mainstream grasp quality metrics. The force closure method considers the friction between the gripper and the object, while GWS can handle frictionless situations. These two metrics are very versatile and have driven the progress of numerous grasp research directions. However, these methods do not take into account that during the execution of grasping, other points on the gripper may contact the object first, causing the object to move and leading to the failure of some initially high-quality grasps. Therefore, we considered this situation and optimized the scoring.

Goldfeder et al. created the Columbia Grasp Database using GraspIt combined with form closure. It takes approximately 10 minutes to generate 15 form closure grasps, and the total computation time to build the grasp dataset on a server is about one month. This computational cost is evidently too high and not suitable for use during fine-tuning.

Lei et al. proposed a fast grasp generation method “Fast grasping ” This method requires downsampling of the object first, followed by PCA analysis to obtain the principal axes. Then, planar grasp annotations are performed on the object’s cross-section along two orthogonal principal axes, which are then converted back to 3D grasps. Although this method is significantly faster than others, its drawback is the low diversity of grasps, as the grasp vectors are all along the PCA principal axes constructed by the object, making it unsuitable for fine-tuning as well.

As data-driven grasping algorithms may fail to successfully grasp some novel objects when applied to different scenes, we need to fine-tune them to better grasp these novel objects.

The first requirement is the CAD model of novel objects. However, creating a detailed model of an object is very cumbersome. We have designed a framework that requires only a robotic arm and an RGB-D camera to output the CAD model of a new object for subsequent grasp generation.

First, we use the robotic arm to capture images based on the depth returned by the camera(see Fig. 2). The robotic arm takes 32 effective photos of the object from different angles. These photos are then input into Nerf (Reconstructing 3D scenes from a set of sparse 2D images) to generate the object’s CAD model. Finally, the generated object is upsampled to output a more detailed CAD model for subsequent grasp generation.

In this way, we achieve the CAD model of our novel objects in the same quality of open-source datasets which set for grasp generation.

The process of generating grasp datasets using Antipod-based methods, such as Columbia Grasp[24] and PointnetGPD[2], are time-consuming,with thousands of grasps taking up to ten hours(On 12th Gen core i7 12700H). Therefore, we propose a new method to generate grasps, which can fine-tune more quickly.

First, the grasp configuration is defined as ,

$p=[x,y,z]\in\mathbb{R}^{3}$ represents grapspoint , $r=[r_{x},r_{y}r_{z}]\in\mathbb{R}^{3}$ represents the gripper’s orientation.

The Antipod-based methods are based on contact points to generate grasps. Specifically, it first finds a point on the object, samples grasps with the gripper width as the radius, and then samples grasp poses within a certain range based on the sampled contact points. The grasp poses are evaluated for collision and force closure scores. which obviously causes many repetitive grasp configurations for parallel jaws, such as

n1 n2 are the contact points.

Since we aim to obtain diverse closing regions of the hand during the fine-tuning process and to accelerate the grasp generation, we propose a framework that differs from other similar approaches[8]. We sample a set of grasp orientations that are uniformly distributed in space. These grasp orientations are used to generate grasps by iterating through every point on the object.

In the space we set a grasp orientation and then we obtain the collection of grasp orientation r.

The algorithm for our entire grasp generation process is as follows:

In this process, we use collision detection and local reorientation of the object’s normal vector to ensure that the generated grasps do not collide with the object. The algorithm complexity of our sampling process is $O(n)$(Compared to the Antipod-based algorithm, the algorithm complexity is $O(n^{2})$ ). And then we remove duplicate closing region of the hand contains.Finally, we employ the force closure scoring strategy same as PointnetGPD[2] and GraspNet[3], and the results show that our method is 400% faster. The score distribution and grasp success rate of our method are basically consistent with previous algorithms.

The reasons why our method is faster than the antipodal approach and achieves equivalent results are as follows. First, our method does not generate duplicate grasps, as previously mentioned, the contact point-based antipodal approach produces repetitive grasps. Second, we filter out unreasonable grasps based on the differential normal vectors of the object faces(see in Fig. 3), which is more time-efficient than using force closure to eliminate these grasps. Finally, we employ the same scoring and labeling method as the antipodal approach, ensuring consistency in the generated grasp data and scoring effectiveness.

In the process of executing grasping, the gripper moves to the grasp point according to the grasp approach and then closes the gripper. However, during this process, an important step has been overlooked. As shown in Fig. 4, when the gripper is closing, if one side of the gripper touches the object first, the force balance of the object will be disrupted, resulting in the object’s movement.resulting in movement and an unstable grasp.

We define a stable grasp as:

$r=I-2uu^{T}\in\mathbb{R}^{3},u=[0,0,1]\in\mathbb{R}^{3}$ represents the mirror rotation matrix.

$PC_{1}\&PC_{2}\in\mathbb{R}^{3}$ represent the point clouds on the left and right sides of the grasp approach within the closing region of the hand, respectively.

Slight differences may cause the gripper to touch the object on one side first during the grasping process, leading to object movement and grasping failure. Some grasps that are judged to be good under static force closure conditions may also fail during execution. Thus, we need to filter out these grasps.

Our goal is to minimize the discrepancy between grasps generated under static conditions and the execution of these grasps.

Optimize the grasps generated under static conditions $G(p,r)$ and the scoring annotations of grasps after the gripper closes $S(p,r)$ .

For the reasons mentioned above, we adopt large-scale simulation for grasp filtering. First, we perform the grasps generated under static conditions using large-scale simulation. After the gripper closes, if the object is outside the gripper space, the grasp is considered invalid and removed from the grasp configuration. In addition, after the grasp execution is completed, the score of the grasp is reassigned based on the ratio of the distance moved by the object’s center of mass to the size of the gripper. This makes the annotation of the grasp dataset more effective.we have designed a simple weighting function.

d1 and d2 represent the Euclidean distances between the first points on both sides of the gripper to contact the object along the gripper closing direction and the object, respectively. width reperesent the width of gripper.

In some cases, grasp may fail, and objects may slip out of the gripper. This is mainly due to an upper limit on the contact force. When a larger torque balance is required for stable grasping, the grasp may fail.(bad grasp in Fig. ???2)

To address the issue of power constraints and contact force limits during gripper closure. Firstly, we assign a certain mass to the objects to more accurately simulate the dynamic characteristics of objects during the actual grasping process. Next, we set the contact force limit during gripper closure to ensure that the gripper operation in the simulation environment conforms to the actual situation.

During the large-scale simulation, we test each grasp configuration and observe whether the object can be successfully moved to the target position after the gripper closes. We filter out grasp configurations that cannot successfully move the object within the contact force limit range. In this way, we can select grasp configurations that are more likely to succeed in actual operations, thereby generating better strategies.

Our method significantly outperforms other approaches in terms of grasp generation speed. For the same set of novel objects, our method is capable of generating 6,500 optimal grasping configurations in just 1.2 hours, while other methods take an average of 10 hours to achieve the same results.(see in Table I)

We employ sampled grasp orientation to generate grasping actions. To evaluate and annotate these grasping actions, we adopt the force closure method, which is consistent with traditional methods. Throughout this process, we observe that the grasp score distributions generated by this approach remain similar for different novel objects, even though their shapes and characteristics vary. This indicates that our method is logically consistent and can be effectively applied to various types of objects. By combining the grasp orientation and force closure methods, we can ensure the logical clarity of the grasp generation process while maintaining its versatility for different objects.