ONE PIECE: One Patchwork In Effectively Combined Extraction for grasp

The Cornell Dataset [18]consists of 885 RGB-D images of 240 different objects, with 8019 hand-labeled grasp rectangles.The dataset can be used to train neural networks to detect grasps in images. However, compared to traditional datasets used in deep learning, the dataset with only 885 images is very small and may lead to poor performance in generalizing to different images or object configurations. The hand-labeled grasps may also be biased towards grasps that are easy for humans to execute, but not necessarily using parallel-jaw grippers.

The Jacquard Dataset[19] can be considered a larger and more objective version of the Cornell Dataset. It contains 54K images generated in simulation using a mixer, using 11K unique object mesh models. Each image is labeled with multiple grasp rectangles that roughly cover the space of possible grasps (using the same representation as in Cornell), for a total of 110M grasp labels. The labels are generated by evaluating the quality of a large number of possible grasps, sampled randomly and simulated in PyBullet. Like Cornell, the performance of Jacquard is measured as accuracy at 25% or 30% IOU detection thresholds.

To overcome the time-consuming data generation problem, Mahler et al. created Dexnet-2.0[26], a synthetic dataset of 6.7 million depth images labeled with grasps that succeeded when executed at the center of the image. These images were created using 1500 mesh models generated in simulation. Each depth image is labeled with up to 100 grasp detections obtained using grasp planning methods on the underlying mesh model. These object models were selected from the 3DNet and KIT object databases and extracted from 50 different object categories. The dataset is augmented with Gaussian noise at each pixel and with 180-degree image reflections and rotations.

Due to the complexity of the factors involved, such as the object, the scene, and the grasp pose, datasets of this type are difficult to cover all cases, leading to limitations. Therefore, it is necessary to combine other methods for dataset generation and algorithm learning and validation.

Early research on grasp prediction assumed that robots have a perfect understanding of their environment and aimed to plan grasps based on the 3D models of objects [28][29]. Using this technique, Goldfeder et al. [27] created the Columbia Grasp Database, which contains over 230k grasps. These methods only used the CAD models of the gripper and the object. This may be the earliest grasp dataset for SE(3) grasping [32]. It consists of 7256 object mesh models extracted from the Princeton Shape Benchmark [33]. A total of 238k SE(3) grasps were marked on the 7256 models using the GRAPIT! simulator[34].

YCB (Yale, Columbia, Berkeley)[16] is an object set rather than a grasp dataset, but it is used so frequently that we include it here. YCB is a set of 77 household objects for which 3D mesh models and RGB images are available online. The Berkeley Dex-Net project was published with the following three object sets : 1) 1.3k synthetic 3D mesh models from a 50 category subset of 3DNet; 2) 129 mesh models from the KIT Object Database; 3) 13 ”adversarial” models.

EGAD! is an interesting dataset comprised of 2331 objects [20]. The focus here is on an algorithmic approach to generating 3D object geometries that span a space of shape complexity and grasp difficulty. In particular, each object is ranked from 1 to 25 according to both measures (shape complexity and grasp difficulty) and placed on a 25 × 25 grid. The 2331 objects were generated while ensuring that the dataset contains no more than four objects belonging to any single grid square.

GraspNet-1Billion is another grasp dataset worth mentioning [24]. Although it includes data drawn from only 88 objects (32 of which are from YCB and another 13 from DexNet 2.0), it is paired with 97k RGBD images of 190 different cluttered scenes containing the objects. Each of these RGBD images is labeled with tens of thousands of SE(3) grasp poses, for a total of 1.1B labeled grasp examples.

ACRONYM is a recent SE(3) dataset comprised of 17.7 million labeled grasps performed on 8872 object meshes[23].

The object meshes were obtained from ShapeNet[21]. The grasps were labeled using the FleX physics simulator[11]to simulate the Franka Panda gripper (maximum grip aperture of 8cm). Observations are available either as depth images or point clouds, produced using PyRender2. A notable aspect of this dataset is that it does not include data in the standard dense clutter setting.

Object-based grasp dataset generation methods can generate large amounts of annotated data and cover various object shapes and surface features, thereby improving the performance of robot grasping tasks. However, this method also has some limitations, such as the potential distribution differences between synthetic and real data, which may reduce the algorithm’s generalization ability. Therefore, this method needs to be combined with real data for training and validation to improve the algorithm’s performance.

Currently, most publicly available datasets online consist of everyday objects, while some specific tasks may require objects with highly distinctive features that are completely different from everyday objects. This can result in the loss of important features for downstream tasks, which may significantly reduce the effectiveness of algorithms that were supposed to perform well. Therefore, it is necessary to supplement the existing object datasets according to downstream tasks.

In addition, for our proposed method, due to the large size of CAD datasets and the relatively poor maintenance of open-source datasets, some everyday objects with significant grasping features are missing in the downloading process.[29] Therefore, we use this method to supplement the missing everyday objects with significant grasping features to ensure the completeness and applicability of the CAD model dataset. Then we process the complete CAD model dataset to extract and generate object models.

We provide a fast and convenient method that can generate CAD models and depth images for objects that need to be supplemented. The specific process is as follows: first, the mechanical arm is used to densely capture images of the object based on depth information. Second, the Nerf method is used to process the captured images and generate CAD models. Finally, the CAD models are imported into pybullet[30] to generate depth images at different angles.

This supplementary process can generate CAD models that comply with file formats and usage requirements, and effectively supplement objects with highly distinctive features that are missing in publicly available CAD datasets. This method can be applied to various specific tasks, providing more comprehensive data support for research and practice in robot grasping algorithms.

Using the advantages of graphics simulators and GPU computing, we randomly sample grasping points for each object and generate grasp configurations based on the normal vectors constructed by nearby points. These large amounts of sampled grasp configurations are simulated in Isaac Gym to extract the point cloud of gripper intersections with objects. Compared to traditional methods such as GPD and GPG, this method has the advantages of clear process and faster speed.

In addition, we extracted the part of the CAD model dataset that can interact with the gripper and removed the gripper point cloud that cannot intersect with the point cloud of the object. For these parts that cannot intersect with the object, $p=\sum_{i}^{n}(x,y,z)(n=0)$, since they cannot form a dataset, they also waste computing resources.

For CAD models generated under static conditions, the grasping area under dynamic conditions is different from that under static conditions, $[R,w]p_{(u,v,t)}\neq p_{(u,v,t)}$. When the gripper is closed, it will come into contact with the object, causing a small displacement of the object and thus a change in the grasping area. Therefore, it is necessary to obtain the grasping area where the gripper intersects with the object. Under static grasping configurations, the grasping area where the gripper intersects with the object may be an invalid area, because when this grasping area is used for grasping, the obtained grasping area is completely different from the static grasping configuration. Moreover, the scoring or labeling of the generated grasping dataset is also based on the static grasping configuration, which may result in a large number of errors. Ignoring the correspondence between dynamic grasping configuration and the grasping area where the gripper intersects with the object will introduce noise into the grasping network, because the grasping network assumes that there is a one-to-one correspondence between the grasping configuration and the gripper point cloud. With this method, we can obtain the noise under dynamic conditions and perform denoising processing on the subsequent generated grasping dataset. The method we designed ensures that the extracted gripper point cloud is in the grasping and intersecting area of the object and will not give high scores or good labels to the wrong areas during scoring or labeling.

In the gripper coordinate system, there are many gripper regions that intersect with the object and have the same shape. These gripper regions with identical shapes are redundant in the input dataset of the network and therefore need to be removed. To solve this problem, we construct a feature filter to remove the overlapping gripper regions that are the same in the object’s coordinate system at a specified resolution of voxel space.

Firstly, the gripper is voxelized in the gripper space, and the voxelization is performed according to the downsampling scale corresponding to the resolution of the grasping network algorithm. Here, we take the example of cm scale to illustrate the feature filter. First, the voxelization expression of the gripper space is constructed:$Length_{gripper}=a,Width_{gripper}=b,Height_{gripper}=c$.

Therefore, the total number of voxels in the gripper space $Sum_{voxel}=a*b*c$

Based on the voxel space at the corresponding resolution, encoding is performed. If a point falls within the voxel grid, the voxel is represented as 1, and if there is no point within the voxel, it is represented as 0. Therefore, the encoded voxel space can be represented as:

Then, the following constraints are used to construct the feature selector to determine whether $A,a_{ij}\in[0,1]$ and $A_{n}^{*},a_{nij}\in[0,1]$ are consistent for the encoded grasp point clouds in the claw space.

We define a matrix power operation between two matrices $a_{1}$ and $a_{n}$, which performs element-wise power and then sum. Additionally, we reverse the values of one of the matrices and substitute the resulting matrix into the constraint, where the dimensionality of the resulting matrix is the same as that of the unit matrix subtracted from $a_{n}$. If the constraint is satisfied, the two matrices are identical, indicating that the shape of the grasping space is the same, and thus the regions of the object intersecting with the gripper that have the same shape are filtered out.

After extracting all the grasping regions that intersect with objects using the simulator Isaac Gym and filtering out regions with redundant shapes using the above method, the next step is to assemble them into an object. The main problem to consider when assembling is how to handle duplicate feature positions in space, as otherwise many features will be obscured by stacking.

To this , we designed a set of assembly classifiers that classify the projected features on the three gripper planes, $(u,v),(u,t),(v,t)$ as the most important features in the gripper space $(u,v,t)$. During the grasping process, the projection of the gripper point cloud on the three different planes is not the same in the gripper coordinate system, $P_{uv}\neq P_{ut}\neq P_{vt}$. Based on this, each gripper point cloud that intersects with the object is projected onto these three planes in the gripper coordinate system, and then features are extracted for classification. After classification, based on the richness of different projection planes, better assembly can be achieved, avoiding the problem of feature masking.

The classifier is designed to extract the most favorable features for grasp generation at the corresponding resolution. For example, if a grasp feature region $P_{i}$ has the most distinct features on a certain projection plane, then this grasp feature region is assembled on that plane to effectively avoid the problem of feature vectors stacking and masking. We use voxelized gripper space with a convolution kernel of $weight=1$ to extract features on the $(v,t)$ plane.

We use the classifier to perform convolutional sliding on the voxelized point cloud projections. When the sum of the convolutional results is equal to 3, we increment the $N_{feature}$ counter. The classifier uses the total number of features to classify the grasping point cloud, obtaining the required edge features, and then assembles the features on the projection plane with the most edge features.

To assemble each feature, we compared the areas of each projection $S_{u,v}$, $S_{u,t}$, and $S_{v,t}$ in the grasping space coordinate system. Specifically, we compared the ratio $S/(S_{u,v}+S_{u,t}+S_{v,t})\times N_{feature}$ and selected the projection area with the maximum ratio as the assembly plane for the corresponding feature. This ensures that the grasping features are maximally preserved. We repeated this process for all features.