Attribute-Based Robotic Grasping with Data-Efficient Adaptation

Though there are different taxonomies, the existing work of robotic grasping can be roughly divided according to approaches and tasks: 1) model-driven [7] and data-driven [8] approaches; 2) indiscriminate [9] [10] and instance grasping [2] tasks. Our approach is data-driven and focuses on instance grasping. Typical instance grasping pipelines assume a pre-trained object recognition module (e.g., detection [2], segmentation [3] [11], template matching [12], and object representation [13], etc.), limiting the generalization for unknown objects and the scalability of grasping pipelines. Our model is end-to-end and exploits object attributes for generalization. Some recent research also proposes end-to-end learning methods for instance robotic grasping. [14] learns to predict the grasp configuration for an object class with a learning framework composed of object detection, classification, and grasp planning. In [15], CCAN, an attention-based network, learns to locate the target object and predict the corresponding grasp affordances given a query image. Compared to these methods, the main features of our work are two-fold. First, we collect a much smaller dataset of synthetic basic objects to learn generic attribute-based grasping. Moreover, our generic grasping model is capable of further adapting to new objects and domains. Second, our approach takes a description text of target attributes as a query command, which is more flexible when grasping a novel object.

Object attributes are middle-level abstractions of object properties and generalizable across object categories [16]. Learning object attributes has been widely studied in the tasks of object recognition [17], [18], [19], [20], while attribute-based robotic grasping has been much less explored, except for [21], [22]. Cohen et al. [21] developed a robotic system to pick up the target object corresponding to a description of attributes. Their approach minimizes the cosine similarity loss between visual and textual embeddings as well as predicts object attributes. However, they only show generalization across viewpoints but not object categories. In [22], the proposed Text2Pickup system uses object attributes to specify a target object and removes ambiguities in the user’s command. They use mono-color blocks as training and testing objects but fail to show generalization to novel objects. In contrast, our work learns generic attribute-based robotic grasping (only using synthetic basic objects) and generalizes well to novel objects and real-world scenes.

Model generalization is one of the most important challenges in robotic manipulation. To improve model generalization, various approaches to bridging domain gaps have been proposed. Domain randomization [23] is one frequently used method, which collects more diverse data by randomizing simulation settings. Some recent research [24], [25], [26] have applied domain randomization to improve the real-world generalization of a simulation-trained robot policy. We build a simulation environment and apply domain randomization during the pre-training of a generic model. In addition to domain randomization, we propose two adaption methods following the form of domain adaptation and few-shot learning. Domain adaptation [5], a subcategory of transfer learning [27], is used to reduce the domain shift between the source and target domain when the feature space is the same but the distributions are different. Inspired by adversarial domain adaptation [28], our approach learns a domain classifier and the image encoder learns domain-invariant features to confuse the classifier. We propose an object-level augmentation method to enrich the image dataset for adversarial training, increasing the generalization of the encoder to new domains. While there exist similar work, for example, Chen et al. [29] investigated domain adversarial training in their work, their approach focuses on updating the feature adaptor and the discriminator using unlabeled data, rather than updating the grasp synthesis model. In contrast, our grasping adaptation approach, consisting of unsupervised adversarial adaptation and supervised few-shot learning, jointly updates the grasping pipeline.

Few-shot learning [30] is the paradigm of learning from a small number of examples at test time. The key of metric-based few-shot learning method, one of the most popular categories, is to supervise the latent space and learn a versatile similarity function by metric loss [31], [32]. The supervised metric space supports fine-tuning using minimal adaptation data (also known as support set), and the similarity function generalizes to unknown test data [33], [34]. Motivated by the idea of few-shot learning methods, our approach first learns a joint metric space that encodes object attributes and then fine-tunes recognition and grasping of our model when testing on novel objects.

We formulate attribute-based grasping as a mapping from pairs of workspace images and query text to target grasping affordances. The proposed visual-textual manipulation architecture assumes no prior linguistic or perceptual knowledge. It consists of two modules, a multimodal encoder and an affordances decoder, as illustrated in Fig. 2.

Multimodal Encoder: As shown in Fig. 1(a), our robot system uses an overhead RGB-D camera to capture the workspace. The RGB-D image is projected into a 3-D point cloud and then orthographically back-projected in the gravity direction to construct a heightmap image $v_{\text{pre}}$ of RGB and depth channel. To specify an object in the image as the grasping target, we give a text command $t$ composed of color and/or shape attributes, e.g., “red cuboid”. The workspace image $v_{\text{pre}}$ and query text $t$ are the input to visual spatial encoder $\phi_{v,\text{spa}}$ and text encoder $\phi_{t}$ respectively. We use the ImageNet-pretrained [36] ResNet-18 [37] backbone as our image encoder $\phi_{v,\text{spa}}$. We replace the first convolutional layer of the ResNet backbone with a 4-channel convolutional layer to match the RGB-D heightmap input. The encoder encodes the RGB and depth observation into 3D visual matrix $\varphi_{v,\text{spa}}\in\mathbb{R}^{H\times W\times 512}$. The text encoder $\phi_{t}$ is a deep averaging network [38] represented by three fully-connected layers and interleaved ReLU [39] activation functions. We first map each token in a sentence text to an embeddings vector of $128$ dimension. The mean token embeddings (i.e., continuous bag-of-words [40] model) of the text are input to the 3-layer MLP text encoder to produce a text vector $\varphi_{t}\in\mathbb{R}^{512}$. The visual matrix $\varphi_{v,\text{spa}}$ and the text vector $\varphi_{t}$ are then fused by the gated-attention mechanism [41]: each element of $\varphi_{t}$ is repeated and expanded to an $H\times W$ matrix to match the dimension of $\varphi_{v,\text{spa}}$. The expanded matrix is multiplied element-wise with $\varphi_{v,\text{spa}}$ to produce a fusion matrix $F_{\text{att}}$. The gated-attention unit is designed to gate certain pixels in the visual feature matrix matching to the text vector, resulting in the fusion matrix containing the visual features selected by the query text. By this means, we can detect different attributes of the objects in the image, such as color and shape.

Affordances Decoder: Grasping affordances decoder $\phi_{g}$ is a fully-convolutional residual network [37], [42] interleaved with spatial bilinear $4\times$ upsampling and ended with the sigmoid function. The decoder takes as input the fusion matrix $F_{\text{att}}$ and outputs a unit-ranged map $Q_{g}$ with the same size and resolution as the input image $v_{\text{pre}}$. Each value of a pixel $q_{i}\in Q_{g}$ represents the predicted score of target grasping success when executing a top-down grasp at the corresponding 3D location with a parallel-jaw gripper oriented horizontally concerning the map $Q_{g}$. The grasping primitive is parameterized by a 3-D location and an angle. To examine different grasping angles, we rotate the input $F_{\text{att}}$ by $N=6$ (multiples of $30^{\circ}$) orientations before feeding into the decoder, which predicts pixel-wise scores of horizontal grasps within the rotated heightmaps. The pixel with the highest score among all the $N$ maps determines the parameters (i.e., location and angle) for the grasping primitive to be executed. As in Fig. 2, our model predicts accurate target grasping location and valid (e.g., the selected angles for the red cuboid) target grasping angle.

The motion loss $\mathcal{L}_{grasp}$, which supervises the entire encoder-decoder networks, is the error from predictions of grasping affordances:

where $N_{s}$ is the size of the dataset that is collected in simulation, $q_{e}$ is the grasping score in $Q_{g}$ at the executed location, and $\bar{q}_{e}$ is the ground-truth label (see Sec. IV-C). The second term ensures lower grasping scores for the pixels in background mask $M$ (obtained from the depth image) with weight $\lambda_{M}$ [43], and $q_{i}$ is the grasping score of a background pixel.

To learn generic object attributes, we perform multimodal attributes learning, where visual or textual embedding vectors corresponding to similar attributes are encouraged to be closer in the latent space. Inspired by [13], we take advantage of the object persistence: the embedding difference of the scene before and after grasping is enforced closer to the representation of the grasped object. During data collection, we record image-text data $(v_{\text{pre}},v_{\text{post}},t)$, where $v_{\text{pre}}$ and $v_{\text{post}}$ are the workspace image before and after grasping respectively, and $t$ is the query text that describes attributes of the grasped object.

We add one layer of global average pooling (GAP) [44], [45] at the end of the encoder $\phi_{v,\text{spa}}$ and denote the network as visual vector encoder $\phi_{v,\text{vec}}$. The output from $\phi_{v,\text{vec}}$ is a visual embedding vector that represents the average of scene features and has the same dimension of $\phi_{t}(t)$. We express the logic of the object persistence as an arithmetic constraint on visual and textual vectors such that $(\phi_{v,\text{vec}}(v_{\text{pre}})-\phi_{v,\text{vec}}(v_{\text{post}}))$ is close to $\phi_{t}(t)$. We use the triplet loss [46] to approximate the constraint, and the set of triplets $\mathcal{T}$ is defined as

where $f_{i}$, $f_{i}^{+}$ and $f_{i}^{-}$ are random samples from the pool of vectors $(\phi_{v,\text{vec}}(v_{\text{pre}})-\phi_{v,\text{vec}}(v_{\text{post}}))$ and $\phi_{t}(t)$, and $\mathpzc{a}_{f}$ is an $n$-dimensional attribute label vector corresponding to the feature vector $f$ (e.g., color, shape, and category name, etc.). Function $s(\cdot,\cdot)$ is an attribute similarity function that evaluates the similarity between two attribute label vectors:

where $\mathpzc{a}^{i}$ denotes the $i$-th element of the label vector $\mathpzc{a}$, and the indicator function $\mathds{1}(\cdot,\cdot)$ evaluates the element-wise similarity. Note that $0$ indicates null attribute meaning no attribute is specified in the label. As an example, suppose we have the dictionary $\text{dict}=\{``eos":0,``red":1,``black":2,``yellow":3,``cylinder":4,``cube":5\}$; then, “red cylinder” can be represented as a label vector $\mathpzc{a}_{f_{0}}=[1,4]$, and “red cube” can be represented as $\mathpzc{a}_{f_{1}}=[1,5]$. The similarity between the two label vectors is computed using (3) such that $s(\mathpzc{a}_{f_{0}},\mathpzc{a}_{f_{1}})=s([1,4],[1,5])=0.5$. Additionally, when “red” and “black” are used without any additional attribute description, they are mapped to the vectors $[1,0]$ and $[2,0]$, respectively. In this case, the similarity between the two labels is derived as $s([1,0],[2,0])=0$. With the triplets of embedding vectors, multimodal metric loss $\mathcal{L}_{attr}$ is defined as

where $\alpha$ is a hyperparameter that controls the margin between positive and negative pairs. By encoding workspace images and query text into a joint metric space and supervising the embeddings through the equation of object persistence (as shown in Fig. 3), we learn generic attributes that are consistent across object categories, as discussed in Sec. VII-A.

To achieve self-supervision, we create a simulation environment in which objects are identified and grasped based on a description of semantic attributes. We collect training data in simulation with the following procedure, as summarized in Algorithm 1. Several objects are randomly dropped into the workspace in front of the robot. Given a workspace image and a query text, the robot learns to grasp a target under $\epsilon$-greedy exploration [47] ($\epsilon\!=\!0$ during testing, i.e., an argmax policy). We save the workspace images, query text, background masks, executed actions, and results into a bounded buffer. The ground-truth labels are automatically generated for learning grasping affordances. The label $\bar{q}_{e}$ in (1) is assigned as the attribute similarity in (3) between the query text and the grasped object (0 if no object grasped). We also save the workspace image after a successful grasping for learning object attributes. To deal with sparse rewards in target-driven grasping, the hindsight experience replay (HER) technique [48] is applied. If a non-target is grasped, we add the additional positive sample by relabeling the target text. Fig. 4 shows the basic objects of various colors (red, green, blue, yellow, and black) and shapes (cube, cuboid, cylinder, and sphere) used in our simulation. We choose these colors and shapes because they are foundational for common objects in daily life. To enrich the distribution of training data, we perturb color RGB values, randomize sizes and heights of the objects, and randomize textures of the workspace. Using the domain randomization techniques [24], we can generate a number of randomized properties in simulation and achieve a model with better generalization. At every iteration, we sample a batch from the buffer and run one off-policy training. The training loss is defined as

by combining both motion loss $\mathcal{L}_{grasp}$ (1) and metric loss $\mathcal{L}_{attr}$ (5). We train the proposed model using stochastic gradient descent with a learning rate of $10^{-4}$, momentumn of $0.9$, and weight decay of $2\times 10^{-5}$ for 5k iterations. Each training iteration involves capturing data, computing a forward pass, executing an action, and backpropagating. After collecting a dataset of 5k samples, we replay the entire data for 100 epochs.

Despite that our generic model trained using the simulated basic objects shows good generalization (see Sec. VII-B), the visual feature shifts (e.g., objects, lighting conditions, and scene configurations, etc.) are inevitable. As a result, the image encoder is likely to produce out-of-distribution visual embeddings, leading to the failure of the grasping model. To reduce the influence of the domain shifts, we propose to use adversarial adaptation [28] to learn domain-invariant visual features that are transferable across different domains. In our problem setup, the simulated basic objects constitute the source domain, and our goal is to transfer the learned model to a target domain that is prone to domain shifts.

As shown in Fig. 6(a), adversarial adaptation regularizes the weights of the image encoder $\phi_{v}$ by enforcing a two-player game similar to the generative adversarial network (GAN) [49]. A domain classifier (i.e., discriminator) learns to distinguish between two domains, while the image encoder learns to fool the domain classifier by learning domain-invariant features. To achieve adversarial training, we connect the encoder and the discriminator via a gradient reversal layer (GRL) [50] that has reverse forward and back-propagation schemes. The GRL $\mathcal{R}$ is an identity mapping during forward-propagation but reverses the sign of the gradients during back-propagation:

where $I$ is an identity matrix, and $\lambda_{r}$ is a positive constant.

During adaptation, images from the two different domains are fed into the image encoder. The domain classifier $f_{d}$ takes as input the encoded features, and is trained to predict which domain the feature is from, by maximizing the binary cross-entropy $\mathcal{H}_{d}$. Using the source domain dataset $\mathcal{D}{s}=\{v{s},t{s},y{s}\}^{N{s}}$ ($\{$image, text, label$\}$ collected in Algorithm 1) and assuming a target domain dataset $\mathcal{D}{t}=\{v{t}\}^{N{t}}$, the model is updated to be optimal on the training loss $\mathcal{L}_{train}$ (6) under the regularization of the adversarial loss $\mathcal{L}_{adv}$

where $\mathcal{L}_{advadp}$ is the overall loss for adversarial adaptation, $d_{i}\in\{0,1\}$ are the binary domain label for each input $v_{i}$. Note that the reversal layer $\mathcal{R}$ is augmented between the classifier $f_{d}$ and the encoder $\phi_{v,\text{vec}}$ and updates them in reversal directions.

The target dataset $\mathcal{D}{t}$ for the new domain is the prerequisite to performing adversarial adaptation. As shown in Fig. 7, we propose a object-level augmentation (ObjectAug) approach instead of collecting data from a vast number of configurations. Since the grasping label is not required in $\mathcal{D}{t}$, synthetic generation of a large image dataset would be more efficient. We begin by collecting RGB-D images of all conceivable objects in the target domain. Using the object mask acquired from the depth channel, we extract each object individually and randomize them with the single-object augmentation methods commonly used (e.g., scaling, flipping, and rotation). To generate an augmented RGB-D image, the augmented objects are randomly sampled and shifted before being overlaid with a background image. We also perform IoU threshold verification to avoid dense overlapping. To simulate varying conditions of target domains, we can apply the visual jitter technique discussed in Sec. VI-A to obtain more diverse data. Given the ease with which unlabeled images may be acquired (e.g., the internet, image collection), generating a target dataset that is of the same magnitude as the source dataset is rather efficient.

By learning domain-invariant features, the adversarial adaptation technique in Sec. V-A improves model generalization using unlabeled images of the target domain. However, the adversarial loss uses unlabeled images to only update the image encoder and leave the text encoder and the grasping affordance decoder unadapted. When deploying in a new domain, end-to-end model fine-tuning is often necessary, but this comes at the cost of a large dataset covering all potential testing object configurations. To further adapt to novel objects and new scenes in a data-efficient manner, we present a one-grasp adaptation scheme (see Fig. 6(b)) that only requires one successful grasp of a novel object. The inductive bias of object attributes in Sec. IV-B is the key to adaptation in this limited-data regime. If similar objects are enforced closer in the embedding space, the adaptation distance for a novel object is likely to be shorter [51].

The proposed one-grasp adaptation method improves the model performance on a novel target object at the cost of only one grasp. The adaptation data is collected with the following one-grasp data augmentation (OneGraspAug) procedure. We place the object solely in the workspace and run the generic model to collect one successful grasp. The setting of a sole object facilitates grasping and avoids combinatorial object arrangements. Because convolutional neural networks are not rotation-invariant by design, we also augment the grasp data by rotating with various orientations to achieve rotation-invariance [52], [53], i.e., the ability to recognize and grasp an object regardless of its orientation. As shown in Fig. 8, we rotate the collected image and action execution to have rotated versions of the collected data.

In the adaptation stage, we add the category name of the object as an additional token to the query text, e.g., “apple, red sphere” for the testing object apple. The token embedding of the object name is initialized properly to keep the embedding vector of the query text unchanged. The addition of the object name allows for a more specific grasping instruction and distinguishing from similar objects via adaptation. By optimizing over motion loss $\mathcal{L}_{grasp}$ in (1), we jointly fine-tune the recognition and grasping of our model for unknown objects and scenes. As delineated in Sec. VII-C, the adapted model outputs higher affordances on the target objects that are not seen and grasped before.

We use CoppeliaSim [54] to build our simulation environment. The simulation setup includes a UR5 robot arm and an RG2 gripper, with Bullet Physics 2.83 for dynamics and CoppeliaSim’s internal inverse kinematics module for robot motion planning. We simulate a statically mounted overhead 3D camera in the environment from which perception data is captured. The camera renders RGB-D images with a $640\times 480$ pixels resolution using OpenGL. We use various basic and novel objects for training and testing in the simulation. As shown in Fig. 4, the basic objects consist of 36 different 3D toy blocks, whose shapes and colors are randomly chosen during experiments. We collect 34 different 3D household objects from the YCB dataset [55] or the 3D mesh library SketchUp [56], as shown in Fig. 5(a). To produce more diverse simulation configurations, the simulation environment supports several domain randomization techniques, including background randomization, color jitter (i.e., randomly changing the brightness, contrast, and saturation of the color channel), and depth jitter (i.e., adding Gaussian noise to the depth channel).

Fig. 1(a) shows our real-world setup that includes a Franka Emika Panda robot arm with a FESTO DHAS soft gripper and a hand-mounted Intel RealSense D415 camera overlooking a tabletop scene. We use the soft fingers because they are more suitable for grasping the objects in our experiments and are similarly compliant to the RG2 fingers. For perception data, RGB-D images of resolution $640\times 480$ are captured from the RGB-D camera, statically mounted on the robot arm. The camera is localized with respect to the robot base using the automatic calibration procedure of ViSP [57], during which the camera tracks the location of a checkerboard pattern taped on the table. For a given pose, the robot follows the corresponding trajectory generated with MoveIt [58] in open-loop. The entire system is implemented under the Robot Operating System (ROS) framework and runs on a PC workstation with an Intel i7-8700 CPU and an NVIDIA 1080Ti GPU. Objects vary throughout tests, with a collection of 20+ different household objects being used to test model generalization to novel objects, as shown in Fig. 5(b).

By embedding workspace images and query text into a joint metric space, the multimodal encoder ($\phi_{v}$ and $\phi_{t}$), supervised by metric loss $\mathcal{L}_{attr}$ and motion loss $\mathcal{L}_{grasp}$, learns attending to text-correlated visual features. We visualize what our model “sees” by computing the dot product of text vector $\varphi_{t}$ with each pixel of the visual matrix $\varphi_{v,spa}$. This computation obtains an attention heatmap over the image, which refers to the similarity between the query text and each pixel’s receptive field (see Fig. 9(a)). We quantitate the attention of our model and report its attention localization performance in Table II (see Ours-Attention). Evaluation metrics: An attention localization is considered correct only if the maximum value in the attention heatmap lies on the target object.

Ours-Attention (in Table II) performs target localization at a 74.5% accuracy on simulated novel objects and a 70.2% accuracy on real-world objects, without any localization supervision provided. In summary, our multimodal embeddings demonstrate a consistent pattern across object categories and scenes. Though the localization results are not directly used for grasping, the consistent embeddings facilitate learning, generalization, and adaptation of our grasping model, as shown in Fig. 9 and discussed in the following subsections.

We compare the instance grasping performance of our generic model with the following baselines:

• 1)

  Indiscriminate is an indiscriminate grasping version of our approach and composed of a visual spatial encoder $\phi_{v,\text{spa}}$ and a grasping affordances decoder $\phi_{g}$. We collect a dataset of binary indiscriminate grasping labels and train Indiscriminate using $\mathcal{L}_{grasp}$ in (1).

• 2)

  ClassIndis extends Indiscriminate with an attributes classifier that is trained to predict color and shape attributes on cropped object images. We filter the grasping maps from Indiscriminate using the mask of a target recognized by the classifier.

• 3)

  EncoderIndis is similar to [21] and is another extension of Indiscriminate, which leverages a multimodal encoder ($\phi_{v,\text{vec}}$ and $\phi_{t}$ in Sec. IV-A) for text template matching. The encoder is trained using $\mathcal{L}_{attr}$ in (5) to evaluate the similarity between each cropped object image and query text. During training, we also include attributes classification as an axillary task.

• 4)

  AttrID is for an ablation study of our text encoder $\phi_{t}$. The only difference between AttrID and the proposed method is that AttrID takes the attribute shape and color ID one-hot encoding as the system input, but our generic model uses Word2Vec continuous bag-of-words (CBOW) model to convert the texts into vector inputs. During training, we use both the motion loss and the metric loss to update the model.

• 5)

  NoMetric is for an ablation study of multimodal metric loss. We simply remove the metric loss on the basis of our approach during its training.

Evaluation metrics: These methods have different target recognition schemes: ClassIndis and EncoderIndis recognize a target by classification and text template matching respectively; NoMetric, AttrID and Ours are end-to-end. We report their target recognition performance (in addition to instance grasping performance, as in the next paragraph). A target localization is correct only if the predicted grasping location lies on the target object. The instance grasping success rate is defined as $\frac{\text{\# of successful grasps on correct target}}{\text{\# of total grasps}}$. In each testing scene, we only execute grasping once.

We evaluate the methods on both simulated basic (sim basic) and simulated novel (sim novel) objects in simulation, where there are 1200 test cases for the basic objects (Fig. 4) and 3400 test cases for the 34 novel objects (Fig. 5(a), mostly from the YCB dataset [55]). We assume the objects are placed right-side up to be stable while their 4D pose (3D position and a yaw angle) can vary arbitrarily. For each testing object, we pre-choose a query text that best describes its color and/or shape. In each test case, four objects are randomly sampled and placed in the workspace, except avoiding any two objects with the same attributes. The robot is required to grasp the target queried by an attribute text. We report the results of target recognition in Table II and the results of instance grasping in Table II.

Overall, our approach outperforms the baselines remarkably (in both recognition and grasping) and achieves a 98.4% grasping success rate on the simulated basic objects and an 72.1% grasping success rate on the simulated novel objects. ClassIndis extends Indiscriminate that is well trained in target-agnostic tasks and performs well on the basic objects, but the attributes classifier generalizes poorly. EncoderIndis utilizes a more generalizable recognition module and performs better on the novel objects. However, EncoderIndis fails to reach optimality because its separately-trained recognition and grasping modules have different training objectives from instance grasping. Despite training the recognition and grasping modules simultaneously, AttrID using sparse attribute one-hot IDs as a substitute for text inputs yields lower recognition accuracy and grasping success rate compared to Ours. As an ablation study, the performance gap between NoMetric and Ours shows the effectiveness of multimodal metric loss, which supervises the joint latent space to produce consistent embeddings, as discussed in Sec. VII-A. Our approach successfully learns object attributes that generalize well to novel objects, as shown in Fig. 9.

We further evaluate our approach and the baselines on the real robot before any adaptation (see Table II and II). Fig. 5(b) shows 21 testing objects of various colors and shapes used in our real-robot experiments. The robot is tasked to grasp the target within a combination of 6 objects placed on the table. We use the same 21 object combinations that are randomly generated and repeat each combination twice, resulting in a total of 252 grasping trials for each method. Overall, the grasping performance of all the methods decreases due to the domain gap. However, our approach shows the best generalization and achieves a 63.1% grasping success rate, before adaptation, in the real-world scenes.

The generic model in Sec. VII-B infers the object closest to the query text as the target. Overall, our generic model demonstrates good generalization despite the gaps in the testing scenes. Specifically, these gaps are 1) RGB values of the testing objects deviate from training ranges, 2) some testing objects are multi-colored, 3) shape and size differences between the testing objects and the training objects, and 4) depth noises in the real world causing imperfect object shapes. To account for the gaps, we further adapt our generic model to increase instance recognition and grasping performance.

We first collect one successful grasp of a solely placed target object and then augment the collected data by rotating with additional $N-1$ angles, as discussed in Sec. V-B and shown in Fig. 8. The compared methods that are adapted with the same adaptation data are as follows:

• 1)

  ClassIndis updates its attributes classifier for a better recognition accuracy on the adaptation data.

• 2)

  EncoderIndis minimizes the latent distance between cropped target images and query text to improve text template matching.

• 3)

  NoMetric takes as input images and text, and minimizes motion loss on the adaptation data.

• 4)

  One-Grasp is our prior work [1] which updates the encoder-decoder in an end-to-end manner.

• 5)

  AttrID-One-Grasp uses the same adaptation method and data as One-Grasp baseline on AttrID from Sec. VII-B.

We also update Indiscriminative grasping in ClassIndis and EncoderIndis. We keep the experimental setup the same with Sec. VII-B and evaluate the instance grasping performance of the adapted models. We report the adapted instance grasping success rate in Table III and the adaptation gains in Fig. 10. The attributes classifier in ClassIndis suffers from insufficient adaptation data, limiting its target recognition and adaptation performance. While EncoderIndis minimizes embedding distances in its latent space and shows better performance, it is still worse than One-Grasp. By fine-tuning over the structured metric space, One-Grasp updates the end-to-end model and improves target recognition and grasping jointly. At the cost of minimal data collection, One-Grasp achieves an 83.7% grasping success rate on the simulated novel objects and an 76.6% grasping success rate on the real objects, which shows the significant adaptation gains. On the contrary, the unstructured latent space in NoMetric limits its adaptation, demonstrating the importance of attributes learning for grasping affordances learning. The difference in adaptation performance between NoMetric and One-Grasp demonstrates the significance of the regulated feature space in adaptation. Furthermore, the substantial adaptation gain observed in AttrID validates the applicability of our adaptation method with sparse feature inputs.

As an improvement, we propose applying Adversarial adaptation on the image encoder to learn domain-invariant features before One-Grasp adaptation. The image encoder’s domain invariancy results in superior transferring performance for Adversarial+One-Grasp: an instance grasping success rate of 86.0% in the domain of sim novel and 81.7% in the domain of real-world novel (real novel). The qualitative results in Fig. 11 suggest the efficacy of the two adaptation methods: 1) both Adversarial adaptation and One-Grasp adaptation increase the recognition and grasping performance of the models, and 2) Adversarial adaptation minimizes grasping noises around non-target objects (by reducing domain feature changes), while One-Grasp adaptation can rectify recognition and grasping errors through end-to-end updates. The more compact and centered contour in Fig. 11(b) could be explained by the hypothesis that domain-invariant features improve the output consistency of the encoder across domains. The corrected target recognition in Fig. 11(c), on the other hand, is attributed to the One-Grasp adaptation which effectively shifts the affordances from irrelevant objects to the target object through the end-to-end model updates. Another noteworthy finding is that the combined adaptation Adversarial+One-Grasp appears to benefit from both Adversarial adaptation and One-Grasp adaptation, as their focuses are complementary.

In addition to the instance grasping experiments, we perform another quantitative evaluation of the adapted model on sim novel objects. For comparison, we utilize the CLIP model [59], a recently prevailing multimodal (text and image) foundation model. CLIP aligns language and image features through training with millions of text-image pairs and is widely acclaimed for its robust generalization capability across various objects. To evaluate the performance of CLIP, we segment and crop randomly placed objects in the workspace. For each object crop and its attribute text description, the CLIP model calculates a matching score, which is then used to construct the confusion matrix shown in Fig. 12(a). The pairwise matching scores reflect the similarity between the object crops and the text inputs. As for our Adversarial+One-Grasp model, we compute another confusion matrix shown in Fig. 12(b) by collecting the highest affordance value, under each target attribute description, from affordance maps $Q_{g}$ within each object’s segmentation mask. While CLIP matching uses cropped images of single objects, Adversarial+One-Grasp is tested using workspace images of multiple objects, which is a more natural but also more challenging setting. Fig. 12 shows that Adversarial+One-Grasp achieves more accurate grounding of the correct target objects. In contrast, the matching scores produced by the CLIP model often lack discrimination, leading to some misclassifications (e.g., “sphere soccer” and “yellow cylinder yellowcup”). This comparison highlights the limitations of zero-shot generalization in a foundation model and showcases the effectiveness of our adaptation method.

The proposed adaptation approaches efficiently improve the instance grasping performance of our model. As discussed in Sec. VII-C, the finding that the approaches have complementary adaptation focuses leads to one of the major features: the two adaptation methods can be employed individually or in combination, depending on the availability of adaptation data. To investigate their independent and combinative performance, we conduct an ablation study to compare the grasping models as follows:

• 1)

  Generic is the baseline generic model obtained in Sec. VII-B before any adaptation.

• 2)

  Adversarial adapts the generic model to learn domain-invariant features using a large augmented data, as discussed in Sec. V-A.

• 3)

  One-Grasp adapts the generic model using one grasping trial of the target object, as discussed in Sec. V-B.

• 4)

  Adversarial+One-Grasp applies the two adaptation approaches on the generic model sequentially to improve the recognition and grasping.

As shown in Fig. 14, the adaptation methods are compared in four testing environments with an increasing extent of domain shifts: 1) sim basic jitter—simulated basic objects with visual jitter (see Sec. VI-A) applied on color and depth channels as well as background, 2) sim novel—simulated novel objects, 3) sim novel jitter—simulated novel objects with visual jitter, and 4) real novel—real novel objects. As the training environment uses simulated basic objects, the testing environments include the domain shifts caused by novel objects, visual jitter, and real-world noises.

We report the experimental results of the instance grasping success rate in Fig. 13. Overall, all adaptation methods improve the grasping performance across the testing environments. It is not surprising that adaptation is likely to be more effective (i.e., leading to more performance increments) if the domain shifts are severer. For example, the adaptation gain in the environment of sim basic jitter is less than 4%, while the real environment witnesses an adaptation gain of over 18%. Moreover, when encountering complex novel objects that are more challenging (e.g., drill) than the basic training objects (e.g., red cuboid), the One-Grasp method provides more adaptation power than the Adversarial method. In Adversarial adaptation, we use unlabeled data from the target domain to update the image encoder, while the text encoder and grasping decoder remain unadapted. As a result, the Adversarial adapted model is more prone to encountering difficulties with challenging novel objects (e.g., drill and spatula). On the other hand, One-Grasp adaptation adapts the entire model and demonstrates better performance on these challenging objects, but it requires additional labeled data (at least one grasp) of the objects. Another observation is that we can combine the two adaptation methods to achieve an even higher adaptation performance. Through various tests, the combined method Adversarial+One-Grasp adaptation consistently shows the best performance in all the testing environments. This suggests that the two adaptation methods adapt our model complementarily for accumulative adaptation gains. In practice, we can choose a configuration of adaptation methods based on the availability of the adaptation data.

The quality of adaptation data is critical for grasping adaptation. The two data augmentation methods, ObjectAug and OneGraspAug, are proposed for the two adaptation methods respectively. We execute the ablation studies in Table V and Table V to examine the augmentation methods. The results of adapted instance grasping are presented in the tables. In the ablations for object-level augmentation, the compared approaches are

• 1)

  Objects uses the raw data (images of single objects) as the adaptation data.

• 2)

  ObjectOverlay overlays the randomly sampled objects on the background image to synthesize a large dataset covering possible object combinations and locations.

• 3)

  ObjectAug is our object-level data augmetation method discussed in Sec. V-A, where much richer object configurations (i.e., orientations and scales) are covered in the synthesized dataset.

For the above augmentation methods, we keep the dataset size constant and use each augmentation data in (9), accordingly. Even though the data is unlabeled, the adaptation data quality has a direct impact on grasping performance, as seen in Table V. The performance difference between Objects and ObjectAug, for example, is up to 4% on real novel objects, despite the fact that they nominally contain the identical objects. This finding demonstrates that the suggested object-level data augmentation successfully reduces domain shift by supplying rich unlabeled data.

In the ablations for one-grasp augmentation, the comparable approaches include

• 1)

  OneGrasp uses the raw data of one successful grasp trial, including an RGB-D image and the corresponding grasping action.

• 2)

  OneGraspRpt simply repeats the one-grasp data $N$ times without rotating the data, where $N=6$ is the angle discretion parameter.

• 3)

  OneGraspAug is our one-grasp data augmetation method discussed in Sec. V-B, where we augment the one-grasp data by rotating for $N$ orientations to enrich possible orientations of objects and robot grasping.

As shown in Table V, OneGraspAug outperforms the compared methods by over 4% grasping success rate on real robot experiments, which demonstrates how angle-augmented data can be used to make the grasping model rotation-invariant for object recognition and grasping.