Towards Open-World Grasping with
Large Vision-Language Models

Large Vision-Language Models VLMs receive a set of RGB images of size $H\times W$: $\mathcal{I}_{1:M},\;\mathcal{I}\in\mathbb{R}^{H\times W\times 3}$ and a sequence of text tokens $\mathcal{T}$, and generate a text sequence $\mathcal{Y}$ of length $L$: $\mathcal{Y}\doteq w_{1:L}=\left\{w_{1},\dots,w_{L}\right\}$ from a fixed token vocabulary $w_{i}\in\mathcal{W}$, such that: $\mathcal{Y}=\mathcal{F}(\mathcal{I}_{1:M},\mathcal{T})$. The images-text pair input $\mathcal{X}=\left\langle\mathcal{I}_{1:M},\,\mathcal{T}\right\rangle$ is referred to as the prompt, with the text component $\mathcal{T}$ typically being a user instruction or question that primes the VLM for a specific task.

Grasp Representations We represent a grasp via an end-effector gripper pose $\mathcal{G}$, with $\mathcal{G}\in\mathbb{R}^{4}$ for 4-DoF and $\mathcal{G}\in\mathbb{R}^{6}$ for 6-DoF grasping. Such representation contains a 3D position and either a yaw rotation or a full SO(3) orientation for 4-DoF and 6-DoF respectively. 4-DoF grasps assume that the approach vector is calibrated with the camera extrinsics, and hence can be directly drawn as rectangles in the 2D image plane (see bottom of Fig. 2), which happens to be a favorable representation for VLMs, as grasp candidates can be interpreted as part of the input image prompt. A motion primitive is invoked to move the arm to the desired gripper pose $\mathcal{G}$, e.g. via inverse-kinematics solvers. ¹¹1More sophisticated motion planning algorithms, e.g. with integrated obstacle avoidance, can be utilized orthogonal to our approach.

Problem Statement Given an RGB-D observation $\mathcal{I}_{t}\in\mathbb{R}^{H\times W\times 3}$, $\mathcal{D}_{t}\in\mathbb{R}^{H\times W}$ and an open-ended language query $\mathcal{T}$, which conveys an instruction to grasp a target object, the goal of OWG is to provide a policy $\pi(a_{t}\mid\mathcal{I}_{t},\mathcal{D}_{t},\mathcal{T})$. Assuming $n\in\{1,\dots,N\}$ the $N$ objects that appear in the scene and $n^{*}$ the target object, then at each time step $t$, the policy outputs a pose for grasping an object: $a_{t}=G_{t}(n),\;\;G_{t}(n)=G(n,\mathcal{I}_{t},\mathcal{D}_{t}),\;t=1,\dots,T$, where the last step $T$ always maps to grasping the target object: $a_{T}=G_{T}(n^{*})$. We refer to the function $G$ as the grasp generation function, which corresponds to a pretrained grasp synthesis network from RGB-D views [38] ²²2Other point-cloud [59] or voxel-based [62] methods for 3D grasp generation can be utilized orthogonal to our approach, which uses single RGB-D view. We note that our policy $\pi$ outputs directly the actual gripper pose $\mathcal{G}=G(n)$, and the object-centric abstraction $n$ is used implicitly (details in next sections).

We wish to highlight that in most grasp synthesis pipelines [38, 55, 53, 57, 56], it’s always $T=1$ and $a_{1}=G_{1}(n^{*})$, which corresponds to an open-loop policy attempting to grasp the object of interest once. Our formulation for $T>1$ allows the VLM to close the loop by re-running after each step, which enables visual feedback for planning and recovery from failures / external disturbances.

OWG combines VLMs with pretrained 2D instance segmentation and grasp synthesis models. Segmentation methods like SAM [37] and its variants [70, 71] have demonstrated impressive zero-shot performane. Similarly, view-based grasp synthesis networks [56, 55, 38, 53, 57] have also shown to be transferable to unseen content, as they are trained without assumptions of objectness or semantics in their training objectives. The zero-shot capabilities of these models for low-level dense spatial tasks is complementary to the high-level semantic reasoning capabilities of VLMs, while both use images as the underlying representation, hence offering a very attractive coupling for tackling the open-world grasping problem. The overall pipeline can be decomposed in three subsequent stages: (i) open-ended referring segmentation, (ii) grounded grasp planning, and (iii) grasp generation and ranking. A schematic of OWG is shown in Fig. 2 and described formally in Algorithm 1. Prompt implementation details can be found in Appendix A.

Open-ended referring segmentation In this stage, the target object of interest must be segmented from the input RGB image $\mathcal{I}_{t}$ given the instruction $\mathcal{T}$. To enable this, we first run our segmentation model $S:\mathbb{R}^{H\times W\times 3}\rightarrow\{0,1\}^{H\times W}$ and then draw the $N$ generated masks $M_{1:N}=S(\mathcal{I}_{t})$ with additional visual markers in a new frame $\mathcal{I}_{t}^{m}$. This step aims to exploit the VLM’s OCR capabilities and link each segment in the frame with a unique ID that the VLM can use to refer to it. After augmenting the image with visual markers, we pass the prompt $<\mathcal{I}_{t},\mathcal{I}_{t}^{m},\mathcal{T}>$ to the VLM. We refer to this VLM generation as $\mathcal{F}^{ground}$, such that: $n^{*}=\mathcal{F}^{ground}(\mathcal{I}_{t},\mathcal{I}_{t}^{m},\mathcal{T})$ where $n^{*}$ the target object and $M_{n^{*}}$ its segmentation mask. We note that $\mathcal{T}$ can contain free-form natural language referring to a target object, such as open object descriptions, object relations, affordances etc.

Grounded grasp planning This stage attempts to leverage VLM’s visual reasoning capabilities in order to produce a plan that maximizes the chances that the target object $n*$ is graspable. If the target object is blocked by neighboring objects, the agent should remove them first by picking them an placing them in free tabletop space. Similar to  [27], we construct a text prompt that describes these two options (i.e., remove neighbor or pick target) as primitive actions for the VLM to compose plans from. We provide the marked image $\mathcal{I}_{t}^{m}$ together with the target object $n^{*}$ (from the previous grounding stage) to determine a plan: $p_{1:T}=\mathcal{F}^{plan}(\mathcal{I}_{t}^{m},n^{*}),\;p_{\tau}\in\{1,\dots,N\}$. Each $p_{\tau}$ corresponds to the decision to grasp the object with marker ID $n\in\{1,\dots,N\}$. As motivated earlier, in order to close the loop, we take the target of the first step of the plan $\tilde{n}=p_{1}$ and move to the grasping stage of our pipeline.

Grasp generation and ranking After determining the current object to grasp $\tilde{n}$, we invoke our grasp synthesis model $G$ to generate grasp proposals. To that end, we element-wise multiply the mask $M_{\tilde{n}}$ with the RGB-D observation, thus isolating only object $n^{*}$ in the input frames: $\tilde{\mathcal{I}_{t}}=\mathcal{I}_{t}\odot M_{\tilde{n}},\;\tilde{\mathcal{D}_{t}}=\mathcal{D}_{t}\odot M_{\tilde{n}}$. The grasp synthesis network outputs pixel-level quality, angle and width masks which can be directly transformed to 4-DoF grasps $\mathcal{G}_{1:K}=G(\tilde{\mathcal{I}_{t}},\tilde{\mathcal{D}_{t}})$ [56, 55, 38], where $K$ the total number of grasp proposals. Then, we crop a small region of interest $c_{\tilde{n}}$ around the bounding box of the segment in the frame $\mathcal{I}_{t}$, from its mask $M_{\tilde{n}}$. We draw the grasp proposals $\mathcal{G}_{1:K}$ as 2D grasp rectangles within the cropped image $c_{\tilde{n}}$ and annotate each one with a numeric ID marker, similar to the grounding prompt. We refer to the marked cropped frame as $c_{\tilde{n}}^{\prime}$. Then, we prompt the VLM to rank the drawn grasp proposals: $\mathcal{G}_{1:K}^{\prime}=\mathcal{F}^{rank}(c_{\tilde{n}}^{\prime})$ where the prompt instructs the VLM to rank based on each grasp’s potential contacts with neighboring objects. Finally, the grasp ranked best by the VLM $\mathcal{G}_{1}^{\prime}$ is selected and sent to our motion primitive for robot execution.

In order to evaluate the open-ended potential of OWG for grounding, we create a small subset of OCID-VLG test split [39], which we manually annotate for a broad range of grasping instructions. As we strive for zero-shot usage in open scenes, we mostly experiment with previous visual prompting techniques for large-scale VLMs, such as CLIP [43, 44, 72], as well as the recent Set-of-Mark prompting methodology for GPT-4v [26], which constitutes the basis of our method. We also include comparisons with open-source visually-grounded LVLM QWEN-VL-2 [31]. Please see Appendix C for details on the test dataset, baseline implementations and more comparative ablations and qualitative results.

We observe that both CLIP-based visual prompting techniques and open-source LVLMs are decent in object-based but fail to relate objects from the visual prompts. Even GPT-4v-based SoM prompting method is not directly capable of handling cluttered tabletop scenes from depth cameras, as is evident by the $53.1\%$ averaged mIoU across all query types. Overall, our OWG-grounder achieves an averaged mIoU score of $80.8\%$, which corresponds to a $27.7\%$ delta from the second best approach. Importantly, OWG excels at semantic and affordance-based queries, something which is essential in human-robot interaction applications but is missing from modern vision-language models. We identify two basic failure modes: a) the LVLM confused the target description with another object, e.g. due to same appearance or semantics, and b) the LVLM reasons correctly about the object and where it is roughly located, but chooses a wrong numeric ID to refer to it.

In this section we wish to evaluate the full stack of OWG, incl. grounding, grasp planning and grasp ranking via contact reasoning, in scenarios that emulate open-world grasping challenges. To that end, we conduct experiments in both simulation and in hardware, where in each trial we randomly place 5-15 objects in a tabletop and instruct the robot to grasp an object of interest. We conduct trials in two scenarios, namely: a) isolated, where all objects are scattered across the tabletop, b) cluttered, where objects are tightly packed together leading to occlusions and rich contacts. We highlight that object-related query trials contain distractor objects that share the same category with the target object.

Baselines We compare with two baselines, namely: a) CROG [39], an end-to-end referring grasp synthesis model trained in OCID [40] scenes, and b) SayCan-IM [12], an LLM-based zero-shot planning method that actualizes embodied reasoning via chaining external modules for segmentation, grounding and grasp synthesis, while reasoning with LLM chain-of-thoughts [74]. Our choice of baselines aims at showing the advantages of using an LVLM-based method vs. both implicit end-to-end approaches, as well as modular approaches that rely solely on LLMs to reason, with visual processing coming through external tools. See details in baseline implementations in Appendix B.

Implementation Our robot setup consists of two UR5e arms with Robotiq 2F-140 parallel jaw grippers and an ASUS Xtion depth camera. We conduct 50 trials per scenario in the Gazebo simulator [75], using 30 unique object models. For real robot experiments, we conduct 6 trials per scenario having the initial scenes as similar as possible between baselines. In both SayCan-IM and our method, Mask-RCNN [76] is utilized for 2D instance segmentation while GR-ConvNet [38] pretrained in Jacquard [52] is used as the grasp synthesis module. Our robotic setup is illustrated in Fig. 4, while more details can be found in Appendix B. To investigate generalization performance, all method are evaluated in both scenarios, in two splits: (i) seen, where target objects and queries are present in the method’s training data or in-context prompts, and (ii) unseen, where the instruction refers to objects that do not appear in CROG’s training data or SayCan-IM’s in-context prompts. Averaged success rate per scenario is reported, where a trial is considered successful if the robot grasps the object and places it in a pre-defined container position.

Results We observe that the supervised method CROG struggles when used at unseen data, in both scenarios. In contrary, both SayCan-IM and OWG demonstrate immunity to seen/unseen objects, illustrating the strong zero-shot capabilities of LLM-based approaches, which can naturally generalize the concepts of object categories/attributes/relations from language. SayCan-IM is limited by the external vision models and hence struggles in cluttered scenes, where its detector sometimes fails to perceive the target object, resulting in lower final success rates compared to OWG, especially in the real-world experiments. OWG consistently outperforms both baselines both in simulation and in the real robot, with an $\sim 15\%$ and $\sim 35\%$ improved averaged success rate respectively. In Fig. 5, we illustrate the decomposition of failures across grounding and grasping in our baselines for $25$ Gazebo trials per scenario, where we automatically test for the target object’s grounding results alongside success rate. We observe that OWG consistently reduces the error rates in both grasping and grasping compared to the baselines in all scenarios and test splits. We believe that these results are encouraging for the future of LVLMs in robot grasping.

In out ablations we wish to answer the following questions: a) What is the bottleneck introduced by the segmentation model in the open-ended grounding performance?, b) What are the contributions of all the different visual prompt elements considered in our work?, and c) What is the contribution of the LVLM-based grasp planning and ranking in robot grasping experiments? The grounding ablations for the first two questions are organized in Table 3, while for the latter in Table 4.

Instance segmentation bottleneck We compare the averaged mIoU of our OWG grounder in a subset of our OCID-VLG evaluation data for three different segmentation methods and ground-truth masks. We employ: a) SAM [37], b) the RPN module of the open-vocabulary detector ViLD [28], and c) the RGB-D two-stage instance segmentation method UOIS [77], where we also provide the depth data as part of the input. ViLD-RPN and UOIS both achieve a bit above $70\%$, which is a $\sim 15\%$ delta from ground-truth masks, while SAM offers the best baseline with a $10.8\%$ delta from ground-truth. Implementation details and related visualizations in Appendix C.

Visual prompt components Visual prompt design choices have shown to significantly affect the performance of LVLMs. We ablate all components of our grounding prompt and observe the contribution of each one via its averaged mIoU in the same subset as above (see details in Appendix A.2). The most important prompt component is the reference image, provided alongside the marked image. Due to the high clutter of our test scenes, simply highlighting marks and label IDs in a single frame, as in SoM [26] hinders the recognition capabilities of the LVLM, with a mIoU drop from $86.6\%$ to $23.2\%$. Further decluttering the marked image also helps, with overlaying the numeric IDs, using high-resolution images and highlighting the inside of each region mask being decreasingly important. Surprisingly, also marking bounding boxes leads to a $12\%$ mIoU drop compared to avoiding them, possibly due to occlusions caused by lots of boxes in cluttered areas. Finally, self-consistency and chain-of-thought prompting components that were added also improve LVLM’s grounding performance by $\sim 16$ and $10\%$ respectively, by ensembling multiple responses and enforcing step-by-step reasoning.

Grasp-Related Ablations We quantify the contribution of our grasp planning and ranking stages in the open-world grasping pipeline, by replicating trials as in the previous section and potentially skipping one or both of these stages. As we see in Table 4, the effect of these components is not so apparent in isolated scenes, as objects are not obstructed by surroundings and hence most proposed grasps are feasible. The effect becomes more prominent in the cluttered scenario, where the lack of grasp planning leads to a success rate decrease of $22\%$. This is because without grasp planning the agent attempts to grasp the target immediately, which almost always leads to a collision that makes the grasp fail. Grasp ranking is less essential, as a lot of contact-related information is existent in the grasp quality predictions of our grasp synthesis network. However, it still provides an important boost in final success rate ($8\%$ increase). When skipping both stages, the agent’s performance drops drastically in cluttered scenes, as it is unable to recover from grasp failures, and hence always fails when the first attempted grasp was not successful.