GRaCE: Balancing Multiple Criteria to Achieve
Stable, Collision-Free, and Functional Grasps

Grasping an object is typically influenced by the intended goal, which directly impacts the choice of grasp. For instance, we naturally grasp scissors by the handle for cutting, but by the blade when passing them safely to someone else. However, the goal isn’t the sole or primary factor in determining a grasp; if the blade is obstructed or inaccessible, as illustrated in Fig. 1, we would opt to grasp the scissors by the handle, even if the intention was to hand it over. This scenario highlights that (i) multiple criteria influence the selection of a grasp and (ii) there exists a hierarchy of priorities among these criteria. The necessity for the grasp to be stable, accessible, and to avoid collisions with surrounding objects (like a mug) takes precedence over the functional goal of the grasp.

In this work, we are motivated by the problem of generating grasps that satisfy multiple criteria of differing importance. We apply hierarchical rule-based logic to robot grasping [1] and introduce a grasp utility function that is rank-preserving [2], i.e., it assigns larger utility values to grasps that satisfy higher ranked constraints. For example, robots are bound by their kinematic and dynamic constraints, which limits whether a proposed grasp can be performed, and environmental constraints (e.g., grasps should not collide with other objects). A stable grasp that satisfies these constraints should have larger utility than one that sacrifices these criteria for a functionally appropriate (but non-executable) grasp.

We take a probabilistic approach and optimize the expected utility of a grasp, where the probability of a grasp satisfying a specific criteria is given by a classifier. Additional classifiers (whether data-driven or hand-crafted) can be added (or removed) depending on the precise requirements of the application. This modular approach — which we call Gasp Ranking and Criteria Evaluation (GRaCE) — enables a robot to trade-off multiple conflicting criteria in complex contexts where not all desired objectives can be satisfied. GRaCE’s probabilistic nature incorporates uncertainty in a principled manner. This is crucial in real-world robotic scenarios, which often involve partial observability and noise. By utilizing the likelihood of a grasp meeting various criteria, GRaCE moves beyond binary satisfaction assessments and reduces the risk of misclassification inherent in the true/false assignments used in prior work (e.g., [3]).

Although the utility function enables scoring of grasps, it is a complicated non-convex function to optimize, especially when the classifiers are themselves complex (e.g., a deep neural network). Inspired by progress in gradient-free methods (e.g., [4]), we propose GRaCE-OPT — a hybrid optimization method that combines gradient-based and gradient-free optimization. Specifically, GRaCE-OPT applies gradient-free optimization, an Evolutionary Strategy (ES) [5], to conduct a more “diverse” exploration over the landscape and prevent the optimization process from getting stuck at local optima. However, on its own, this gradient-free method can be slow to converge. As such, we use gradient-based optimization on a surrogate function, the lower-bound of the utility, to improve convergence speed. Experiments in complex environments show that GRaCE-OPT requires significantly fewer samples and less time to achieve comparable (or better) performance to a filtering method used in prior works [6, 7]. Our evaluations involved two simulated grasping scenarios — shelf and diner (Fig. 2) — in IsaacGym [8] and two real-world scenarios; these test scenarios are designed to be challenging (cluttered with small optimal grasping regions) and where the probability of satisfying multiple criteria may be traded-off.

To summarize, this paper contributes GRaCE which comprises a utility function that assigns higher values following user-specified hierarchical rules and an optimization method that uses both gradient-free and gradient-based optimization of the expected utility. Code and implementation details can be found online at https://github.com/clear-nus/GRaCE.

GRaCE is a probabilistic framework for optimizing 6-DoF grasps. It builds upon related work on 6-DoF grasp candidate generation and prior work on the optimization of multiple criteria specified via rule/constraint hierarchies. We briefly review these topics in this section.

An alternative approach to optimize grasps directly. Early work on multi-fingered end-effector grasping [11] demonstrated that a scoring function for grasp quality (a pre-trained classifier) can be used to optimize grasps toward high quality regions. More recent work have applied optimization together with sample-and-filter methods, e.g., GraspNet [9] optimizes/refines grasp samples using the quality evaluator. These methods focus on a single quality criterion, where else our work addresses the problem of trading-off multiple conflicting criteria.

GRaCE can also be seen as a contrasting approach to “end-to-end” data-driven 6-DoF grasping [12, 13] where the sampler is trained to generate grasps that satisfy multiple criteria. However, these methods require retraining when a new criterion is added/removed, which is computationally expensive. GRaCE enables the inclusion and removal of grasp criterion “on-the-fly”, which we believe is more practical for many real-world applications. This aspect is similar to very recent work [10] that refines grasps using gradient flows, but GRaCE enables the ranking of multiple criteria and we propose a hybrid optimization technique.

Current methods for achieving task functional grasping are largely based on a sample-and-filter methodology with two-stage filtering. To elaborate, grasps are first sampled and non-stable ones are filtered out. Then, the remaining grasps are filtered based on appropriate affordances by segmenting objects [18, 19, 20]. Deep learning methods are often used provide a grasp score given a grasp and a specific target task [15]. Very recent methods employ Multimodal Large Language Models (LLMs) to reason about functional grasps [17, 21, 22, 23]. However, robustness of these models remains a challenge and research on LLMs for task-oriented grasps is ongoing.

The modularity of the GRaCE framework allows the use of many existing differentiable methods for assessing grasp functionality. In our work, we use TaskGrasp [15] as a criterion evaluator due to code availability and its good performance on various objects.

In this section, we present our approach for trading-off criteria for grasp generation. A grasp $\mathbf{g}$ is typically defined as a set of contact points with an object which restricts movement when external forces are applied [26]. For simplicity, we will refer to end-effector poses as grasps (or grasp candidates) and denote them as $\mathbf{g}$, even if they do not satisfy the definition above (e.g., the pose does not make contact with the object). We first discuss how grasp criteria can be ranked, followed by our a utility function, and, finally, formulate an optimization based grasp generation method.

Let $p_{j}^{(i)}:=P(c_{j}^{(i)}(\mathbf{g})|\mathbf{o})$ be the probability that criterion $c^{(i)}_{j}(\mathbf{g})$ is satisfied under observed context $\mathbf{o}$. For notational simplicity, we will drop the explicit dependence of $p_{j}^{(i)}$, $c_{j}^{(i)}$ and $\phi^{(i)}$ on $\mathbf{g}$ and $\mathbf{o}$. We assume that criteria are conditionally independent given the grasp and context. As such, the probability that a rule $\phi^{(i)}$ is satisfied is given by $\prod_{j=1}^{M_{i}}p_{j}^{(i)}$. Table I shows a list of priorities, rules, and their associated probabilities.

Table II summarizes grasp ranks for the rule hierarchy and our utility is defined as the negative expected rank,

This simplified form can be obtained by observing that $\text{eval}(\phi^{(i)})$ is a Bernoulli variable with probability $\prod_{j=1}^{M_{i}}p_{j}^{(i)}$,

In this section, we introduce GRaCE-OPT, a hybrid optimization technique that leverages both gradient-free and gradient-based methods to optimize Equation (2). As an initial step, we considered optimizing a lower-bound of Eq. 2 using Jensen’s inequality:

where $C$ is a constant independent of $\mathbf{g}$. Empirically, we find $L(\mathbf{g})$ to be easier to optimize and numerically stable, but inspection of its form shows that it is no longer rank preserving since the utilities are factored out.

As such, we only use this gradient-based optimization as an inner-loop within a gradient-free ES setup [5], shown in Algorithm 1 below. We assume that we have access to a grasp sampler $q_{0}$ from which we can sample initial grasps $\mathbf{G}_{0}$ from (line 1). In practice, $q_{0}$ can be any grasp candidate sampler, e.g., GraspNet-VAE [9] or a heuristic sampler such as GPD [6]. We then optimize these grasps over $T$ outer gradient-free iterations (lines 2-10). In detail: new batches of grasps are sampled using a multivariate Gaussian distribution with mean $\mathbf{g}_{t}$ and covariance matrix $\Sigma$. The covariance $\Sigma$ is manually selected in our experiments but can also be adaptive [29]. Lines 4 to 6 optimizes the lower bound $L(\mathbf{g})$. Line 8 and 9 assesses grasps using $U(\mathbf{g})$ and selects the top $R$ grasps. In preliminary experiments and ablations (Sec. VIII), we found GRaCE-OPT to be superior to using either gradient-based or gradient-free methods alone.

In this section, we describe different grasp criteria used in our experiments. We assume a setup where a human user is asking the robot to perform a task, e.g., to “handover the scissors”. The robot has access to the natural language utterance from the human as well as observations of the environment (a point cloud).

As previously mentioned, we assume a probabilistic setup where the probability of criteria satisfaction is given by a classifier $P(c^{(i)}_{j}(\mathbf{g})|\mathbf{o})$. We used four different classifiers that capture different quality aspects of a grasp: stability, executability, collision-free, and functional. We discuss these classifiers at a high-level and leave implementation details to the online code repository (made public if accepted).

We calculate the manipulability score for a given grasp:

where $\mathbf{J(\bm{\theta})}$ is the Jacobian matrix and $\bm{\theta}$ is the Inverse Kinematics (IK) solution. Then, we define the predicted grasp pose, $\tilde{\mathbf{g}}$, using the IK solution:

Finally, we combine this two quantities to yield,

where $\sigma(z)=\frac{1}{1+\exp^{-z}}$ is the logistic sigmoid, $C_{m}$ and $C_{w}$ are scaling coefficients, $\omega_{\text{th}}$ is a lowest manipulability threshold that allows safe grasp execution, $d(\cdot,\cdot)$ is a distance function between predicted grasp pose from the IK solution and current pose calculated in SE(3) [32] and $d_{\epsilon}$ is a IK tolerance.

The goal of our experiments is to establish if using the GRaCE framework (i) yields suitable grasps in a complex environment, and (ii) trade-off multiple criteria. Moreover, GRaCE-OPT is more computationally expensive compared to a simple sample-and-filter approach (principally due to gradient computation). Is this added cost justified? Moreover, are the multiple criteria necessary for finding successful grasps and if so, can they be traded-off effectively? Our experiments are aimed at answering these questions. To simplify exposition, we will refer to the process of using GRaCE-OPT to optimize the expected utility in (2) as GRaCE.

Thus far, we have discussed GRaCE in simulation settings, but does GRaCE’s performance carry over to the real world? We conducted real-world tests comparing GRaCE against the filter baseline, similar to the simulation setup.

In this section, we report on additional ablation experiments designed to evaluate changes in priorities affect outcomes, and the effect of GRaCE-OPT’s hyperparameters (specifically, the number of steps $T$ and lower-bound update steps $K$). In the following experiments, we re-ran our simulation experiment in the Shelf environment with the Stability (S), Execution (E), and Collision (C) criteria.

In this study, we introduced GRaCE, a probabilistic hierarchical rank-based modular framework designed for optimizing robotic grasps based on multiple, often conflicting, criteria. We formulated these criteria as a utility function based on the expected ranks of grasp; this takes into account the uncertainty inherent in many real-world grasping scenarios. In addition, we presented GRaCE-OPT as a hybrid optimization technique to optimize grasps using both gradient-based and gradient-free methods. Our experimental evaluations show GRACE’s efficacy in generating high-quality grasps in complex, cluttered environments both in simulation and real-world experiments.

This research is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-016).