Learning Diverse and Physically Feasible
Dexterous Grasps with Generative Model and
Bilevel Optimization

Dexterous grasping is a long standing problem in the robotics community. In general, the literature can be split into learning-based and analytical methods. Early learning-based grasp planners seek to fit the space of possible grasps for rapid grasp generation (e.g., [1]). More recent papers have shifted to producing grasps directly with complex model architectures, such as generative models [2, 3, 4] and convolutional neural networks [5, 6]. However, most of these works do not account for physical laws and have restricted, or even missing, hardware evaluation. This issue is exacerbated by the difficulty of simulating contact-rich interactions [7, 8]. The quality of grasps produced by learning exclusively with simulation is questionable. Moreover, many papers do not emphasize the ability to learn diverse grasps, thus forgoing a key benefit of dexterous hands over simple grippers. Notably, some publications on learning dexterous grasping originate from computer vision and graphics communities (e.g., [9, 10, 11, 12, 13, 14]). While these papers cover diverse grasp generation on fully dexterous hands, their ultimate objective tends to be achieving photorealistic human grasps in simulation rather than satisfying strict real-world physical constraints.

On the other hand, analytical dexterous grasp planners are often inspired by physical constraints such as force closure, friction cone, robustness to external disturbance wrenches, and object contact (e.g., [15, 16, 17, 18]). Historically, analytical methods are standalone and includes both grasp modality exploration and physical constraint compliance. We point the readers to [19, 20] for a detailed review on these methods. Some limitations of these approaches include dependency on a high-fidelity object model and lack of diversity in the generated grasps [20].

In recent years, some learning-based approaches have adopted a “grasp refinement” step motivated by analytical metrics [4, 18, 21]. Nevertheless, these metrics tend to be formulated as “relaxed” optimizations to keep the problem tractable. Instead of enforcing the constraints directly, the violation of each constraint is cast as a scalar loss and summed together. This results in an unconstrained optimization which is significantly easier to solve. However, there is no guarantee that the optimization output will indeed satisfy the constraints that motivated the loss design.

Due to the geometry of parallel jaw grippers, simple grasp planning can be reduced to computing a 6-DoF gripper pose in space. As such, analytical simple grasp planners may directly reason about the object geometry (e.g., [22]) or rank grasps using grasp quality metrics [19]. Recently, learning-based approaches that predict wrist poses have gained significant traction (e.g., [23, 24, 25]). We refer the readers to [26, 19, 20, 27, 28] for a thorough review. These approaches seldom scale directly to dexterous grasp planning, which is a significantly more complex problem.

While BO theory is established in literature [29], application of BO on motion planning is relatively new. BO has been applied to continuous systems [30], robotic locomotion [31, 32], simple pushing and pivoting [33], collaborative object manipulation [34], and task and motion planning [35]. To our best knowledge, this work is the first application of BO on dexterous robotic manipulation.

For a static dexterous grasp, the dynamic constraints include the wrench closure constraint and the friction cone constraint. The wrench closure constraint requires the sum of the external wrench from all contact points to be zero: $\sum_{i=1}^{3}\bm{f}_{i}=\bm{0}$ and $\sum_{i=1}^{3}\bm{p}_{i}\times\bm{f}_{i}=\bm{0}$. $\bm{f}_{i}\in\mathbb{R}^{3}$ is the unknown contact force applied at position $\bm{p}_{i}\in\mathbb{R}^{3}$ from the dexterous hand to the object.

Given the static friction coefficient $\mu$ and the outward-pointing surface normal $\hat{\bm{n}}_{i}$, the friction cone constraint requires the normal force to be nonnegative and the contact force to be within the friction cone. Using a polyhedral cone approximation ([36]) with an orthogonal basis $\hat{\bm{t}}_{i,j}\perp\hat{\bm{n}}_{i},\forall j\in\{1,2\}$, we arrive at the following approximated friction cone constraints:

A grasp is dynamically feasible if it satisfies both wrench closure and friction cone constraints. Leveraging the polyhedral approximation, dynamic feasibility can be cast as a quadratic program (QP):

The optimization in Equation (2) has objective value $0$ if and only if the grasp is dynamically feasible. Note that we set an arbitrary lower bound ${f}_{min}\in\mathbb{R}^{+}$ on the normal force to avoid the trivial solution of $\bm{f}_{i}=\bm{0}$. The exact value of ${f}_{min}$ is irrelevant because changing ${f}_{min}$ represents scaling the optimal contact forces, which does not affect membership in the friction cone.

The kinematic constraints include reachability constraints and collision constraints. Reachability constraints enforce the kinematic ability of the robot hand to reach the contact point $\bm{p}_{i}$ on the object surface $\partial\bm{O}$, i.e. $\exists\bm{q}:K_{i}(\bm{q})=\bm{p}_{i}\in\partial\bm{O},\;\forall i\in\{1,2,3\}$. Here $K_{i}:\mathbb{R}^{22}\mapsto\mathbb{R}^{3}$ is the forward kinematics function that maps the hand state $\bm{q}$ to the position of the $i$-th fingertip. Assuming only fingertip contacts, collision constraints ensure that no robot link $\bm{L}(\bm{q})$ is in collision with the object $\bm{O}$ except at the fingertip or with the environment $\bm{E}$, i.e. $\bm{L}(\bm{q})\cap\left\{\bm{O}\cup\bm{E}\right\}=\emptyset.$ Both reachability and collision constraints are nonconvex constraints.

We leverage a generative model to sample potential finger placements $\mathcal{P}$ given an arbitrary object observed as a point cloud $\hat{\bm{O}}$. An immediate challenge faced by the learning approach is the lack of large-scale datasets for dexterous robotic hands. While datasets for real human hands do exist, retargeting human grasping configurations to a robot hand with different kinematic structures and joint limits presents numerous challenges. We opt to synthesize a large-scale dataset of physically feasible grasping configurations for the Allegro robot hand used in this paper.

While the grasps in the training dataset are physically feasible by construction, there is no guarantee that the CVAE output, $\mathcal{P}$, is physically feasible. Additionally, $\mathcal{P}$ only specifies the finger placement instead of the full hand configuration. We propose a BO to obtain a physically feasible grasping pose $\bm{q}$ given $\mathcal{P}$. To seed the BO with $\mathcal{P}$, we first obtain an Euclidean projection of $\mathcal{P}$ onto $\partial\bm{O}$, denoted as $\mathcal{P}^{\prime}\triangleq\left\{\bm{p}^{\prime}_{i}\;\middle|\;\bm{p}^{\prime}_{i}=\operatorname*{arg\,min}_{\bm{p}_{o}\in\partial\bm{O}}{\|\bm{p}_{o}-\bm{p}_{i}\|_{2}},\;i\in\{1,2,3\}\right\}.$ Next, we solve an inverse kinematics (IK) problem for a grasp configuration finger placement problem specified by $\mathcal{P}^{\prime}$ up to a numerical tolerance $\epsilon$, i.e. $\textrm{find }\bm{q}^{\prime}:\|K_{i}(\bm{q}^{\prime})-\bm{p}^{\prime}_{i}\|_{2}\leq\epsilon,\;K_{i}(\bm{q}^{\prime})\in\partial\bm{O},\;\forall i\in\{1,2,3\}.$ $K_{i}:\mathbb{R}^{22}\mapsto\mathbb{R}^{3}$ is the forward kinematics function of the $i$th finger. $K_{i}(\bm{q}^{\prime})\in\partial\bm{O}$ is implemented as a constraint on the finger-object signed distance $D(\bm{p},\bm{O})\in\left[d_{min},d_{max}\right]$. The IK solution $\bm{q}^{\prime}$ serves as the initial guess for the BO.

The hardware setup and evaluated objects are shown in Figure 3(c).
Hardware. A 16-DoF Allegro v4.0 right hand was used for hardware grasping experiments. The Allegro hand was mounted on a 7-DoF Franka Emika Panda arm to realize the planned wrist pose. The object point cloud was captured by 4 stationary Intel RealSense D435 depth cameras. The implementation details are available in the supplementary material.
Evaluated Objects. We evaluated our method on 20 rigid household objects resting on the table. We categorize our objects into three sets:
$\bullet\;\,$3 Seen objects. (Leftmost column) mustard bottle, soup can, and sugar box from the training set.
$\bullet$ 4 Familiar objects. (Second-from-left column) Unseen objects with geometries similar to training objects. Includes boxes (webcam box, mask box) and cylinders (chip can, soda can).
$\bullet$ 13 Novel objects. Objects whose geometries are distinct from that of any object in the training set. From left to right, front to back: tetrahedron, massage ball, castle, pill bottle, glasses case, condiment bottle, hairspray bottle, ramen, lego, sandwich box (side), pear, sandwich box (upright), alcohol bottle.

At the start of each trial, the object is placed in a specified pose in front of the robot. After observing the point cloud and computing a grasp, the Franka arm first brings the hand to the desired wrist pose using an ad hoc trajectory planner. The finger joint angles $\bm{\theta}^{*}\in\mathbb{R}^{16}$ from $\bm{q}^{*}$ is then executed on the Allegro hand. Contact forces are provided by squeezing the fingertips according to the planned contact forces: $\bm{\theta}^{*}\leftarrow\bm{\theta}^{*}+k(\nabla_{\bm{\theta}}\bm{p}_{i})^{T}\bm{f}_{i}$. Here $\nabla_{\bm{\theta}}\bm{p}_{i}$ is the Jacobian of the fingertip locations with respect to $\bm{\theta}$, and $k$ is a fixed “stiffness” constant as motivated by impedance control. The Franka then attempts to lift the object to a fixed height approximately $43$cm above the table. A trial is considered successful if the object is lifted and all three fingers remain in contact with the object. 3 repeats are performed for hardcoded policy trials as they are deterministic up to the object point cloud observation. All other trials are repeated 6 times. Our hardware pipeline was implemented with the intent to execute our planned grasp as accurately as possible. Figure 3(d) illustrates the simulation-hardware grasp correspondence.

We compare our method against the following approaches. We excluded “BO only” in our ablation studies as it does not return a result without CVAE in practice. BO seldom returns a solution if seeded with a kinematically infeasible solution (e.g., open hand).
$\bullet$ Hardcoded grasp baseline. Using the set of seen objects, we designed a top-down tripod grasp policy that chooses the wrist pose based on the point cloud and executes a fixed grasp.
$\bullet$ CVAE-only as an ablation study. We solved a collision-free inverse kinematics problem that attempts to match the CVAE predicted fingertip positions $\mathcal{P}$. This mimics a “learning only” approach.
$\bullet$ CVAE-kinematic as an ablation study. We ablated the “lower level” part of the optimization (Equation (5)) away and use $\bm{q}^{\prime}$ directly without bilevel optimization. This ablation considers kinematic constraints but not dynamic constraints.

Our method achieved an overall success rate of 86.7% over 120 grasp trials on 20 objects. This is superior to the hardcoded baseline, which achieved an overall success rate of $53.3\%$. On seen objects, our CVAE-only ablation achieved a $38.9\%$ success rate and our CVAE-kinematic ablation achieved an $88.9\%$ success rate. The results are summarized in Table 1, and the details are available in the supplementary material. A video summary is available at youtu.be/9DTrImbN99I.

Our method can grasp challenging novel objects. This includes objects that are difficult for parallel grippers and suction cups to grasp. We discuss the challenges of some of our test objects in the supplementary material. The hardcoded baseline nearly always succeeds on objects that allow top-down grasps and are similar in size to the seen object used for tuning. However, the baseline fails on objects that either require a different grasp type or have significantly different size.

Our method can generate diverse grasps through sampling different latent $\bm{z}$’s. Figure 4 shows 10 distinct successful grasps performed on the upright sugar box.

The median time to produce a grasp with our method is 14.4 seconds. The median repeats for each component of our method when generating a grasp is two CVAE $\bm{z}$ samples, two IK solves for $\bm{q}^{\prime}$, and two bilevel optimizations for $\bm{q}^{*}$. Detailed timing and repetition results are available in the supplementary material.

Table 2 summarizes the quantitative evaluation of physical constraints.
Kinematic constraints. The maximum finger-object distance constraint violation in a grasp planned by our method is 0.08cm, which is negligible in practice as it is smaller than other error sources such as camera observation error. The CVAE-kinematics ablation, which only considers kinematic constraints, achieved comparable results. The CVAE-only ablation has a maximum violation of 2.032cm, confirming that while CVAE alone can produce qualitatively correct grasp, it cannot enforce kinematic constraints precisely.
Dynamic constraints. To evaluate dynamic constraint satisfaction, we chose “force (torque) ratio” as our metric, i.e. $\frac{\|\sum_{i=1,2,3}\bm{f}_{i}\|_{2}}{\frac{1}{3}\sum_{i=1,2,3}\|\bm{f}_{i}\|_{2}}\times 100\%,$ and $\frac{\|\sum_{i=1,2,3}\bm{p}_{i}\times\bm{f}_{i}\|_{2}}{\frac{1}{3}\sum_{i=1,2,3}\|\bm{p}_{i}\times\bm{f}_{i}\|_{2}}\times 100\%.$ If wrench closure is achieved, both ratios should be zero. All ratios are reported as median, (25th percentile, 75th percentile). We computed these ratios for grasps planned with our method and CVAE-kinematics ablations. These ratios are not computed for CVAE-only because there are often fewer than 3 finger-object contacts. Our method achieved a median of $\leq 0.01\%$ on both ratios, showing that BO is effective in enforcing wrench closure. We note that while the CVAE+kinematics ablation does not explicitly consider external wrench, it still produced many grasps that achieve wrench closure by coincidence. This reflects CVAE’s ability to produce qualitatively correct grasps.

To show that physical feasibility is a necessary condition for a successful grasp, we examined the CVAE+kinematic trials, which may not satisfy dynamic constraints, on “ramen” and “mustard bottle.” We also executed grasp plans that are reported to be infeasible by BO. Table 3 summarizes the results. There is a strong correlation between hardware success and wrench closure. Moreover, all grasps reported to be infeasible by BO failed on hardware. This confirms that grasp refinement derived from rigorous physics-inspired metrics can significantly improve the final grasp plan quality.