IPC-GraspSim: Reducing the Sim2Real Gap for Parallel-Jaw Grasping
with the Incremental Potential Contact Model

Analytic grasp metrics use contact force and torque constraints generated by contact models to measure grasp robustness. As summarized by [39], common metrics describe the ability of a grasp to resist external disturbances (force closure [3, 38]), be at equilibrium between contact forces and external forces [21], or allow the object to resist gravity (wrench resistance [32, 31]); these metrics can be determined using linear or quadratic programs based on the force and torque constraints at each contact.

Grasp contact models measure the ability of grasp contacts between the gripper jaw and the object to resist external forces and torques via the contact area, contact pressure and friction applied at the contact. Models of compliant gripper contacts typically use the quasistatic approximation. [4] summarize analytic physics-based contact models, including soft point contact models [42, 6, 15, 19, 20], which exert forces in the plane tangent to the contact surface and a torsional moment about the contact normal. Frictional limit surface models jointly limit the tangential force and torque about the contact normal [12, 27, 15, 48] and can be extended to model non-planar contacts [44]. Another approach is to approximate contact area with geometric primitives, such as planes, spheres, or cylinders, that have analytic solutions [14, 5]. Similarly, [8] and [45] discretize the contact area into a triangular mesh and formulate per-triangle or per-contact limit surface constraints based on a static friction assumption. In contrast to these quasistatic grasp contact models and metrics, we use a simulator that incorporates both grasp dynamics and frictional area contacts.

Grasp simulators, unlike grasp contact models, can incorporate dynamics which are crucial to grasping: they can predict which jaw makes contact first, and how that affects post-grasp displacement as the gripper and object settle into a stable position [47]. In this paper, we focus on dynamic simulators that model deformable materials [34, 25, 7, 28], as it is important to model frictional contact dynamics simultaneously with gripper compliance: deformation affects surface area, which in turn affects object dynamics.

Isaac Gym is a GPU-enabled simulator favored by the robotics community for its speed and python-based simulator with a straightforward robotics interface. NVIDIA FleX engine integration has recently made deformables available for robotics environments, as evidenced by recent papers that model the SynTouch BioTac [36] and grasping of deformable objects with rigid grippers [16]. However, many simulators, including Isaac Gym, require manual parameter tuning to avoid interpenetration artifacts and may be sensitive to friction and closing force parameters [40].

In simulation, grasp success may be defined as (1) the object can be lifted above the surface [30, 16], (2) the object can be lifted and withstand perturbations via shaking motions [9, 10], (3) the object is removed from its environment after grasp and move actions [46], or (4) the object’s post-grasp pose matches a desired pose [1]. In this paper, we use the first definition of grasp success.

Simulating deformable materials is challenging due to their many degrees of freedom and friction between deformable and rigid objects. However, due to the high computational load in more accurate representations such as Finite Element Methods (FEM), most work relies on simplified models as provided by mass-spring systems and position-based dynamics, which are computationally efficient but do not have a rigorous mathematical model for contact and friction [2]. With FEM (e.g., PyBullet’s FEM option and graphics implementations), Neo-Hookean materials are often used to simulate hyperelastic materials where the stress-strain relationships are highly nonlinear, such as silicone rubber [33]. Isaac Gym models softbody dynamics using corotational linear-elastic constitutive models, and leverages a custom GPU-based solver [29].

A common model for applying loading conditions onto the deformable bodies is the penalty-based contact, which calculates contact force as proportional to the mesh penetration depth and penalty value $p$, assuming nonzero interpenetration for all contacting bodies. High values of $p$ can decrease the penetration depths, but also make the problem ill-conditioned [22]. Furthermore, any contact violation can cause catastrophic failures [26]. A common way to handle interpenetration is to add a boundary layer on the deformable to “thicken” the mesh for collisions, but the layer depth must be manually tuned for each scenario [28, 40].

Incremental Potential Contact (IPC) [26] is a computer graphics model and algorithm presented in 2020 based on a custom nonlinear Newton-based solver that solves elastodynamics contact problems with the smoothed barrier method. It guarantees a collision-free state at every timestep to provide robust solutions for all choices of material parameters, timestep sizes, impact velocities, deformation severities, and enforced boundary conditions. IPC provides inversion-free and intersection-free solutions, so that objects do not penetrate and remain stuck to one another in future timesteps. IPC accounts for static friction with a smoothed approximation to eliminate an explicit Coulomb constraint, which leads to the assumption that static and dynamic friction coefficients are equal between each material combination. The authors justify this decision by implementing an approximation to static friction that can successfully model stiction.

We leverage IPC as the physics engine of a grasping simulator that avoids the interpenetration issues described in Section II-C and more accurately models grasp outcomes observed on a physical system than existing quasistatic or dynamic simulator approaches.

We make the following assumptions:

1. 1.

   A singulated rigid object of known geometry and uniform density rests on a planar surface.

2. 2.

   The gripper has known geometry and two parallel jaws.

3. 3.

   The silicone rubber material on the gripper jaws can be simulated with the Neo-Hookean material model.

The true state x of the grasping environment includes geometry (including deformations), pose, mass, and frictional properties of the object and gripper jaws. Each grasp u is represented by the parallel gripper jaw center and the grasp axis orientation $\varphi\in\mathbb{R}^{3}$. The modifiable gripper jaw parameters include the deformable material elasticity and the friction coefficient between the gripper jaw pads and the grasping object. Success $S(\mathbf{x},\mathbf{u})$ is measured with a binary reward function $S$, where $S=1$ if the grasp successfully lifts the object and $S=0$ otherwise.

To model uncertainty and imprecision in robot control, we define grasp robustness $R(\mathbf{x},\mathbf{u})=\mathbb{E}[S(\mathbf{x},\mathbf{u^{\prime}})]$, where the perturbed grasp pose $\mathbf{u^{\prime}}=\mathbf{u}+\varepsilon$ is offset by $\varepsilon$ sampled from a normally-distributed pose random variable. To calculate robustness in our experiments, we use Monte-Carlo sampling to estimate $R$ with the sample mean of $N$ trials each with a perturbed gripper pose: $R=\frac{1}{N}\sum_{i=1}^{N}S(\mathbf{x},\mathbf{u^{\prime}}_{i})$. We emphasize the importance of using sampling with grasp trials; although the robot may intend the same grasp on a given object, not all trials will behave equally due to randomness introduced by pose estimation, robot control, and frictional forces.

To quantify how closely the predicted grasp robustness from the simulator matches the robustness measured on the physical system, we measure the F1 score, a harmonic mean of average precision (AP) and average recall (AR) . F1 score is a common measure of binary classification performance for imbalanced datasets in computer vision [41].

IPC-GraspSim models grasping with a velocity-controlled compliant parallel-jaw gripper as follows:

In our experiments, we use the physical dataset from [8]: a set of 2,000 grasps across a set of 16 3D-printed adversarial objects on a physical ABB YuMi robot with a compliant parallel-jaw gripper [13].

We evaluate grasp outcomes on 3 objects with 1) a similar number of successful and unsuccessful grasps and 2) varied geometries. Then, we identify the set of parameters $(\hat{E},\hat{\mu})$ with the highest F1 score in the training set, and use these parameters to simulate grasps on the remaining 13 objects in Section VI.

We vary the Young’s modulus of the jaw pad from $10^{7}$ Pa (rubbery material) to $10^{11}$ Pa (metal material). We linearly vary the friction coefficient between the pad and object from 0.3 to 0.6. Note that the exact friction between the jaw pad and object is unknown due to unknown material properties and the unmodeled surface variations on the jaw pad, as shown in Dex-Net’s silicone rubber fingers in Figure LABEL:fig:comp_gripper_wild. The millistructure design results in increased friction between the jaw pad and the object [13]. We aim to find both the Young’s modulus and single friction coefficient that closely models such a structure.

For both gripper jaw models, lower Young’s modulus values $(E,\mu)=(10^{8},0.4)$ have higher AP, AR, and F1 score as compared to higher Young’s modulus values $(10^{11},0.4)$. For rounded jaws, more compliant jaws increase AP by 13%, AR by 2%, and F1 by 7%; for rectangular jaws, they increase AP by 14%, AR by 11%, and F1 by 12%. Increase in $\mu$ appears to correlate with an increase in AR, as the additional frictional forces allow for a greater number of successful grasps; however, AP drops as the number of false positives increases. We hypothesize the increase in AR occurs due to the increased contact area in compliant grippers, allowing the grippers to deform around the contact point and distribute contact forces across the object’s surface. However, when $\mu$ is too high, objects no longer can slip out of the gripper jaws and the number of false positive predictions increases.

The compliant rounded jaws result in a 17% increase in AP and 8% decrease in AR, leading to an overall increase of 5% in F1 score, compared to the compliant rectangular jaws when using the highest-performing parameters for each geometry $(E=10^{8},\mu=0.4)$. We hypothesize that the slight precision-recall tradeoff occurs as the smaller cross-sectional area of the rounded gripper jaw causes grasp robustness predictions to be more conservative. However, the F1 score ultimately increases, as the extra contact area due to the rectangular jaws had allowed grasps to incorrectly succeed.

Based on the parameter sensitivity analysis, we use the rounded gripper jaw model with $(\hat{E},\hat{\mu})=(10^{8},0.4)$ to model the compliant gripper jaws in dynamic simulation. We note that this optimal set of parameters closely matches the physically estimated parameters ($10^{7}-10^{8}$ Pa, 0.46).

To evaluate the precision and recall of IPC-GraspSim, we use the 13 held-out objects from physical dataset from [8]. The objects in this dataset were chosen because they span a wide range of object geometries and grasp success rates. All objects are 3D printed such that simulated and physical attempted grasps are identical. For benchmark experiments, we run 5 trials for each grasp while perturbing the grasp pose in each trial for dynamics randomization. The trials are then averaged to calculate the grasp robustness predicted by each grasp model.

In addition to IPC-GraspSim, we benchmark 4 other models; three analytic models used in the literature (soft point contact model, REACH [8], 6DFC [45]), and a dynamic softbody simulation implemented in NVIDIA Isaac Gym with Flex backend [28]. We measure the performance of Isaac Gym and IPC-GraspSim by simulating grasps with the rounded gripper jaws and material properties $(E=10^{8},\mu=0.4)$ determined in Section V. Soft Point, REACH, 6DFC, and IPC-GraspSim are run on an Ubuntu 18.04 on a single core of a Dual 20-Core Intel Xeon E5-2698 v4 2.2 GHz, and Isaac Gym on an Ubuntu 20.04 machine with an Intel i7-6850K processor and NVIDIA Titan X GPU.

The results in Table IV suggest IPC-GraspSim can predict grasp robustness with F1 score of 0.85, an improvement in F1 score by 0.03 to 0.20 over analytic baselines and 0.09 over Isaac Gym, at a cost of 8000x more compute time. We observe an average increase of 0.17 in average recall in IPC-GraspSim compared to grasp models that approximate compliance (Soft Point, REACH, 6DFC) and a 0.05 increase in average recall compared to Isaac Gym.

We hypothesize that the increase in recall as compared to the analytic models is due to the ability of IPC-GraspSim to model dynamics and compliance simultaneously. Modeling dynamics results in correct predictions for grasps that rotate into alignment with the gripper, a common failure mode for quasistatic models [8, 45]. In addition, modeling compliance keeps the object closer to the gripper as it moves by 1) conforming to the shape of the object and 2) applying normal and frictional forces that stabilize the object. The combination of dynamics and compliance models the changing contact area and contact pressure over the course of the grasping motion.

We also note that despite modeling dynamics, Isaac Gym has F1 score 0.06 lower than 6DFC, a quasistatic contact model; this result suggests that if the underlying simulator for modeling compliant gripper grasps is not sufficiently robust (i.e., simulator returns sudden large forces due to incorrect collision force handling), it may perform worse than a robust model that ignores dynamics. Although IPC-GraspSim only increases F1 score by 0.03 compared to 6DFC [45], the simulator has significant room for improvement: most of its failure modes are due to unmodeled physical effects (e.g., gripper jaw cantilever effects, further detailed in Section VI-D), which we hope to address in future work.

In comparison to Isaac Gym, IPC-GraspSim increases average precision by 0.11, average recall by 0.05, and F1 score by 0.09. The performance of IPC-GraspSim highlights the importance of intersection-free and robust modeling of soft contacts for compliant gripper modeling. First, poorly-resolved contacts may cause grasped objects to move unpredictably (e.g., float or swing into air, as shown in Figure 3C-D for Isaac Gym). Second, incorrect contacts introduce force imbalance between the two compliant jaws and cause the target object to wobble around or become lodged in the gripper, resulting in unrealistic object behavior.

Although dynamic simulators (Isaac Gym, IPC-GraspSim) run at least three orders of magnitudes slower than analytic grasp models, these evaluations are intended to run offline for building large-scale datasets. Therefore, these simulations can be performed in parallel over extended periods of time.

Table III shows results for each grasp attempt of the “vase” (object 21 in the grasp dataset collected by [8], shown in Figure 1). A full breakdown of grasp attempts and predictions for all objects can be found in the supplement. For this object, we observe a high correlation between IPC-GraspSim predictions and physical experiments. IPC-GraspSim achieves an F1 score of 0.85 as compared to 0.65-0.82 over analytic methods and 0.68 for Isaac Gym.

We observe a false positive prediction for grasp 7, where gripper jaws bend and cantilever away from the object under the stress from the closing forces. The contact forces then push the vase away from and out of the gripper. However, in simulation, the gripper backings are perfectly rigid, and and do not account this bending phenomenon.

We also observe a false negative prediction for grasp 1, where the right jaw contacts the vase first and rotates the vase out of the gripper. This phenomenon is also observed in Figure 3A. This missed prediction suggests that precise gripper geometry and gripper closing speed have an impact on grasp predictions. We additionally hypothesize that the dynamics between the ground plane and the object are not well-modeled in simulation.

It is critical for grasp simulators to account for uncertainty and imprecision in robot control, as there are two main sources of uncertainties when grasping in physical environments: robot position and object pose estimation. For some grasps that are very sensitive to contact area, as shown in Figure 5, perturbations have higher chances of affecting grasp success rates.

The calculated post-grasp object pose and grasp success in IPC-GraspSim is deterministic for each input grasp. To introduce randomness for Monte-Carlo sampling in this simulator, we inject grasp pose perturbation matrix $\varepsilon$ with normally distributed translation and rotation error. Although Isaac Gym (wiht FleX backend) is not deterministic, we inject the same perturbation effects in the simulator to mirror our changes in IPC-GraspSim.

Due to the limitations in the IPC simulator, IPC-GraspSim uses $\Psi$, the deformation or strain energy in a given mesh, to approximate closing force applied by the compliant gripper jaws on the target object. $\Psi_{th}$ is a user-defined threshold to limit the amount of deformation in each pad ($\Psi_{1}$, $\Psi_{2}$), and corresponds to the $\Psi$ that best matches our maximum gripper closing force for the specified gripper. We attempt multiple grasps on the ‘cube‘ object with various values of $\Psi_{th}$ (shown in Figure 5). We find that $\Psi_{th}=5\times 10^{4}$ properly models the gripper pad thinning under pressure while avoiding unrealistic deformation.

As mentioned in the main paper, the computationally efficient penalty-based contact methods are prone to large interpenetration effects, as shown in Figure 6A: the gripper pad intersects with the target object. These effects become more extreme when simulating softbody contacts with large Young’s Modulus values $E$, or contacts between materials with large differences in $E$.

In order to avoid large interpenetration errors, other works have modified penalty-based contact methods to calculate contact forces based on a “thickened” version of the mesh, such that nonzero contact forces can be applied without observing any mesh penetration. However, this method may introduce “force field” effects if the layer depth is too large: an example is shown in Figure 6B, where the gripper pad is able to push and grab onto the target object although no contact is visually observed.

Neither of these behaviors are not acceptable in grasping contexts, as the error in Figure 6A will lead to false sticking behavior (false positive), and Figure 6B will lead to incorrect dynamics from inaccurate collision handling.

This dataset contains 21 adversarial objects where analytic grasp methods perform poorly due to large post-grasp displacements. Even for seemingly simple objects, such as object 5 (“cube”) and object 10 in Figure 7, small perturbations as mentioned in Section VIII-A introduce discrepancies between pre-grasp and post-grasp contact area. However, a subset of 16 objects were chosen for our experiments due to 1) skewed success/unsuccessful grasp ratio and 2) timing constraints, as IPC-GraspSim was not optimized for speed. Out of 16, 3 objects were chosen for their 1) similar number of successful and unsuccessful grasps and 2) varied geometries.

To study this effect in IPC-GraspSim, we choose the rigid cube, a simple object, and vary its resolution by subdividing the surface mesh before creating the volume mesh inputted to IPC. Then, we run grasping simulation on the varied meshes on a successful grasp. The runtime results are shown in Table V. Runtime increases almost linearly with the number of subdivisions in the cube, which is linearly proportional to the number of contacts made between the mesh surfaces. We hypothesize that the runtime is not heavily affected in the simpler models since the gripper pads are more complex than the cube themselves, and serve as the bottleneck to the code complexity.