Benchmarking the Sim-to-Real Gap in Cloth Manipulation

There exists a broad variety of deformable object simulators. One of their main differences lies on the dynamics model used, ranging from particle-based systems such as the mass-spring (MuJoCo [13]) or Position Based Dynamics (PBD) (Flex [14]), to constitutive models such as the Finite Element Method (FEM) (Bullet [15], SOFA [16]). Although simulators such as Arcsim have been fine-tuned to match the dynamics of fabric materials [17], the reality gap when performing manipulation tasks has not been evaluated.

As a result of the benefits of learning controllers in simulation, recent work has focused on measuring the simulators’ accuracy against real-world data. Collins et al. [9] benchmarked the accuracy of different simulation engines in a rigid-object manipulation task. More recently, Acosta et al. [18] measured the error of simulated rigid-body contact after optimising the parameters of different simulators. However, no prior work has evaluated the reality gap in dynamic deformable object manipulation.

The problem of benchmarking manipulation tasks can be viewed from different perspectives: 1) designing datasets [2] and tasks [19] for benchmarking robotic systems, 2) measuring the performance of multiple algorithms on a task, 3) evaluating the disparity between simulation and real task performance for a given algorithm, and 4) measuring the reality gap between simulation and a real-world dataset.

Most works in deformable object manipulation have focused either on 2) evaluating multiple algorithms in a simulation engine [20, 21], or 3) evaluating the gap when transferring a skill to the real world [4, 22]. However, these works do not quantify the reality gap of the simulations used to train the learning algorithms, which can result in poor performance when performing sim-to-real transfer in a zero-shot manner.

More recently, Lim et al. [23] proposed an approach to learn controllers from real data and simulators fine-tuned with real data for planar cable manipulation, evaluating the reality gap in terms of the cable trajectory. Similarly, Sundaresan et al. [24] fine-tuned a differentiable simulator with data from the real-world, evaluating the reality gap in quasi-static tasks. To the best of our knowledge, our work is the first to study the reality gap in a dynamic cloth manipulation task against a real-world benchmark dataset. In this work we measure the performance of four simulation engines widely used for deformable object manipulation: MuJoCo [6, 7, 11], Bullet [8, 25, 26], Flex [4, 12, 20], and SOFA [10, 27]. Our benchmark is agnostic to the simulator and can be easily applied to forthcoming simulators.

We propose a fabric placement manipulation task performed by a bi-manual system that involves two pre-defined trajectories (see Fig. 2). The first trajectory consists of a dynamic motion of the fabric. The second trajectory brings the garment in contact with a rigid surface and then drags it through the planar surface. The objective of the two trajectories is to evaluate two different dynamics: 1) the dynamics of the fabric without contact, and 2) the dynamics of the contact between the garment and a rigid object.

The goal of the fabric placement task is to end in a flattened configuration starting from a position free of contact. In order to focus on the accuracy of the simulation, the task assumes a successful grasp state. Thus, the fabric starts in a grasped position, where two corners of a rectangular piece of cloth are grasped by a manipulator using a pinch grasp [28]. We decide to use a pinch grasp as this does not require any additional set-up such as those required for interfacing and simulating, e.g., touch-based sensors [29].

In order to place the cloth in a flattened configuration, dynamic motions can control the fabric outside of the working space of the manipulators while efficiently placing the cloth flat in a single attempt. Thus, we design a fling motion [4] and define it with a quintic polynomial

$$x(t)=a_{0}+a_{1}t+a_{2}t^{2}+a_{3}t^{3}+a_{4}t^{4}+a_{5}t^{5},$$

(1)

which is detailed in Sec. III-B. When performing a highly dynamic motion the fabric suffers an abrupt deformation. This is challenging to simulate due to the inertia forces generated by high accelerations and high number of dof of the garment. Therefore, it is a great candidate trajectory for evaluating the reality gap. In addition, to evaluate the capability of different physics engines to simulate frictional and inertia forces, we design a quasi-static motion which consists in entering in contact with the rigid surface by slowly lowering and dragging the fabric.

Our dataset is collected using three different cloths from the public household dataset [2]: a towel rag, a linen rag, and a chequered rag. The garments have a size of 50 $\times$ 70 cm, each with different weight and elasticity, providing a variety that helps assess the ability of the engines to simulate different fabric materials and dynamics. We decided to use these cloths from the dataset as they can be easily lifted by two robotic manipulators and placed on a flat surface. In addition, the fabrics have a rectangular shape rather than square, which results in larger deformations of the non-manipulated corners of the cloth when a high-velocity is applied. As shown in Sec. IV, the towel and the chequered rags have a similar final configuration after the fling motion. However, the linen rag, which is more brittle, is partially folded. By contrast, all fabrics exhibit a similar final configuration after the quasi-static motion.

We use two Franka Emika Panda robots to perform the quintic trajectories. The dynamic trajectory performs a motion on the $YZ$ axes and the roll angle $\phi$, keeping the other axes fixed throughout the trajectory (see Fig. 3a). By contrast, the quasi-static trajectory performs a motion only on the $YZ$ axes (see Fig. 3b). Each trajectory is computed using multiple via-points, where the number of via-points is $n_{Y}=4$, $n_{Z}=3$ and $n_{\phi}=3$ for each the dynamic trajectory, and $n_{Y}=2$, $n_{Z}=2$ for the quasi-static trajectory. For each via-point we define a quintic polynomial, where the coefficients of the polynomial are computed following [30]. The starting and final velocity and acceleration values, as well as the time of the trajectory for each via-point, are defined empirically. The position and velocity trajectories for each axis are shown in Fig. 3. During the trajectory, both robots have the $X$–axis fixed at $51$ cm from their origin. Since one of the manipulators is rotated $180$ degrees with respect to the other, its roll angle is inverted.

In the dynamic motion (Fig. 3a), we distinguish between the phase where the cloth undergoes its natural dynamics and when it makes contact with the surface. The cloth dynamics phase, concluding approximately 3 seconds after starting the trajectory, is used in the benchmarking. Although the trajectory remains consistent across all trials and materials, the cloth contacts the table at slightly different time steps due to inherent randomness in cloth behaviour and material differences¹¹1These values can be found in our open-source code.. Consequently, we manually refine the change of phase time step for each case. By contrast, in the quasi-static trajectory we evaluate both the time instant where the fabric enters into contact as well as the entire contact phase.

The point clouds of the dataset are captured using a Microsoft Azure Kinect RGB-D sensor. The RGB-D images have a dimensionality of $1280\times 720$, and are captured at a frame rate of 30 fps. To compare how well the garments resemble reality in a simulator, we propose to compare the dense point cloud $\mathcal{P}$ obtained by the sensor in the real setup with the meshes $\mathcal{V}$ of the garment provided by the simulator. This enables to quantitatively compare the reality gap, as we can measure the distance between the simulated and real fabric points, rather than performing a qualitative comparison by e.g. comparing their deformation using RGB images.

To obtain the point clouds $\mathcal{P}$ we use the real-world RGB-D images, as well as the position of the camera w.r.t. the manipulators and the intrinsic and extrinsic camera parameters. First, we segment the RGB images with MiVOS [31], which allows to interactively refine the segmentation on individual frames and obtain temporally coherent results. This enables filtering out points that are not part of the garment. Since the positions of the robots and the boundary positions of the garment are known, we use these positions to filter the points. Then, we discard points further away from their neighbours compared to the average. This is performed by applying an statistical outlier removal. Finally, we need to account for the possibly different coordinate systems used across simulators. To achieve this, we define the appropriate coordinate transformation matrices and apply them to convert the simulated meshes into the observation space.

To benchmark the sim-to-real gap in the cloth manipulation tasks we design a framework that is agnostic to the simulation engine and share it open-source²²2https://sites.google.com/view/cloth-sim2real-benchmark. In order to benchmark a simulation engine using our dataset, the simulator needs to have the following capabilities:

• •

  simulation of both rigid and deformable objects,

• •

  control over the cloth points,

• •

  information about the position of the mesh points of the garment,

• •

  adjustable frequency of the simulation engine.

The simulation scene is comprised of the same elements as the real-world dataset: a rigid-object surface, the fabric to manipulate, and two manipulators. Given the limitations present in some simulators (see Sec. IV-A), we consider that a robot is not available in the simulator and assume that only a pinch grasp is available using a dummy manipulator.

The simulated manipulators must take as input the desired target Cartesian coordinate position. The trajectories are given in Cartesian coordinates and calculated according to the specific simulator $\Delta t$. Thus, the trajectories are agnostic to the simulator frequency. To accurately follow the trajectory of the real-world dataset, the simulator must not modify the dataset trajectories or repeat the same action in case that frame-skips are used, as often done in MuJoCo or Bullet.

Given the variety of dynamic models used to approximate the behaviour of cloths in simulation engines, there is no restriction on the cloth parameters. In addition, our benchmark can be used to fine-tune simulator parameters such as the damping coefficient or stiffness that better approximate the dynamics of the garments.

The objectives of our metrics are to: 1) qualitatively measure the reality gap, 2) evaluate the stability of the simulated cloth, and 3) assess the capability of using the simulated scenes in real-time control (hardware-in-the-loop).

There are multiple candidate metrics for measuring the distance between two point clouds, such as the Chamfer Distance (CD), the Hausdorff Distance (HD), or the Earth-mover distance. We select the CD and HD as they do not require point correspondences between the real point cloud and the simulated mesh, are efficient, and permutation invariant. We use the unidirectional (also known as one-way or one-sided) CD and HD to address different mesh resolutions, and incomplete point clouds due to self-occlusions, as done in previous works facing the same issues on clothes [24, 32]. For a point cloud $\mathcal{P}_{t}$ and a simulated mesh with vertices $\mathcal{V}_{t}$, the CD used for evaluating the reality gap is defined as

$\displaystyle\text{CD}(\mathcal{V}_{t},\mathcal{P}_{t}):=\frac{1}{|\mathcal{V}_{t}|}\sum_{v\in\mathcal{V}_{t}}\min_{p\in\mathcal{P}_{t}}\|v-p\|_{1}\,.$

(2)

The unidirectional HD with $\ell_{1}$ norm is defined as

$\displaystyle\text{HD}(\mathcal{V}_{t},\mathcal{P}_{t}):=\max_{v\in\mathcal{V}_{t}}\min_{p\in\mathcal{P}_{t}}\|v-p\|_{1}\,.$

(3)

The HD is closely related to the CD and greater by definition, as it corresponds to the largest error, whereas the CD is an average of errors. Both metrics typically use the squared Euclidean distance. However, we empirically find that the error values obtained with the Manhattan distance are more representative. The reason for that is that the $\ell_{1}$ norm is more robust to outliers, an observation consistent with the use of the un-squared $\ell_{2}$ norm as an evaluation measure in previous works [33, 34]. Note that $|\mathcal{V}_{t}|\ll|\mathcal{P}_{t}|$, further motivating the use of the $\ell_{1}$ norm, which could cause the metric to blow up in the presence of a few extreme values.

To evaluate the modelling of cloth dynamics, we use the recorded trajectory before the collision, as detailed in Sec. III-B. For this purpose, we report in Table II the average of the Chamfer and Hausdorff distances between the simulated mesh vertices and point clouds up to the change of phase time step, denoted $\text{CD}_{d}$ and $\text{HD}_{d}$.

The quasi-static trajectory is used in its entirety to evaluate the simulation of contacts in the absence of fast dynamic motions. The reported metrics in this case are $\text{CD}_{q}$ and $\text{HD}_{q}$, representing the average of distances across all time steps.

The HD is closely related to the CD, and, by definition, it has a value greater than or equal to it. Both distances determine point correspondences by finding the closest pairs between sets. However, the CD reports the average of distances and hence has higher tolerance for outliers, while the HD is a stricter metric that focuses on the maximum dissimilarity. Overall, both metrics offer complementary and valuable information about the reality gap. One of the drawbacks of both the CD and HD is that they do not consider the connectivity of the mesh [35]. However, in our case, the mesh connectivity is already enforced by the physics simulator.

We provide as a reference the error metrics between each of the target point clouds in Tab. II. The table measures the difference in their deformation and serves as a guide to understand the metric values in Sec. IV-B.

To evaluate the simulator stability, we apply a moving average filter to the simulated vertices and compute the difference between the filtered and non-filtered vertices as

$$\mathcal{L}_{\text{s}}=\dfrac{1}{N}\sum^{N-1}_{t=1}\left|\dfrac{\mathcal{V}_{t-1}+\mathcal{V}_{t}+\mathcal{V}_{t+1}}{3}-\mathcal{V}_{t}\right|.$$

(4)

Finally, to measure the capability of using the simulators in real-time control, we measure the computational time taken to perform a single simulation step and contrast it against the simulator frequency and error metrics aforementioned.

The simulation engines³³3The experiments were performed using MuJoCo v3.1.1, Bullet v3.26, Flex v1.2 and SOFA v23.06. selected for our experiments and their main differences are depicted in Tab. I.

Prior to evaluating the reality gap for each physics engine, we optimise the simulator cloth parameters that best fit the behaviour of each dataset using the standard optimisation procedures of Bayesian Optimisation (BO) [37] and the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [38]. For doing so, we run 500 sweeps of both BO and CMA-ES in each simulation engine, where we minimise the CD against each fabric and quintic trajectory, and keep the random seed constant. Once the number of sweeps is reached, or the optimisation converges, we select the parameters leading to the lowest distance over BO or CMA-ES. The specific parameters used for each simulator can be found in our open-source code. We use the default numerical integrators for Bullet and Flex, semi-implicit Euler for MuJoCo, and implicit conjugate gradient for SOFA.

We evaluate the reality gap against each fabric using 20 random seeds per simulation engine. The quantitative results for each fabric, task and simulator are reported in Tab. III. Overall, all engines perform similarly for the quasi-static task, where all distances are in the same order of magnitude for both CD and HD. In contrast, the difference in performance in the dynamic task is more noticeable across engines. Both MuJoCo and SOFA present distances two times lower than those in Bullet and Flex.

In general, the values for all metrics are comparable, or greater, than the distances between the chequered and linen rags shown in Tab. II.

In addition, we qualitatively assess the reality gap in each simulator by visualising both simulated and dataset cloth point clouds in Fig. 4, where we randomly select one of the simulations obtained for the optimal parameters of the towel rag. The figure also shows the distances associated to each time step, which helps to further understand the metrics. We can notice that MuJoCo is the only engine that closely follows the dynamic trajectory. In contrast, although Bullet presents low values at $t=1.5$, the identified parameters do not produce a stable simulation, resulting in an inconsistent result at $t=3.5$. On the other hand, all simulators are able to closely match the quasi-static trajectory, where Flex has the lowest error at the final time step.

We report the relationship between reality gap, computational time⁶⁶6All experiments were run using an Intel i7-10875H and an RTX 2070. and simulator frequency in Fig. 5, where we report the average CD for the dynamic and quasi-static tasks. We only report the stability values for the dynamic task due to the difficulty of the engines to simulate this trajectory. We used the benchmark data from the towel rag with 10 random seeds per simulation engine, while keeping the same fine-tuned simulation parameters as in Sec. IV-B. We selected $10$, $100$ and $1000$ Hz as frequencies for each engine.

As shown by Fig. 5 b) both Bullet and MuJoCo become unstable when using a low frequency, while Flex and SOFA are more consistent at different frequencies. We can notice a drastic improvement in performance for MuJoCo when increasing the frequency in both Fig. 5 c) and Fig. 5 d). By contrast, Flex and SOFA present similar values at different frequencies. Although higher frequencies result in a more stable computation of the system dynamics, there is no improvement in the distance, or even some detrimental performance. This suggests that the physics engines are quite sensitive to the cloth parameters. Therefore, all engines need to be fine-tuned for the specific simulation frequency.

The computational time taken per simulation step is depicted in Fig. 5 a). We can notice that, for the case of Bullet and MuJoCo, if the simulator is unstable the time drastically increases. Given that the time taken per simulation step for 100Hz for all simulators is in the order of milliseconds, it is unfeasible to perform real-time dynamic manipulation with hardware-in-the-loop. Similarly, although the simulation step time for 10Hz in Flex and SOFA is lower, and they are more stable than MuJoCo and Bullet, their CD is still pretty high for hardware-in-the-loop manipulation.