Sim-to-Real for Robotic Tactile Sensing via
Physics-Based Simulation and Learned Latent Projections

In [41], visual output of the GelSight[4] was simulated using a depth camera in Gazebo[42], a calibrated reflectance model, and blurring to approximate gel contact. Quantitative evaluation was limited. In [43], a custom marker-based tactile sensor[9] was simulated using FEM, optics models, and synthetic noise. A U-net was trained on the synthetic data to regress to contact force fields with a resultant error of $0.14N$; almost all training and testing was conducted on 1 normally-oriented indenter. In [44], the pin locations of the TacTip[5] were simulated using an approximate deformation model in Unity[45]. Parameters were tuned for a plausible baseline, and domain randomization was performed. A fully connected network (FCN) was trained on synthetic data to regress to contact location and angle with a minimum error of $0.5$-$0.7mm$ and $0.25rad$. The sensor examined was large, and contact was made over limited orientations.

In comparison to these studies, we use a compact sensor, accurately simulate contact and deformation, do not perform domain adaptation beyond projection, and conduct simulations and experiments on 17 objects over diverse kinematics.

In [46], the electrical output of the BarrettHand[47] capacitive sensor array was simulated using soft contact in PyBullet[48]. RL policies for stable grasps were trained on the simulator and transferred to the real world. Binary thresholding was applied to tactile signals, limiting precision. In [49], an electrical resistance tomography sensor was simulated using a simplified FEM deformation model, an empirical piezoresistive model, and an FEM conductivity model. FCNs were trained on synthetic data to predict discrete contact location and resultant force, with an $82\%$ success rate for contact and a mean force error of $0.51N$. For unseen contact scenarios, the error increased to $5.0N$.

Finally, for the BioTac sensor, multiple studies have addressed the inverse problem of converting electrical output to physical quantities, such as contact location[50, 22, 23], force[50, 14, 21, 22, 23], and deformation[22, 23]. In particular, our previous work[22] presented a 3D FEM model of the BioTac, which was built with the industry-standard, CPU-based simulator ANSYS[51] and carefully validated against experimental data. Electrode signals were then mapped to simulator outputs via PointNet++[52]. The simulations were slow ($7min$ each on 6 CPUs). In addition, the forward problem of synthesizing electrical output was not addressed; some progress was made in our extension[23].

The forward problem has been further explored in [53, 36, 54]. In [53], the BioTac was simulated with an approximate contact model in Gazebo. A deep network was trained to regress from estimated force and real-world contact location to electrical outputs. An existing location estimator was tested on synthetic and real-world data with a mean difference of $0.83mm$. Training and testing was conducted only on 1 spherical indenter. In [36], PointNet was used to regress from RGB-D images and grasp parameters to tactile readings, and in [54], semi-supervised learning was applied. Electrode prediction errors were relatively high for both the simple case of unseen samples, as well as the difficult case of unseen objects; numerical comparison is provided later.

In contrast to the preceding works, we develop an efficient FEM model, conduct simulations and experiments with 17 objects, regress to continuous electrical signals, and demonstrate accurate predictions for unseen objects.

Numerous studies in robotics have established the utility of multimodal data in task-specific learning. For example, in [55, 56], vision and tactile sensing were combined to predict grasp outcomes and select adjustments, and in [57], vision and force/torque (F/T) sensing were combined to perform manipulation tasks. In addition, in [58], vision, F/T sensing, and proprioception were used to learn a joint latent space via self-supervision with autoencoders; the output served as perception input to an RL agent for peg-in-hole.

Simultaneously, recent works both outside and within robotics have investigated cross-modal transfer. In [59], audio-video transfer was performed by learning a shared representation via a bimodal autoencoder; networks trained on one modality were then able to classify the other. In [60], audio-image transfer was achieved via generative adversial networks (GAN). In [61, 62], cross-modal transfer was performed between data from cameras and vision-based tactile sensors, and in [36, 54], transfer was achieved between cameras and output from the BioTac SP.

Finally, previous efforts have applied distinct network architectures to encode BioTac-specific data. In our previous work [21, 22, 23], a 3D voxel-grid network, PointNet++, and an FCN were implemented to encode BioTac electrode data and regress to physical quantities such as forces and deformations. In [63], a 2D CNN was used to predict tactile directionality, and in [64], a graph convolutional network (GCN) was used to predict grasp stability.

We draw upon the preceding works with some distinctions. Analogous to [59], we learn latent representations of FEM and BioTac electrical data via self-supervision with autoencoders for cross-modal transfer. Like [58], we learn modality-specific representations, reducing training time and eliminating zero-inputs for non-present modalities. Unlike both, we never formulate a joint representation, but instead learn a projection between the latent spaces using a small amount of supervised data. To encode BioTac electrical data, we use VAEs, as for vision-based tactile sensors in [6].

FEM is a variational numerical formulation that 1) divides complex global geometries into simple subdomains, 2) solves the weak form of the governing PDEs in each subdomain, and 3) assembles the solutions into a global one. In 3D FEM, objects are represented as volumetric meshes, which consist of 3D elements (e.g., tetrahedrons or hexahedrons) and their associated nodes. With careful model design, high-quality meshes, and small timesteps, FEM predictions for deformable bodies can be exceptionally accurate [65, 66].

In this work, 3D FEM was performed using NVIDIA’s GPU-based Isaac Gym simulator[37]. Isaac Gym models the internal dynamics of deformable bodies using a co-rotational linear-elastic constitutive model; these bodies interact with external rigid objects via an isotropic Coulomb contact model [67]. The resulting nonlinear complementarity problem is solved via implicit time integration using a GPU-based Newton solver [68]. At each timestep, Isaac Gym returns nodal positions (i.e., deformation fields), nodal contact forces (used to compute resultant forces), and element stress tensors.

To create the FEM model for the BioTac, high-resolution triangular meshes for the external and internal surfaces of the BioTac skin were first extracted from the ANSYS model in [22, 23] and simplified via quadric edge collapse decimation in MeshLab[69]. A volumetric mesh was then generated with fTetWild[70]; similar to [22, 23], the mesh had $\approx 4000$ nodes. Fixed boundary conditions (BCs) were applied to 2 sides of the skin to model the BioTac nail and clamp, respectively; these BCs were implemented by introducing thin rigid bodies at the corresponding locations (visible in Fig. 2A-D), which were attached to adjacent nodes on the skin. External rigid objects (e.g., indenters) were driven into the BioTac via a stiff proportional controller.

Relative to the experimentally-validated ANSYS model in [22, 23], the Isaac Gym model makes 3 critical approximations: 1) collisions are resolved via boundary-layer expanded meshes[71], rather than a normal-Lagrange method, 2) a compressible linear-elastic model is used for the skin, rather than a Neo-Hookean model[72], 3) the internal fluid is not modeled. The effects of the first approximation are mitigated by using small collision thicknesses and timesteps ($1\mathrm{e}{-4}s$). However, the second and third approximations are mitigated by endowing the Isaac Gym model with sufficient expressivity and optimizing it to reproduce ANSYS results.

Specifically, 4 distinct geometric configurations of the BioTac were considered in Isaac Gym (Fig. 2A-D). The material properties (elastic modulus $E$, Poisson’s ratio $\nu$, and friction $\mu$ of the BioTac skin) were designated as free parameters. In [22, 23], 1 shear-rich indentation of the BioTac was used to calibrate the ANSYS model against real-world data; in this work, the same indentation was resimulated in Isaac Gym and used to calibrate the Gym model against ANSYS data (Fig. 2E). For each configuration, the material properties were optimized via sequential least-squares programming to reproduce the force-deflection profile from ANSYS for the indentation. The cost was defined as the RMS $\ell^{2}$-norm of the force error vector over time. Subsequently, each optimized configuration was evaluated by resimulating 358 additional indentations from 8 indenters in [22, 23], and comparing the results to the force-deflection profiles from ANSYS.

Among the 4 configurations, the deformable solid (Fig. 2D) produced the lowest cost during optimization; however, the deformable shell with rigid core (Fig. 2B) produced the lowest cost during evaluation and was thus selected. For this representation, the optimal values of $E$, $\nu$, and $\mu$ were $1.55e6Pa$, $0.316$, and $0.783$, respectively; the mean $\ell^{2}$-norm of the force error vector was $0.125N$ (Fig. 2E). In comparison, the optimal values for the ANSYS model in [22] were $2.80e5Pa$, $0.5$, and $0.186$, indicating that the Isaac Gym model compensated for its linearity and lack of fluid by increasing elastic modulus, compressibility, and friction.

Although FEM captures the effects of contact on BioTac deformations, the BioTac then transduces deformations to fluidic impedances measured at electrodes. Simulating the mapping between deformations and impedances is complex; thus, this mapping was learned. Specifically, modality-specific latent representations were learned using self-supervision, facilitating compression, mitigating noise, and reducing overfitting. Projections were then learned between the latent spaces, enabling cross-modal transfer (Fig. 3).

To learn a latent representation for the FEM deformations, convolutional mesh autoencoders from [39] were trained, which applied graph convolutional networks (GCN) and reduced the mesh data from 4000 nodal positions ($\mathbb{R}^{4k\times 3}$) to 128 units. To learn a latent representation for real-world BioTac electrode signals, an autoencoder composed of FCNs was trained, which reduced the electrode data from 19 impedances to 8 units. Latent dimensions were chosen via hyperparameter search. Both networks were trained as VAEs to generate smooth mappings to the latent space[38].

To learn the projections between the latent spaces, 2 FCNs were trained. The first network projected forward from the FEM mesh latent space $z_{m}$ to the BioTac electrode latent space $z_{e}$, whereas the second network projected inversely from $z_{e}$ to $z_{m}$. During training, the previously-described VAEs were frozen and provided with supervised data from [22], generating latent pairs of $z_{m}$ and $z_{e}$. These pairs were used to train the projection networks with an RMS loss. Without freezing the VAEs, the networks overfit.

In the FEM deformation VAE, the encoder consisted of an initial convolution with filter size 128, 4 “convolve+downsample” layers with sizes $[128,128,256,64]$ and downsampling factors $[4,4,4,2]$, a convolution with size 64, and 2 fully-connected layers with dimensions $[512,128]$. The decoder was symmetric with the encoder, using “upsample+convolve” instead of “convolve+downsample”. In the BioTac electrode VAE, the encoder consisted of 4 fully connected layers with $[256,128,64,8]$ neurons, respectively. The decoder was again symmetric with the encoder. The forward and inverse projection networks consisted of 3 fully connected layers with $[256,128,128]$ and $[128,128,256]$ neurons, respectively, and dropout of $0.3$. Exponential linear unit (ELU) activations were applied, and the Adam optimizer was used with initial learning rate of $1e$-$3$ and decay of $0.95$.

For the preceding learning steps, data was collected in both simulation and the real world. For learning the latent representations, unlabeled mesh data was collected by simulating kinematically-randomized interactions on the optimized BioTac model with 6 pegs and 3 table features (surfaces, edges, and corners) in Isaac Gym. Unlabeled experimental electrode data was collected by manually indenting these objects in the real world. For learning the latent projections, labeled data was collected by exactly resimulating 359 indentations from [22, 23] on the optimized BioTac model in Isaac Gym (as stated in Sec. III-A); these were directly matched to corresponding experimental electrode data in the dataset from [22, 23]. 72% of contact interactions were allocated for training, 18% for validation, and 10% for testing.

In total, 2.6k unique contact interactions were executed and 50k timesteps of FEM data were sampled. All objects used in simulation and experiments are shown in Fig. 4. Data collection visualizations are in the supplementary video.

As described earlier, the force-deflection profiles produced by the optimized model in Isaac Gym were compared to those produced by ANSYS over 358 indentations distributed across 8 indenters (Fig. 5). The mean $\ell^{2}$-norm of the force error vectors over all indentations ranged from $0.0876N$ for a medium-sized cylindrical indenter (less than the training error of $0.125N$) to $0.259N$ for a medium-sized spherical indenter, with a mean of $0.153N$ across all indenters. Thus, despite being optimized using force-deflection data from only a single indentation, the Isaac Gym model strongly generalized across a diverse range of objects and indentations.

The corresponding FEM deformation fields (i.e., the nodal positions of the deformed FEM meshes) were also compared between Isaac Gym and ANSYS (Fig. 5). For each dataset, the maximum and mean $\ell^{2}$-norms of the nodal displacement vectors were computed over all indentations. The mean error between the maximum norms across all indenters was $1.41\mathrm{e}{-4}m$, and the mean error between the mean norms was $1.81\mathrm{e}{-5}m$. Thus, again, the Isaac Gym model was shown to strongly generalize. These low errors were particularly important, as the nodal deformation fields from Isaac Gym were used as input for subsequent learning.

Finally, simulation speed was compared between the Isaac Gym and ANSYS models. The total simulation time for all 359 indentations was approximately 42 hours (7.08 minutes per sim) on 6 CPUs in ANSYS, but 33 minutes (5.57 seconds per sim) using 8 parallel environments (1 per indenter) on 1 GPU in Isaac Gym. For clarity, Isaac Gym can only currently simulate deformable solids with a linear-elastic model and linear tetrahedral elements; such a model comprises only a small fraction of those that can be simulated within state-of-the-art FEM software such as ANSYS. However, for the current application, Isaac Gym is highly favorable.

For regression from FEM deformations to BioTac electrode signals, 2 learning methods were evaluated: 1) the method proposed in this paper, denoted Latent Projection, which used unlabeled data for latent representation and labeled data for projection, and 2) a PointNet++ baseline [52], denoted Fully Supervised, which used only labeled data, with 128 nodes sampled from the FEM mesh as in [22, 23]. For reference, output is also shown for the Electrode VAE, which is used when training Latent Projection.

When evaluating generalization to novel objects, networks were trained on all objects except the ring (see Fig. 5) and tested on this indenter; the ring has the most unique (thus, challenging) geometry in the dataset. These experiments are denoted “Unseen Object.” (Results for other unseen objects are on our website.) When evaluating generalization to novel contact interactions with seen objects, the trained networks are tested on unseen interactions with the other indenters. These experiments are denoted “Unseen Trajectory.”

A qualitative comparison of regression results between the learning methods is depicted in Fig. 6A for 2 high-signal, high-variation electrodes over numerous interactions. Raw electrode values were between [0, 4095] digital output units and were tared and normalized to [-1, 1]. For the challenging “Unseen Object” case, Latent Projection could predict several signal peaks over multiple electrodes that Fully Supervised could not capture. Additionally, Latent Projection predictions were consistently noise-free, whereas Fully Supervised ones exhibited low-magnitude, high-frequency noise. Finally, Latent Projection often outperformed Electrode VAE, showing the importance of mesh information in electrode signal synthesis. Predictions for the easier “Unseen Trajectory” case have higher fidelity and are thus not shown.

Quantitative comparisons between the learning methods are illustrated in Fig. 6B-C. RMS errors over all electrodes and interactions are compared in Fig. 6B. The Latent Projection approach performs better than Fully Supervised for both “Unseen Trajectory” and “Unseen Object,” in terms of both median error and interquartile range. Median errors for “Unseen Trajectory” were Fully Supervised: $0.012$ (25 raw units) and Latent Projection: $0.010$ (20 units), and those for “Unseen Object” were $0.016$ (32 units) and $0.015$ (31 units), respectively. Our errors are substantially lower than the errors from [54], which were $195$ units for an easier case of unseen samples, and $305$ units for unseen objects. A coverage plot is shown in Fig. 6C, where the $\ell^{1}$ distance to the ground-truth electrode signals is depicted. For both “Unseen Trajectory” and “Unseen Object,” Latent Projection outperforms Fully Supervised for nearly the entire distribution of data.

For regression from BioTac electrode signals to FEM deformations, we again evaluate Latent Projection. For reference, we also show output of the FEM Mesh VAE, which is used when training Latent Projection. A visual comparison is shown against ground-truth FEM output in Fig. 7 for random indentations. For “Unseen Trajectory”, Latent Projection consistently predicted ground-truth, capturing deformation magnitudes and distributions. For “Unseen Object”, Latent Projection consistently captured magnitudes and distributions, but not bimodality. As seen from Mesh VAE, this limit is due to the mesh autoencoder (specifically, bottleneck size) rather than the projection. As before, we only show results for the most challenging unseen object, the ring; strong performance on other objects is shown in the video.

As a final demonstration, we also conducted free-form interactions of the BioTac with unseen objects and visualized the estimated contact patches (Fig. 8). For diverse pegs and table features, predicted deformations were visually accurate. For instance, contact patch locations were accurately predicted across the full spatial limits of the BioTac, and interactions with the corners of a cuboid peg and table showed high-magnitude, highly-localized patch deformations.