Elastic Tactile Simulation Towards Tactile-Visual Perception

The basic idea of our method is to assume both the tactile sensor and the manipulated object to be solid and model them in the form of particles. Differently, the sensor and the object are considered elastic and rigid respectively. In general, the tactile simulation process depicted in Figure 2 consists of 3 steps: voxelization from meshes to particles, interaction simulation, and reconstruction from particles to meshes. We detail each step below.

This subsection presents details of how we simulate the interaction process. In practice, the tactile sensor is basically made of elastic materials, and our main focus is on the change of its shape, or called deformation. We apply the elastic theory to constrain the deformation of the particles in the tactile sensor during its interaction with the manipulated object, and the deformation at each time step will be recorded as the tactile sensing data.

Suppose that the sensor is composed of $m$ particles. The coordinate of the $p$-th particle is represented by ${\bm{x}}_{p}\in\mathbb{R}^{d}$, where $d=3$ throughout our paper. We define the deformation map as $\bm{\Phi}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$. The Jacobian of $\bm{\Phi}$ with respect to the $p$-th particle, denoted as ${\bm{F}}_{p}\in\mathbb{R}^{d\times d}$ (a.k.a deformation gradient), is calculated by

When the particle deforms, its volume may also change. The volume ratio by the deformation, denoted as $J_{p}$, is the determinant of ${\bm{F}}_{p}$, i.e.,

To describe the stress-strain relationship for elastic materials, we adopt a strain energy density function $\bm{\Psi}$, a kind of potential function that constrains the deformation ${\bm{F}}_{p}$. We follow a widely used method called Fixed Corotated (stomakhin2012energetically, ), which computes $\bm{\Psi}$ by

where $\mu=\frac{E}{2(1+v)}$ and $\lambda=\frac{Ev}{(1+v)(1-2v)}$ are Lamé’s 1st and 2nd parameters, respectively; $E$ and $v$ are Young’s modulus and Poisson ratio of the elastic material, respectively; $\sigma_{i,p}$ is the $i$-th singular value of ${\bm{F}}_{p}$. The derivative of $\bm{\Psi}$ (a.k.a the first Piola-Kirchhoff stress) will be utilized to adjust the deformation process, derived by

where ${\bm{R}}_{p}$ is obtained via the polar decomposition (higham1986computing, ): ${\bm{F}}_{p}={\bm{R}}_{p}{\bm{S}}_{p}$.

In the following context, we will characterize how each particle deforms, that is, how its position ${\bm{x}}_{p}$ changes during the interaction phase. For better readability, we distinguish the position ${\bm{x}}_{p}$ and the quantities in Eq. (1-4) at each different time step by adding a temporal superscript, e.g. denoting the velocity at time step $n$ as ${\bm{x}}_{p}^{(n)}$. We leverage MLS-MPM (hu2018moving, ) to update ${\bm{x}}_{p}^{(n)}$, which divides the whole space into grids of a certain size. For each particle, its velocity is updated as the accumulated velocity of all particles within the same grid, which to some extent can emulate the physical interaction between particles. Specifically, we iterate the following steps for the update of ${\bm{x}}_{p}^{(n)}$. The flowchart is sketched in Algorithm 1.

CNN-based architectures have shown the superiority in homogeneous multimodal fusion, e.g., RGB-depth fusion (wang2020cen, ), where feature maps of different modalities are naturally aligned. But here, the feature maps of vision and touch are unaligned due to the discrepancy of viewpoint and tactile direction. Existing methods mostly leverage estimated global or local depth for multi-modal alignment. In this section, we propose an end-to-end tactile-visual perceptual framework that directly combines multi-modal features without explicitly predicting depth or sparse point cloud.

Specifically, we focus on the 3D geometric reconstruction of the manipulated objects, where we utilize the simulated tactile signals as complementary information to single-view visual images. During the prediction process, the network input is composed of an image and a set of tactile data obtained from different grasps based on different directions.

The structure of our tactile-visual perception framework is illustrated in Figure 5. We adopt a visual branch and a tactile branch to process visual and tactile inputs, respectively. We choose ResNet (he2016deep, ) as the backbone network for both branches, and thus there are four stages depicted. Multi-scale tactile features are extracted and densely connected to each network stage in the visual branch for multimodal feature fusion. We then concatenate multi-stage feature maps as a perceptual feature, which is finally sent to a Graph Neural Network (GNN) to predict the vertices deformation of the 3D mesh following the scheme in Pixel2Mesh (wang2018pixel2mesh, ).

The main contributions of our method include a designed Aggregation (Agg) block and the global-to-local feature fusion method to alleviate the misalignment issue. As illustrated in Figure 6, Agg transforms tactile features before integrating them into the visual branch. The transformation consists of a Global Average Pooling (GAP), which results in a global embedding, and two $1\times 1$ Conv layers for feature extraction. Given that the touch direction plays an important role in tactile representation, we additionally involve the direction information into our tactile processing, by using a Feed-Forward Network (FFN) (that is shared in different Agg blocks) to produce the direction embedding which is integrated with the tactile global embedding via dot product. Note that in Agg, each $1\times 1$ Conv layer is followed by a ReLU activation and a batch normalization layer. As described in Sec. 1, the tactile-visual feature maps are not naturally aligned due to the discrepancy of viewpoint and tactile direction. In our tactile generation, we are able to record the position (i.e. the coordinate center of tactile pixels). With the help of the tactile position, we then locate the corresponding local region (the $3\times 3$ region colored in red in Figure 6) in each visual feature map and then perform addition between the global tactile embedding by Agg and every pixel within the local visual region. It is known that the different-stage layer in neural networks characterizes different-scale patterns. Hence, we conduct the above tactile-visual Agg between every two stages across the tactile and visual branches, to enable multi-scale and delicate fusion.

Inspired by (wang2018pixel2mesh, ), the perceptual feature is sent to a GNN to guide the deformation process of GNN vertices, which are also the vertices of the 3D geometric model to be reconstructed. The geometric model is initialized to an ellipsoid mesh and the deformation is realized by updating 3-dimensional vertices embeddings (coordinates) of GNN. Each vertex on the mesh would be projected to the closest visual feature (wang2018pixel2mesh, ). The deformation update process is illustrated in the right part of Figure 5, and can be formulated as below,

where ${\bm{f}}_{p}$ denotes the perceptual feature learned by tactile-visual fusion, which has been introduced above (annotated in a blue box in Figure 5); $v$ denotes the index of the vertex on the mesh; $\mathcal{N}(v)$ denotes the neighbors of $v$, which is available as the mesh is initially a sphere; ${\bm{f}}_{v}^{l},{\bm{x}}_{v}^{l}$ are the feature representation and the learned coordinate of vertex $v$ for the $l$-th GNN layer, respectively; ${\bm{g}}_{v}^{l}$ is the hidden feature given by the concatenation of ${\bm{f}}_{v}^{l},{\bm{x}}_{v}$ and a multimodal feature projection on ${\bm{f}}_{p}$, denoted as $\text{proj}({\bm{f}}_{p})$; ${\bm{W}}_{f0},{\bm{W}}_{f1},{\bm{W}}_{x0},{\bm{W}}_{x1}$ are learnable weights.

In Figure 7, we provide the tactile patterns when pressing a spoon, and compare the patterns under different grid numbers (described in § 3.2). We observe that the larger the number of grids is the more fine-grained simulation we will attain. We set the grid number as $128^{3}$ considering the trade-off between efficacy and efficiency. To illustrate how the deformation behaves during the contact, in Figure 8, we keep pressing the tactile sensor on a gear object and record results at different time steps. For each time step, we also provide its corresponding format for CNN input, and the Chamfer distance $l$ as described in § 3.2; The deformation becomes more remarkable as the contact proceeds. Figure 9 contrasts the influence by Young’s modulus $E$ under the same press displacement. It is shown that the tactile range gets smaller with the increase of Young’s modulus, which is consistent with the conclusion in elastic theory. In our simulation, we choose $E=3$ and $v=0.25$ in Eq. 3 by default.

We build a tactile dataset containing 7,000 tactile images of 35 different object classes. These tactile images are collected through different contact policies including press directions and forces. The dataset summary and train/test splits are provided in Table 1. Figure 10 illustrates an example subgroup of the tactile dataset.

Once we obtain the tactile data (namely, the deformation of particles of the sensor), we can apply these data for object recognition which partially evaluates the quality of the tactile data. We suppose the tactile deformation of the contact surface as ${\bm{X}}^{(N)}\in\mathbb{R}^{H\times W\times d}$, where $H$ and $W$ denote the height and the width of the sensor, respectively, and $N$ denotes the final time step. Then, we train a neural network $\hat{y}=f({\bm{X}}^{(N)})$ to predict the object label. In practice, we prefer to try several attempts of the touch for more accurate recognition. All the deformation outcomes of different touching, denoted as $\{{\bm{X}}_{i}^{(N)}\}_{i=1}^{I}$ will be concatenated along the channel direction, leading to ${\bm{Z}}\in\mathbb{R}^{W\times H\times(Id)}$, as input of the network $f$.

To predict the object category given tactile patterns, we train a ResNet-18 (he2016deep, ) with ImageNet pretraining. We also conduct prediction experiments with more than one input tactile image. Specifically, during each training iteration, we randomly choose $N$ tactile images which are yielded by different press directions to the same object. As introduced in § 3.3, we treat the number of images as the number of the input channel. Table 2 summarizes the accuracies of the tactile perception with the number of input images (touches) from 1 to 10. It reads that increasing the number of touches consistently improves the classification accuracy, and when the number is equal to 10, the accuracy becomes close to $93\%$, which implies the potential usage of our tactile simulation for real robotic perception.

In this part, we assess the performance of 3D mesh prediction given a single-view image and a certain number of simulated tactile data by our method. The evaluation is accomplished on 3D mesh models of 10 classes. We collect visual images under random view and tactile data from different press directions. Per each training iteration, we randomly sample 1 visual image and 1$\sim$10 tactile images as the network input. We perform random deformation on meshes so that there are no intersections of 3D meshes for training and testing.

We adopt ResNet-18 as our tactile-visual perception backbone and resize both visual and tactile images to the resolution $224\times 224$. In addition, we adopt the cosine learning rate scheduler with an initial learning rate $2\times 10^{-5}$, and we train 50 epochs in total.

The quantitative comparison is provided in Table 3, where we calculate the average Chamfer distance as the evaluation metric by sampling both 1,000 points from the predicted and target meshes. We vary the input number of tactile images from 2 to 10 in analogy to the perception task before. Even surprisingly, our method with only 2 touches is sufficient to gain a smaller reconstruction error than Pixel2Mesh (the Visual column in the table), and the error will become much smaller if we increase the number to 10. Table 3 also provides comparison results with a current SOTA the Chart-based method (DBLP:conf/nips/SmithCRGMMD20, ) which adopts estimated local depth and point cloud for tactile-visual fusion. We use the public codes by the authors and share their default setting of generating 5 touches at each time. Thus, the tactile input by (DBLP:conf/nips/SmithCRGMMD20, ) could only be the multiplier of 5 (5, 10, etc). According to the results in Table 3, our method surpasses the current the Chart-based method with a large margin, probably thanks to the more elaborate multimodal fusion in our method.

Table 4 provides detailed evaluation results for each of the 10 classes. We also perform ablation studies to verify the effectiveness of each component that we propose. The ablation variants include:

• •

  W/o GAP: the global average pooling is removed from our model and the feature maps of tactile and visual modalities are fused pixel-by-pixel. This ablation study is to justify the validity of our global-to-local fusion by GAP on alleviating multimodal feature misalignment.

• •

  W/o tactile direction: the learned embedding of tactile direction (as illustrated in Figure 6) is no longer integrated with the Agg embedding.

• •

  $1\times 1$ per-pixel fusion: as illustrated in Figure 6, each aggregated tactile embedding is added to the $3\times 3$ visual pixels. We change $3\times 3$ to 1 single visual pixel to observe if attending the local region is necessary.

• •

  $5\times 5$ per-pixel fusion: conversely, we change $3\times 3$ to $5\times 5$ to see whether the performance is benefited from a larger visual receptive field.

• •

  Single-stage fusion: we remove the dense multi-scale connections between the tactile and visual branches and only allow one connection between each corresponding stage.

Observed from Table 4, we have the following findings:

• •

  GAP plays an essential role in our tactile-visual perception method, as removing GAP leads to noticeable performance drops regarding all objects especially for those elaborate ones (e.g. spoon). This is possibly because adopting GAP helps alleviate the misalignment issue of multimodal features as mentioned in § 3.3.

• •

  The tactile direction is also crucial in replenishing the information in tactile embeddings, since the generation of tactile data depends on what touch direction we apply.

• •

  For some elaborate objects, e.g. spoon and scissors, adopting $1\times 1$ per-pixel fusion can achieve promising or even better performance than the default $3\times 3$ setting; similarly, $5\times 5$ yields the superiority accuracy regarding some uniform objects. Overall, the $3\times 3$ setting comes as the best choice for general classes of the evaluated objects.

• •

  Performing single-stage fusion still produces desired results, but it is inferior to the multi-scale version.

Figure 11 displays the generation process of the 3D mesh models. We compare our method with Pixel2Mesh (wang2018pixel2mesh, ) that only adopts the visual input. Qualitatively, by adding the tactile input, our approach obtains better predictions than Pixel2Mesh in Figure 11. The experimental results here well verify the power of our tactile simulation in capturing the fine-grained patterns of the touched object.

We integrate our tactile simulation with the robot environment, to perform the pick-and-place task for several different objects. We first fuse RGB and depth information to get the corresponding semantic segmentation based on the multimodal fusion method in (wang2020cen, ). With the segmentation at hand, we detect the 3D position of the can and then pick it up and finally put it down at a different place. The whole process is depicted in Figure 12, below which we plot the corresponding tactile simulation for each phase. We observe that our tactile simulation does encode the cylinder shape of the can. Besides, the last column shows that the simulated tactile sensor can return to its original state after the grasping process.