ThinTact: Thin Vision-Based Tactile Sensor by Lensless Imaging

Tactile sensing has been increasingly applied in robot manipulation, especially for contact-rich scenarios [15]. According to Wang et al.’s review [16], tactile sensors’ principle can be categorized into the following modes: resistive, capacitive, piezoelectric, frictional, and optical mechanisms. Vision-based tactile sensors, a type of sensor that operates on optical mechanisms, have recently attracted significant attention. Their advantages lie in their superior spatial resolution, robustness against electromagnetic interference, cost-effectiveness, and the ability to capture rich contact information including simultaneous measurement of normal and tangential forces [5, 6]. The GelSight sensor represents a notable example of vision-based tactile sensors [17, 9, 18, 19]. It captures high-resolution color images of the deformed elastomer surface which is illuminated from various angles. These images provide rich information of the contact surface, including 3D geometries and deformation. Aside from GelSight, there are other vision-based tactile sensors which can also provide rich contact information [20, 10, 21, 22]. However, the common limitation of these vision-based tactile sensors is their large thickness and volume, which restricts the application in cluttered scenes and integration with dexterous grippers.

To address the aforementioned issue, compact tactile sensors have been developed. A common approach involves the use of mirrors to fold the optical path, as seen in systems like GelSlim [23] and GelSight Wedge [24]. However, despite that the camera is relocated, constraints in sensor integration persist due to the considerable scene-to-CMOS distance. Several studies have comprehensively designed robotic fingers or grippers with tactile sensing [25, 26, 27, 28, 29]. These designs allocate substantial space for optical transmission. Consequently, altering their form factor for integration into other robotic components poses a significant challenge. Other works have developed sensors based on small-sized cameras [30] or wide-angle cameras [19] and achieve impressive results. However, the resulting thickness still exceeds that of human fingers. In summary, none of these sensors have a thickness of less than 20 mm, making it challenging to achieve human-level dexterity.

Recently, several vision-based tactile sensors have adopted multi-lens imaging systems instead of the traditional single-lens models [31, 32, 33, 34, 35]. Despite the flexibility offered by multi-lens systems, most of them still possess a bulky size. Therefore, our focus is on the work of Chen et al. [35], who proposed a tactile sensor with only 5 mm thickness. The sensor utilizes microfabricated micro-lens arrays (MLA) for imaging. However, each micro-lens still adheres to the constraints of object distance, image distance, and magnification ratio, leading to overlapping sub-image areas and a low utilization rate of the image sensor. Additionally, the process of fabricating the MLA and preventing crosstalk introduces higher manufacturing costs.

In this work, we propose a novel, compact, vision-based tactile sensor that utilizes lensless imaging principles. This approach eliminates the constraints in traditional imaging systems and results in a sensor with a thickness of less than 10 mm and a sensing field of over 200 mm². By employing a simple binary mask composed of a glass substrate and a thin chrome film, our sensor offers ease of manufacturing and cost-effectiveness.

In a lensless camera, an optical encoder replaces the traditional lens to encode scene information into sensor measurements. The scene is later recovered using appropriate algorithms. We refer to [36] for a comprehensive review of lensless imaging. In the context of tactile sensing, the appealing advantages of lensless imaging include reduced camera size and large field of view (FOV), because lensless imaging breaks the constraint between working distance and FOV in traditional lens-based imaging.

Designing a lensless imaging system is a systematic task, as the imaging conditions, the choice and design of the optical encoder, the reconstruction algorithm, are closely coupled. To make lensless imaging applicable in tactile sensing, several issues must be addressed. Firstly, the computational burden should be minimized to ensure real-time measurement. Secondly, to design a tactile sensor with a large sensing area and low thickness, the scene-to-CMOS distance is smaller than the CMOS size, which is uncommon in traditional lensless imaging applications.

With these considerations in mind, we delve into the field of lensless imaging. One method to reduce the computational burden is to use a convolutional imaging model, such as DiffuserCam [37] and PhlatCam [38]. This method requires the point spread function (PSF) to be spatially invariant across the entire scene. However, this assumption does not hold when the scene-to-CMOS distance is smaller than the CMOS size. Although [39] proposed a model to account for this situation, it cannot be solved efficiently in real time. Another method to realize real-time reconstruction is to use a separable amplitude mask, known as FlatCam [13]. Adams et al. have extended FlatCam to close-up imaging situations [12], but the corresponding imaging model cannot be solved analytically and efficiently. In this work, we adopt the separable amplitude mask and propose a novel filter to decompose the sensor measurements and obtain the coding component, enabling efficient real-time scene reconstruction. It is noteworthy that recent studies have suggested the application of deep neural networks for lensless image reconstruction [40, 41, 42, 43]. However, in the context of compact tactile sensors, it is difficult to gather scene-measurement-paired datasets for training. Furthermore, these learning-based methods often encounter hallucination artifacts that lack high-fidelity [41]. Therefore, in this work, we have designed the proposed filter based on mathematical and physical properties. This approach not only ensures its generalizability but also guarantees a lower computational burden compared to deep neural networks.

In this work, we choose to use a 2D separable amplitude mask as the coded aperture, which has the advantage of better computational efficiency and lower fabrication cost. The binary mask pattern $\bm{M}\in\{0,1\}^{K\times K}$ is constructed by:

where $\bm{\varphi}_{1},\bm{\varphi}_{2}$ are two vectors, each entry of which is either -1 or +1. In this work, we assume $\bm{\varphi}_{1}=\bm{\varphi}_{2}=\bm{\varphi}$. An example of the separable mask is shown in Fig. 2.

Due to the limited thickness of our proposed tactile sensor, the distance between the scene and the CMOS sensor is small. Consider a point light source present in the scene (Fig. 3). The light source casts a portion of the mask pattern onto the CMOS (Fig. 3), which is referred to as the Point Spread Function (PSF). The PSF can be decomposed into two components: The first component (Fig. 3) corresponds to the term $\bm{1}\cdot\bm{1}^{\top}/2$ in the mask. This component approximates the response of the CMOS when it is exposed to an “open” mask, i.e., a mask without any apertures. The second component (Fig. 3) corresponds to the term $\bm{\varphi}\bm{\varphi}^{\top}/2$ in the mask. This component represents the coding effect of the mask and captures the features introduced by the mask pattern.

Because the measurement $\bm{Y}$ is a superposition of the PSFs of all the points in the scene $\bm{X}$, the imaging model can be expressed as

where $\bm{P}_{\text{o}}\in\mathbb{R}^{R\times N}$, $\bm{Q}_{\text{o}}\in\mathbb{R}^{S\times M}$, $\bm{P}_{\text{c}}\in\mathbb{R}^{R\times N}$ and $\bm{Q}_{\text{c}}\in\mathbb{R}^{S\times M}$ are the system matrices, and the subscripts “o” and “c” denote “open” and “coding”, respectively. We present some typical system matrices in Fig. 3 and Fig. 3. This model is known as the Texas Two-Step (T2S) model [12].

Given the imaging model, the scene $\bm{X}$ can be reconstructed by solving the following optimization problem:

where $\tau$ is the regularization coefficient. Unfortunately, there are no computationally feasible analytical solutions at present. Instead, it can be solved using iterative methods, such as Nesterov gradient method [44] as introduced in [12], whose time consumption typically ranges from a fraction of a second to a few seconds on GPU, which makes it inapplicable in tactile sensing. To address this issue, in the next subsection, we propose a non-iterative reconstruction algorithm to greatly decrease the computational time and enable real-time image reconstruction.

In this section, we propose a non-iterative real-time reconstruction algorithm. It first utilizes a frequency-spatial-domain joint filter to remove the $\bm{P}_{\text{o}}\bm{X}\bm{Q}_{\text{o}}^{\top}$ term. The remaining term is then solved analytically. Fig. 4 shows the proposed algorithm.

In existing separable-mask-based lensless imaging systems, masks are typically designed based on vectors with specific mathematical properties, such as MLS [13]. The advantage of MLS is that, the transfer matrix of the mask it forms will have large and slowly-decaying singular values, which is beneficial for image reconstruction. However, when the scene is close to the CMOS sensor, the system matrices begin to exhibit difference properties because the PSF only covers a small area. This change can result in reconstructed scenes suffering from nonuniform intensity or low Signal-to-Noise Ratio (SNR). To this end, we propose to generate the optimal mask pattern for such scenario using the genetic algorithm. Since the time consumption of the genetic algorithm largely depends on the time of each generation, we develop a method that directly generates corresponding system matrices from the vector, which significantly accelerates the optimization process.

The objective of calibration is to obtain the system matrices $\bm{P}_{\text{o}}$, $\bm{Q}_{\text{o}}$, $\bm{P}_{\text{c}}$ and $\bm{Q}_{\text{c}}$. In [12], since the MLS vector satisfies $\bm{1}^{\top}\bm{\varphi}=0$, the corresponding columns of $\bm{P}_{\text{o}}$ and $\bm{P}_{\text{c}}$ are orthogonal. Thus, an SVD-based calibration method is adopted.

In our case, because our mask is optimized through the genetic algorithm and does not have such constraint, the calibration algorithm in [12] is not applicable. This section presents our calibration algorithm.

In this subsection, we introduce the method for reconstructing the contact geometry (i.e., the depth map) from the reconstructed tactile images. Further, we adopt a Real2Sim method to transfer the real tactile images into the simulation domain to improve the depth reconstruction quality. Finally, we introduce the method for interpreting marker displacements.

The fabrication of the lensless imaging subsystem has been introduced in Sec. IV. Lensless imaging experiments are conducted before the transparent elastomer is covered with the refractive membrane. The calibration setup is shown in Fig. 8. The calibration images are displayed using a 5.5” 3840$\times$2160 LCD monitor, which is tightly attached to the transparent elastomer. The pixel size of the monitor is 31.5 µm, and therefore the pixel size of the reconstructed image $\delta_{\text{scene}}$ is also 31.5 µm. The resolution of the reconstructed images is determined to be 512$\times$400, which corresponds to an area of 16.1$\times$12.6 mm². The key technical specifications of the lensless imaging system are summarized in Table I.

To generate an optimized mask, the vector $\bm{\varphi}$’s length $K$ is set to 770. We run the genetic algorithm with 2000 generations and 200 individuals in each generation. The crossover and mutation probability are set to be 0.8 and 0.1, respectively. In each mutation operation, 10 random loci in $\bm{\varphi}$ are inverted. To evaluate each individual’s fitness, we choose the pure white image $\bm{X}_{\text{white}}$ (the 1st row in Fig. 9) and the reshaped USAF test image $\bm{X}_{\text{USAF}}$ (the 3rd row in Fig. 9), and the fitness score of gene $\bm{\varphi}$ is calculated as:

Finally, the individual with the highest fitness score is selected. The corresponding optimized mask pattern is shown in Fig. 10. It covers an area of 15.4$\times$15.4 mm², sufficient to overlay the full extent of the CMOS underneath.

For comparison, we choose the MLS-based mask in [13] as the baseline. We use a 255-length MLS and repeat it 3 times to be a 765-length sequence. Then, we use (1) to form a 15.3$\times$15.3 mm² mask (Fig. 10), as the baseline.

The experimental setup is the same as the calibration setup introduced in V-A, as shown in Fig. 8. The ground-truth images are displayed on the monitor, and the measurement images are obtained by averaging 100 frames from the CMOS. SSIM and PSNR are used as the quantitative metrics to evaluate the reconstruction quality.

In Fig. 9, we present the comparison results. Fig. 9 presents three representative scenes. From qualitative comparison and quantitative metrics, we draw the conclusion that our proposed algorithm has largely improved the reconstruction qualities compared with the MLS-based mask, especially in terms of uniformity. Another significant advantage is the larger usable FOV provided by the optimized mask. The reason is as follows: In lensless imaging, a point at the FOV boundary only casts a small PSF area on the CMOS. With the MLS-based mask, the CMOS collects minimal information about these boundary points, resulting in decaying intensities in the reconstructed images’s boundary due to the regularization term in (9). In contrast, the optimized mask takes this factor into account, leading to a larger usable FOV. These advantages will considerably enhance subsequent tactile sensing.

A reconstruction algorithm with high computational efficiency and good quality is crucial in the application of tactile sensing. Therefore, we conduct experiments to compare the proposed non-iterative reconstruction and the traditional optimization-based reconstruction algorithm. Specifically, for the optimization-based reconstruction algorithm, we use Nesterov gradient method to solve (3).

In Fig. 11, we present the comparison results of two representative scenes. From qualitative comparison and quantitative metrics, we draw the conclusion that our proposed algorithm has reconstruction qualities similar to the traditional optimization-based method.

We further compare the reconstruction speed by computing the average time of reconstructing 100 images. For the optimization-based method, we have tuned the parameters so that it only takes 800 iterations to converge. Both algorithms are implemented in PyTorch. We used a NVIDIA RTX 3080 Ti GPU and an Intel Core i7-13700K CPU for the experiment. Table II summarizes the results, which clearly reveals that the proposed algorithm is about 3 orders of magnitude faster than the optimization-based counterpart. Our algorithm can achieve real-time reconstruction up to hundreds of frames per second on a desktop GPU, providing enough efficiency for most robotic tactile applications.

In Table III, we present a comparison of several typical vision-based tactile sensors with straight optical paths. We introduce a dimensionless metric, which is the sensing area divided by the square of the thickness, to quantify the compactness of the sensor. The table considers two types of thickness: the actual physical thickness of the sensor and the scene-to-CMOS distance which signifies the minimal possible thickness. The table reveals that the majority of vision-based tactile sensors possess a thickness exceeding 20 mm. However, ThinTact and the MLA-based sensor manage to maintain a thickness of less than 10 mm. In terms of the compactness metric, both ThinTact and the MLA-based sensor significantly outperform the others. It’s important to note that the CMOS we utilized has a thickness of over 4 mm, which contributes to the relatively larger physical thickness of ThinTact. Nonetheless, when considering the scene-to-CMOS distance, ThinTact still holds an advantage over the MLA-based sensor.

Vision-based tactile sensors are widely recognized for their superior resolution in comparison to tactile sensors using other sensing principles. Typically, the resolution referred to in this context is the lateral resolution, which bears a close relation to the high resolution of the CMOS image sensor. In this study, tactile images are reconstructed from raw measurements. It’s important to note that the actual resolution might differ from the resolution of the reconstructed images. In this subsection, we characterize ThinTact’s lateral resolution from two aspects.

First, we figure out the optical resolution from the standard resolution test chart. As shown in Fig. 12, the minimum spacing of white rectangles that ThinTact can recognize is about 0.18 mm. Second, we measure the actual tactile resolution by testing its ability to identify small depth fluctuations. As Shown in Fig. 12, a screw is pressed against the sensor and its screw thread can be clearly seen. The screw’s major diameter is 1 mm and its pitch is 0.25 mm (Fig. 12). The reason that the tactile resolution is lower than the optical resolution is due to the presence of the reflect membrane on the sensor surface, which degrades the transmission and detection of fine geometry details. Fig. 12 shows the tactile image from GelSight Mini [50], which has a higher resolution than our tactile sensor. This can be attributed to not only the lens system, but also higher camera resolution, softer elastomer, better illumination, and thinner reflect membrane.

As mentioned in Sec. III-E, we employ a Real2Sim transfer method to mitigate the non-ideal factors present in reality and improve the depth reconstruction quality.

First, we follow [14] to collect the unpaired synthetic and real data. We design 8 indenters with various shapes, as depicted in Fig. 13. For the real dataset, the tactile sensor remains fixed, while an indenter is secured on a 4-DoF stage (3 translational DoF and 1 rotational DoF), as shown in Fig. 13. For each indenter, we randomize its position and rotation, and set its indentation depths to 0.25 mm, 0.50 mm, 0.75 mm, and 1.00 mm, respectively. We collect 50×4=200 images per indenter. For the simulation, we generate the surface depth map by intersecting the indenter mesh with the surface mesh. The depth map is then processed using Gaussian filters to approximate the deformation of the elastomer [52]. Subsequently, we employ Phong’s Shading to generate ideal tactile images that perfectly meet the requirements of photometric stereo. These simulated images are then blended with the background of the real images. For each indenter, we also generate 50×4=200 images with 4 indentation depths, with randomized positions that are not aligned with the real dataset. Several real and simulated images are displayed in Fig. 13.

We use 6 indenters for training CycleGAN and the remaining 2 indenters for test. Calibration of the LUT is performed in advance using a 3 mm sphere indenter, and the calibration procedure is performed separately using real images and the Real2Sim images. We present several Real2Sim tactile images in Fig. 13. As can be seen in the figure, real tactile images suffer from a noisy background and non-uniform illumination. In contrast, the Real2Sim images exhibit more idealness than the real ones. We further evaluate our sensor’s depth reconstruction accuracy quantitatively. To calculate the depth error, we generate the ground-truth depth using simulation and apply thresholding to select the area for error calculation. The depth error is calculated by first calculating the root-mean-squared error of all the pixels in the aforementioned area in each image, and then averaging the errors across all the tactile images. The results are summarized in Table IV. The data shows that direct reconstruction from real tactile images results in a large depth error due to non-ideal conditions. In contrast, the Real2Sim method achieves an error of 0.13 mm and improves the depth accuracy by approximately 60%.

To evaluate the capability of ThinTact to capture fine details, we conduct an experiment of fabric texture classification, where we use a robot arm to press the tactile sensor against various fabric materials. A deep neural network is trained to classify the reconstructed tactile images.

We use ResNet-50 image classification network [53], and train the network for 100 epochs with a batch size of 64 and a learning rate of 1e-3.

To demonstrate the sensitivity of ThinTact, we conduct experiments to grasp two delicate objects: a 0.5 mm pencil lead and a potato chip. In the experiment, the fingertips of the parallel gripper gradually closes, and the gripper will immediately stop once a tactile sensor detects contact. Fig. 16 shows that these two delicate objects are successfully grasped and lifted by ThinTact. The corresponding tactile images are also shown in Fig. 16. We use a high-precision force sensor (Rtec MFT-5000) with a resolution of 1.5 mN to measure the force applied when the sensor detects contact. The result shows that the force is 0.16 N when a contact is detected, demonstrating the high sensitivity of our proposed tactile sensor.

Using ThinTact for perception and feedback control, we are able to accomplish tasks such as precisely inserting test tubes into a test tube rack, where the diameter of the tube is 20 mm, the diameter of the hole is 22 mm, the distance between two tubes is only 12 mm, and the clearance of the hole is 1 mm. As shown in Fig.17, we utilize a Robotiq 2F-140 gripper fixed at the end of a ROKAE xMate3-pro robotic arm for grasping. Two ThinTacts are installed on both sides of the gripper.

To simulate the positioning errors from vision perception or human handover, we introduce a random offset to the initial pose of the test tube. When the test tube makes contact with the rack during the placement, we capture the marker displacements from the sensors and fit them to derive the direction of the force exerted on the test tube. Subsequently, we adjust the gripper’s pose in the opposite direction, allowing the test tube to be precisely inserted into the hole. For a detailed demonstration, please refer to the accompanying video.