Visual Tactile Sensor Based Force Estimation
for Position-Force Teleoperation

Fig. 1 shows the position-force architecture, in which tactile sensor DIGIT is used to substitute conventional FT sensor, and $x_{h}$, $x_{l}$, $x_{fd}$, $x_{f}$, $f_{l}$, $f_{ld}$, $f_{s}$, denotes for human command position, leader position, follower desired position, follower position, leader force, leader desired force, measured force from DIGIT sensor, respectively. Here the leader robot is the gripper of the Omega 7 haptic device (Force Dimension) and the follower robot is a parallel robotic gripper by PAL robotics. The data communication is achieved through the ROS framework. The control law of the position-force architecture is given as follows,

	$\displaystyle f_{ld}$	$\displaystyle=f_{s},$		(1)
	$\displaystyle x_{fd}$	$\displaystyle=x_{l}$

where, the leader robot is force controlled to track measured force from the DIGIT sensor $f_{s}$, and the term $x_{fd}$ is an input to the position controller of the follower robotic gripper as a reference position to track. Through this position control, the gripping force can be modulated by teleoperation.

This section explains the details of how the depth maps are extracted from DIGIT RGB images. The depth map is defined as the deformation level of each pixel on the gel surface. The process is shown in Fig. 2. To get the depth map, we first train a three-layer Multi-Layer Perceptron (MLP) that learns a mapping from RGBXY values to surface normal values represented as $n_{x}$,$n_{y}$, and $n_{z}$, respectively. We used the Adam optimizer with a learning rate of 0.001 to minimize the MSELoss. In addition, a dropout rate of 0.05 is included. We collect 40 RGB images by pressing a steel ball on the gel to train the MLP model. The steel ball has a diameter of 6 mm, and theoretically, the hemispherical shape enables us to extract surface normals in all directions [6]. Surface normal image has a linear relationship with the surface gradients as described by the following equations,

$$G_{x}=\frac{n_{x}}{n_{z}},G_{y}=\frac{n_{y}}{n_{z}}$$

(2)

where, $G_{x}$, and $G_{y}$ represent image gradients in the X and Y directions, respectively. By applying this equation, we can obtain the surface gradients from surface normal images. Given the surface gradients, by using a fast Poisson solver with Discrete Sine Transform (DST), we can finally build the depth map of the gel surface, similar to prior work by Wang et al. [16], and Paloma et al [17].

Once the depth map is generated, we select the maximum depth value from the depth map. This maximum depth value corresponds to the maximum surface deformation of the DIGIT gel during contact. Then, by building a regression model to map the gel deformation and ground truth contact force, we can later estimate the contact force in real time. This process is similar to using Young’s modulus to estimate the counter force from an object with respect to surface deformation level.

To acquire the mapping between depth values and force values, we programmed the Omega 7 haptic device to press downwards (Z direction) with a constant displacement at each press. A rigid cylinder shape probe with a 5 mm radius and a curved surface facing down is attached to the tip of the device which is in direct contact with the DIGIT gel. The force value is read from the Omega 7 haptic device, and the depth value is read from the DIGIT sensor. The orange dots in Fig. 3 show the collected force data with respect to depth value when pressing the DIGIT gel. Then we applied a least-square regression with 3 degrees polynomial to fit the data. The 3 degrees polynomial is defined as

$$p(x)=p_{1}x^{3}+p_{2}x^{2}+p_{3}x+p_{4}$$

(3)

where $p_{n}$ denotes for coefficients. Once we get $p(x)$, we can use it for real-time force estimation. The result of polynomial fit is shown by a blue line in Fig. 3, where the R-square measurement is 0.9987. The estimated force is then published as a ROS topic with a 30HZ refreshing rate which is the same as the camera’s Frames Per Second (FPS).

A rigid object contact experiment is conducted to show the force feedback capability. In the experiment, an operator controls the leader device to grasp a rigid object once without force feedback and once with force feedback. We treat the user’s hand position $x_{h}$ as the desired position. As Fig. 4 shows, in the beginning, the operator moves freely, then in the region (a) the operator has contact with a rigid object without any force feedback, which leads to a large deviation from the desired position. In region (b), the operator contacts the rigid object with force feedback from the DIGIT sensor. The mean value of the desired position in region (b) is 0.0121 m, and the mean value of the actual position is 0.0127 m. Hence, the mean error is 0.0006 m. The operator could maintain surface contact without large positional deviation and can feel the rigid object’s location. In the free space movement, due to the nature of position-force architecture, no follower dynamics are felt by the operator.

A preliminary experiment of teleoperated in-hand pivoting is carried out to evaluate the performance of the proposed force feedback architecture. In-hand pivoting is an important dexterous manipulation skill in robotics. Through in-hand pivoting, a robotic grasping system can perform repositioning tasks and hence compensate for environmental uncertainties and imprecise motion execution [18]. Humans rely on haptic sensations to perform this dexterous task, and in this preliminary experiment, we evaluate the haptic feedback performance of the proposed architecture for such a teleoperated dexterous task.

In this experiment, as Fig. 5 shows, the experiment task is to pivot a grasped cylinder marker from a horizontal position (Fig. 5a) to align with the bottom left corner of the DIGIT sensor (Fig. 5c). The surface friction can be controlled by controlling the distance between the gripper fingers, i.e. wider finger distance reduces friction hence the object will pivot because of gravity. Throughout the experiment, visual feedback with a fixed point of view is given (Fig. 5). We evaluated two conditions, Visual + Force feedback and Visual feedback. Three subjects participated in the experiment. During the experiment, subjects pivot the object 5 times for each condition. Task completion time and success rate are measured.

The experimental result is shown in Fig. 6. The blue and red plot represents the box chart of task completion time for the Visual + Force feedback condition and the Visual feedback condition, respectively. The median value of the visual + force feedback is 13.75, and the visual feedback condition is 15.37. The task success rate is 86.7% and 40.0%, respectively. The experimental results indicate that by using force feedback, subjects complete the pivoting with a higher success rate and slightly faster median completion time. However, in visual + force feedback, some trials required a higher completion time. This may be due to the subjects needing more time to adjust the grasping according to the perceived force, which requires further analysis in the future. The results also imply that the proposed position-force system can display the force feedback for operators, and proves the potential of tactile sensor DIGIT for such a force feedback application.