DexTouch: Learning to Seek and Manipulate Objects
with Tactile Dexterity

Our system is a multi degree-of-freedom (DoF) robot with a total of 22-DoF, consisting of a 6-DoF UR-5e and a 16-DoF AllegroHand. The robot hand has a total of 16 FSR sensors, with 3 sensors attached to each finger and 4 sensors on the palm, as illustrated by the white dots in Fig. 1 and as suggested by [6]. Using an STM32F103 microcontroller, the sensor’s voltage signal is measured at a sampling rate of 125 Hz, passed through a low-pass filter to remove noise, and then transmitted to the main controller. Binary signals are used to reduce the gap between simulation and real robots and simplify the Sim2Real transfer procedure. For this purpose, preprocessing is performed to convert the transmitted voltage signal into a binary contact signal according to a selected threshold $\theta_{th}$ before being used.

IssacGym simulator [7] was used for the training of our tactile based robot system. The simulation settings for each task are visualized in Fig. DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity (left). A multi-DoF robotic arm-hand system was implemented in the simulation, identical to a real robot. Sixteen virtual contact sensors are configured in the simulation and $\|F\|$ calculated from the net contact force $F=[F_{x},F_{y},F_{z}]$ measured for each step is used as the contact force. Then, using a selected threshold $\tilde{\theta}_{th}=0.01$ $\mathrm{N}$ the binary contact signal is calculated in the same way as in real-world environments. The control frequency was set to 10 Hz in both real-world environments and simulations.

We study dexterity using a robot’s sense of touch through various types of object manipulation tasks aimed at manipulating objects without vision. In this paper, we mainly focus on three distinct types of benchmark tasks that are relevant to our daily lives as illustrated in Fig. 1. (i) Grasping Object. In this task, inspired by human behavior of retrieving items from a high, invisible shelf, the robot is required to utilize its sense of touch to grasp objects randomly positioned on the table and then bring them to the target point without dropping them. Furthermore, the policy should demonstrate that the robot can also manipulate multiple objects with different shapes. Fifteen objects were selected from the YCB dataset [30], as depicted in the Fig. 2, and experiments were conducted on seven of them in a real-world environment. Three of these are unseen objects. (ii) Door Opening. In this task, the robot needs to locate a door handle that is positioned randomly about the x and y-axes, rotate the door handle, and pull it to open the door. In an invisible environment, the robot should use its sense of touch to locate door handles and learn complex movements to open the door. (iii) Valve Rotating. The purpose of this task is to locate the position of a valve located randomly on a plane and rotate the valve clockwise. Compared to objects in other tasks, the size of the valve is small, requiring more sophisticated exploration ability of the robot.

Our tasks are formulated as a Markov Decision Process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P})$. Here, $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{R}$ is the reward function, and $\mathcal{P}$ is the transition dynamics. The objective of robot agent is to find optimal policy $\pi$ to maximize the $\gamma$ discounted return $\sum_{t=0}^{T}\gamma^{t}r_{t}$. The agent observes state $s_{t}$ at each time step $t$. Then, it take action $a_{t}=\pi(s_{t})$ calculated by policy $\pi$ and receive a reward $r_{t}=\mathcal{R}(s_{t},a_{t},s_{t+1})$. During this process, the agent doesn’t know $\mathcal{R}$ and $\mathcal{P}$. An episode terminates when the agent exceeds the maximum number of steps $T$ or achieves the goal of task or reset conditions are achieved.

For a successful application of reinforcement learning, the density of rewards should be enough to facilitate exploration by the agent. At the same time, the agent should be focused on achieving the goal without being distracted from taking actions that deviate from the final goal. To this end, a reward function is proposed that decomposes the task into two phases: reaching the target object and executing the purposeful manipulation task, as illustrated in Fig. 3. Additionally, inspired by [31], a reward function is designed such that the agent can only receive rewards for movements toward the goal.

Reaching the target object. To encourage the robot to get closer to the target object, the reach reward $r_{reach}$ was designed as follows:

$$r_{reach}=\underset{finger}{\sum}\alpha_{reach}\times\max(d_{closest}-d,0)$$

(1)

This reward is common to all three tasks. Here, $d$ and $d_{closest}$ are both the distance between each fingertip and the target object. The target object is the object on the table in Grasping Object, the door handle in Door Opening, the center position of valve in Valve Rotating. $d$ is the current distance, and $d_{closest}$ is the closest distance achieved during actions so far. At the beginning of the episode, set $d_{closest}=d$. $\alpha_{reach}$ is a relative reward weight.

Executing the manipulation. Since the target manipulation varies for each task, a manipulation reward $r_{execute}$ was designed according to each task. First, $r_{execute}$ for Grasping Object task is designed as follows:

$$r_{execute}=(1-\mathds{1}_{picked})\times\alpha_{pick}\times h_{obj}+r_{picked}\\ +\mathds{1}_{picked}\times\alpha_{goal}\times\max(\tilde{d}_{closest}-\tilde{d},0)$$

(2)

In equation 2, the component $r_{execute}$ rewards the agent for picking up an object, lifting it off the table, and bringing it to the target point. Here, $\mathds{1}_{picked}$ is an indicator function that activates when the height of the object $h_{obj}$ relative to the table becomes 1 above the preset threshold of 10 $\mathrm{cm}$. At this moment the agent receives an additional bonus reward $r_{picked}$. $\tilde{d}$ and $\tilde{d}_{closest}$ represent the distance between the object and the target point, where $\tilde{d}$ is the current distance and $\tilde{d}_{closest}$ is the closest distance achieved during attempt so far. $\alpha_{pick}$ and $\alpha_{goal}$ are relative reward weights.

For Door Opening task, $r_{execute}$ is designed as follows:

	$\displaystyle r_{execute}$	$\displaystyle=(1-\mathds{1}_{rotated})\times\alpha_{rot}\times\max(\phi-\phi_{max},0)$
		$\displaystyle+\mathds{1}_{rotated}\times\alpha_{open}\times\max(\psi-\psi_{max},0)$
		$\displaystyle+r_{rotated}+r_{opened}$		(3)

In equation 3, the component $r_{execute}$ rewards the agent for actions such as rotating the door handle and opening the door. $\mathds{1}_{picked}$ is an indicator function that becomes 1 when the rotation angle of the door handle exceeds the preset threshold of 1.047 $\mathrm{rad}$ (60 $\mathrm{deg}$). $\phi$ and $\phi_{max}$ are the door handle angles, $\phi$ is the current door handle angle, and $\phi_{max}$ is the maximum door handle angle achieved during attempt. Similarly, $\psi$ and $\psi_{max}$ are the door angles, $\psi$ is the current door angle, and $\psi_{max}$ is the maximum door angle achieved. When the door handle is rotated beyond the threshold, reward $r_{rotated}$ is received, and when the door is opened beyond a preset threshold of 0.873 $\mathrm{rad}$ (50 $\mathrm{deg}$), reward $r_{opened}$ is received. $\alpha_{rot}$ and $\alpha_{open}$ are relative reward weights.

In case of Valve Rotating task, $r_{execute}$ is designed as follows:

$\displaystyle r_{execute}$

$\displaystyle=\alpha_{rot}\times\max(\theta-\theta_{max},0)+r_{success}$

(4)

In equation 4, the component $r_{execute}$ rewards the agent for the action of rotating the valve. It is similar to the reward for rotating the door handle in the previous task. When the valve is rotated more than 135 $\mathrm{deg}$, the agent receives a success bonus. $\alpha_{rot}$ is a relative reward weight.

Finally, to prevent unstable jerky movements of the robot, we add a simple adjustable penalty containing the L1 norm of the velocity of each joint to promote smooth movement of the robot agent.

The policy is trained using experiences simulated in IsaacGym [7], a highly parallelized GPU-accelerated physics engine. Both the policy and value networks consist of 3 multi-layer perceptrons (MLPs) with sizes 512, 256, and 128, respectively, and ELU [32] as the activation function. To train the control policy, the Proximal Policy Optimization (PPO) [33] algorithm is employed with the following hyper-parameters: clipping $\epsilon=0.2$, discount factor $\gamma=0.99$ and 0.016 KL threshold. To reduce the difficulty of learning, asymmetric observation is applied to policy and value networks [12]. Specifically, the value network includes privileged information such as the exact pose of the target object, linear and angular velocities, distance from the robot hand, and physical parameters, and includes information for calculating rewards such as indicator functions depending on the task.

For the IsaacGym simulation, 4096 parallel environments were used in $dt=0.01667$ $\mathrm{s}$ with 2 simulation sub-steps. The action generated by the policy network is executed over 6 steps, corresponding to a control frequency of 10Hz in real-world.

In experiments, our method is mainly compared with the following baselines.

• 1)

  WO-Sensor (Without-Sensor). This PPO policy is learned without any tactile information from the robot. It should use only the robot’s proprioception to manipulate the target object during the task.

• 2)

  LQ-Sensor (Low Quality-Sensor). The threshold of the tactile sensor to detect touch was set to 0.3 $\mathrm{N}$. In other words, by reducing the sensor’s sensitivity, a lower-quality tactile sensor is employed to train the policy.

• 3)

  WO-PInfo (Without-Privileged Information). This PPO policy is learned without privileged information such as the exact pose of the target object, linear and angular velocities, distance from the robot hand, and physical parameters, and includes information for calculating rewards such as indicator functions.

• 4)

  DA-Sensor (Deactivation-Sensor). Training is conducted following the same procedure as the proposed method, with the only difference being the deactivation of the tactile sensor during evaluation. This policy is used for ablation purposes and to test the extent to which tactile information is used when applied in a real-world.

In this paper, prior information about the environment is assumed. This information does not provide the exact location of the object, but only the range in which it is likely to exist. Since the height of the furniture or door generally does not change, randomness was applied to the vertical and horizontal distances from the robot, with the height remaining constant. In addition, this was set considering the operating range of the robot because the robot arm is fixed. This randomness was applied consistently in both the simulation and the real-world.

The Fig. 4 shows the success rate and episodic return of each task according to the training process of the proposed method and baselines and Table II shows the learning results of each method. The findings are highlighted as follows:

(i) WO-Sensor, which does not have tactile information, learns by leveraging the location information known in the initial training stage (e.g., position of the table in the case of the grasping object task). However, after that, it fails to learn interactions that involve manipulating the target object to succeed in the task. These results show that it is difficult to detect the position of an object during interaction using a robot’s proprioception alone, and that tactile sensors play an important role in object detection.

(ii) High-quality tactile sensors enhance the success rates of dexterous manipulation tasks. As the sensitivity of the tactile sensor decreases, more force is required for object contact detection. Phenomena such as objects falling or being thrown have been observed during this process. These findings are consistent with previous results showing that highly sensitive tactile sensors aid in reliable object manipulation.

(iii) ‘WO-PInfo’ achieved lower performance than other methods that can utilize privileged information. This is consistent with the general result that using asymmetric observation improves the learning efficiency of reinforcement learning systems. Privileged information is not accessible to the policy network, however it contributes to the efficiency and stability of learning.

Ablation studies were conducted to ascertain the tactile sensor most beneficial for manipulation in the absence of vision. Two groups were set for comparison. One group, Fingertips, activates only the sensors on the fingertips, while the other group, Palm, activates only the sensors attached to the palm. Both groups have four activated sensors each, and the touch detection threshold was set to 0.01 $\mathrm{N}$, the same as ours. These were trained in the same simulation environment and then compared to our policy. The results are shown in Fig. 4 (b) and Table II. The higher success rate of the fingertips group compared to the palm group indicates that fingertip sensors are more advantageous for object manipulation. In particular, this group achieved similar success rates to ours in the Valve Rotating task because the fingertips were primarily used for valve manipulation behavior, minimizing the impact of deactivating other sensors.

Force/Torque (F/T) sensors are widely used sensors in robot manipulation. For this reason, they are often considered a good alternative to tactile sensors. We conducted ablation studies to ascertain which type of sensor is more crucial for object manipulation without vision. A new group using F/T sensors on the robot wrist was constructed (F/Tsensor). The 3-axis force and torque were measured using the wrist-mounted F/T sensor and utilized for learning. F/Tsensor was also trained in the same simulation environment and compared with our policy, Fingertips, and Palm. The results are shown in Fig. 4 and Table II. We observed that F/Tsensor achieved lower success rates compared to methods using touch sensors. This suggests that touch sensors attached to the robot hand are more essential than F/T sensors for navigating and manipulating objects without vision.

We transfer the trained policies to a real robot without fine-tuning to test whether tactile sensors continue to provide benefits for object manipulation without vision. The results are shown in Table III. We further analyze the tactile policy, as shown in Fig. 5, which illustrates the object manipulation process for each task. In particular, the analysis focuses on the transferability of the ability to manipulate randomly located objects.

In all tasks, the initial reaching process was observed to have similar trajectories. This means that in the initial stages, when information is lacking, approaches are based on what is known. However, in subsequent manipulation behaviors, different performance was observed depending on the extent to which touch was used.

Due to the low sensitivity of the LQ-Sensor, more contact force was required for manipulation. As a result, excessive manipulation, such as dropping objects, has been observed. These observations indicate that sensitive touch is important for the manipulation of objects. The lowest success rate of the DA-Sensor indicates a significant contribution of the touch sensor to manipulation. As a result, inital process similar to our policy was performed, but the lack of a touch sensor prevented the object from being found, making subsequent manipulation tasks difficult.

In contrast, our approach leveraging touch provides benefits for object manipulation. In each task, manipulation attempts were observed based on object contact. For example, it has been observed that the timing of attempted manipulation of an object varies depending on when contact occurs. This suggests that the robot can first move to an area where an object may exist based on prior information, and then manipulate the object without vision based on touch. Additionally, these observations highlight that sensitive touch can be useful for blind manipulation of objects.

Prior information serves as the starting point for the robot to recognize the existence of an object and to perform the task. From this perspective, prior information can influence the initial stage of object manipulation. However, since the robot cannot determine the exact location of the object with prior information alone, it must learn to manipulate the object by compensating for the loss of information with tactile sense. In addition, the properties of objects encountered by the robot during learning can also be included in the prior information. We observed that, among the three unknown objects used in the Grasping object, the tumbler had the lowest success rate. This is due to the heavy weight and slippery surface of the tumbler. Therefore, it suggests that properties not encountered during learning can affect the task success.

The results demonstrated that policies trained in simulation can be successfully transferred to real-world environments. Even in a real environment, the policy is capable of manipulating randomly located target objects according to the task. This indicates that blind manipulation using tactile information can be effectively applied in real-world scenarios, even in the absence of visual information.