Snake Robot with Tactile Perception
Navigates on Large-scale Challenging Terrain

An MDP is a 4-tuple $M=\left\langle S,A,P,R\right\rangle$ where $S$ is the set of states, $A$ is the set of actions, $P\left(s_{t+1}|s_{t},a_{t}\right)$ is the transition probability that action $a$ in state $s$ at time $t$ that will lead to state $s$ at time $t+1$, $R\left(a_{t},s_{t}\right)$ is the distribution of reward when taking action $a_{t}$ in state $s_{t}$. A policy $\pi\left(a_{t}|s_{t}\right)$ is defined as the probability distribution of choosing action $a_{t}$ given state $s_{t}$. The learning goal is to find a policy $\pi^{*}$ that maximizes the accumulated reward in given horizon $T$, $\pi^{*}=\underset{\pi}{\textup{argmax}}\underset{a_{t},s_{t}\sim\pi}{\mathbb{E}}\left[\sum_{t=0}^{T-1}\gamma^{t}\cdot R\left(a_{t},s_{t}\right)\right]$, where $\gamma$ is discount factor. RL algorithms are common choices to solve MDP problems.

CPG is a neural circuit in the vertebrate spinal cord that generates coordinated rhythmic output signals to control robot locomotion. CPG-based control methods have been successfully applied to many kinds of robots, such as multi-legged robots [23, 24, 25] or snake robots [liu2020learning, 14, 26]. Usually, to improve the terrain adaptability of CPG, optimization algorithms are often applied to adjust CPG parameters in real-time. As multiple CPG structures have been proposed, we adopted the structure in [14]. The dynamics of CPG are shown in Equation 1-3.

$$\begin{split}\dot{\varphi}=\omega+\textbf{A}\cdot\varphi+\textbf{B}\cdot\theta\\ \ddot{r}=a\cdot\left[\frac{a}{4}\left(R-r\right)-\dot{r}\right]\\ x=r\cdot sin\left(\varphi\right)+\delta\end{split}$$

(1)

$$\footnotesize\textbf{A}=\begin{bmatrix}-\mu_{1}&\mu_{1}&&&\\ \mu_{2}&-2\mu_{2}&\mu_{2}&&\\ &&\ddots&&\\ &&\mu_{n-1}&-2\mu_{n-1}&\mu_{n-1}\\ &&&\mu_{n}&-\mu_{n}\\ \end{bmatrix}$$

(2)

$$\footnotesize\textbf{B}=\begin{bmatrix}1&&&&\\ -1&1&&&\\ &-1&\ddots&&\\ &&\ddots&1&\\ &&&-1&1\\ &&&&-1\\ \end{bmatrix}$$

(3)

$\varphi\in\mathbb{R}^{n}$ and $r\in\mathbb{R}^{n}$ are internal states of CPG, $n$ is the number of output channels, typically the number of robot joints. $a$ and $\mu_{i}$ are hyperparameters that control the convergence rate. $R\in\mathbb{R}^{n}$, $\omega\in\mathbb{R}^{n}$, $\theta\in\mathbb{R}^{n-1}$, $\delta\in\mathbb{R}^{n}$ are inputs that control the desired amplitude, frequency, phase shift and offset. $x\in\mathbb{R}^{n}$ is the output sinusoidal waves of $n$ channels.

The Crater Observing Bio-inspired Rolling Articulator (COBRA), shown in Fig. 3, is designed for space exploration [27]. It consists of 11 actuated joints. The head module of the robot contains the onboard computing system, a radio antenna for communicating with a lunar orbiter, and an inertial measurement unit (IMU) for navigation. At the tail end, there is an interchangeable payload module containing a neutron spectrometer to detect water ice. The rest of the system consists of identical 1-DoF joint modules inter-placed perpendicularly containing a joint actuator and a battery. 207 virtual tactile sensors measuring normal pressure forces are evenly placed throughout the robot body.

Given the relatively straightforward solution of global navigation and gait controller, we omit their implementation details. The novelty of this study lies in the development of a new reinforcement-learning paradigm at the local-navigation level to govern the locomotion of the snake robot. The key problem rests in effectively exploiting the whole-body tactile sensing information to regulate the robot’s gait for enhanced terrain adaptability. Our control scheme design adheres to four guiding principles: (1) Individual joint control; (2) using a pre-trained gait library built from curriculum learning; (3) joint gaits solely depend on local tactile signals; (4) the application of Centralized Training and Decentralized Execution (CTDE) to mitigate partial observability and improve learning efficiency.

Snake robots adopt distinct gaits for efficient locomotion on various terrains. For instance, sidewinding is often used on slopes, while lateral rolling is preferable on smoother surfaces. Inspired by this, we train agents across a spectrum of randomly generated distinctive curriculum terrains (Fig. 6). Each agent contains a Central Pattern Generator (CPG) module, with the actor’s outputs tuning the parameters of the corresponding CPG module. By CPG parameter adjustment, the agents generate optimal gaits pertinent to their curriculum terrains. The training terrains are generated by Perlin noise of size $16m\times 16m$, where the robot learns to navigate to reach a random goal pose from any start location in each episode. The corresponding MDP is defined as:

State space: The state space includes the robot state part and tactile readings part. The robot state part consists of the joint positions $\mathbb{R}^{n}$, IMU readings $\mathbb{R}^{3}$, spatial translation between robot frame and goal pose frame $\mathbb{R}^{3}$ and relative rotation parameterized by axis-angle system $\mathbb{R}^{4}$, i.e., 21 dimensions in total. We only use ego-centric observations from the robot, so a motion capture system is not required as in [28, 29, 30], which makes our system more practical in outdoor environments.

Action space: The action space outputs the CPG parameters, including the desired amplitude $R\in\mathbb{R}^{n}$, frequency $\omega\in\mathbb{R}^{n}$, phase shift $\theta\in\mathbb{R}^{n-1}$ and offset $\delta\in\mathbb{R}^{n}$.

Through curriculum learning as discussed earlier, we obtain a set of agents adapted to various types of terrains, achieving specific gaits by modulating the parameters of the CPG modules. The actors of all acquired agents constitute a gait library, as illustrated in the left side of Fig. 7(a), represented by the yellow and green boxes. Importantly, it should be noted that the training process of these gait libraries do not involve tactile information. Our experimental findings reveal that incorporating tactile information simply by adding it to the state space of a single agent does not yield effective terrain-adaptive gaits. Hence, we devised an approach to incorporate the tactile information as in Fig. 7(a), during a second phase of training (after the first phase of curriculum learning).

For each joint, we introduce an Adaptor, which takes as input the localized tactile information from adjacent links on the body, recognizes terrain features, and subsequently selects a gait output from the library in a one-hot manner. We use SAC with discrete action space to train the Adaptors, keeping the weights and biases of the Actors fixed during this training phase. We train on multiple new terrains not in the curriculum to improve the robot’s terrain adaptation capabilities. In this second phase, the state space is the recent tactile readings gathered from the past one second, and the action space is the one-hot gait selection signal, and the reward is unchanged. Since the basic gaits were already learned during the curriculum learning phase, there was no need to learn gaits from scratch in this phase.

Each CPG module outputs a target joint value $\mathbf{q}_{i}\in\mathbb{R}^{n}$, $i\in\left\{1,\cdots m\right\}$, where $m$ is the number of CPGs and $n$ is the number of joints (channels). For each joint $j\in\left\{1,\cdots,n\right\}$, its target joint value shall be chosen as the $j$-th channel from one of the candidate joint values from $m$ CPG outputs. This choice is determined by the Adaptors and becomes the final target joint value to execute (Fig. 7(b)).

This formulation of localized Adaptors relies on the assumption that gait adjustments are locally dependent on tactile signals, with limited reliance on distant tactile signals. For instance, the motion of a robot’s head exhibits negligible correlation with the tactile feedback at its tail. Such framework draws inspiration from the Centralized Training and Decentralized Execution (CTDE) learning paradigm [32, 33] within the context of multi-agent reinforcement learning (MARL). In this analogy, akin to our Adaptors, each agent exclusively bases its decision-making process on a subset of the global observation. This configuration eliminates the redundant inter-dependencies among agents and reduces model dimensions without degrading task performance.

An intriguing observation is that when Adaptors use a softmax-based output instead of a one-hot one, the weighted mixture of gaits from the library as the final gait did not yield effective performance. The Adaptors will converge toward the average of all gaits in the library, completely neglecting the tactile information. Introducing entropy as an additional loss term could circumvent this averaging tendency but simultaneously introduces computational instability. Hence we used SAC with discrete action space to output a hard-max (one-hot) gait selection.

Due to the introduction of tactile sensors, simulation becomes slow and does not scale well to the substantial amount of experience required in RL. Table I illustrates the operational efficiency of several commonly used robot simulators concerning various numbers of tactile sensors. It can be observed that as the number of sensors increases, there is a noticeable decline in the simulator’s efficiency, as manifested by the maximum real-time acceleration achievable by the simulator, denoted as the Real-Time Factor (RTF). Therefore, we developed a distributed RL framework deployable across multiple workstations (Fig. 8) to mitigate this situation. One of these workstations serves as a server, with an agent comprising a critic and an actor (gait library and Adaptors), along with a centralized replay buffer to store experiences. The other workstations run multiple simulator instances (workers), each instance containing only one agent interacting with the environment. The experiences gained by the workers are transmitted to the server via the TCP/IP protocol, and agent training is exclusively conducted at the server end. The server periodically synchronizes the actors to each worker. Notably, each Adaptor only receives a local tactile pattern, recorded from two adjacent links of a joint, while the critic receives global tactile patterns from all sensors across the entire body. The neural networks architectures we use are shown in Fig. 9.

The training curves for the two phases of our designed algorithms are shown in Fig. 12. On the left are the results of 6 curriculum learning on different terrains, during which the robot learns basic gaits without the use of tactile perception. On the right are the results of our terrain adaptation method (decSAC) trained on 6 terrains beyond the curriculum learning. It can be observed that at the beginning of the second phase, due to change in terrains, the gaits learned in the first phase are not readily adaptable to the new environment. However, after training, our algorithm demonstrated performance similar to curriculum learning on various new terrains. Additionally, we observed that there is little difference in the final performance between centralized and decentralized Adaptors (SAC vs decSAC), thus demonstrating the feasibility of training using MARL. For the method that does not use tactile information but relies solely on domain randomization (DR), it can be observed that learning is possible to some extent, but there are performance bottlenecks. Furthermore, we can see that directly incorporating tactile information as part of the state space (Tac) yields ineffective results. All the results are averaged from 10 independent trials.

Analysis of terrain adaptability can be referenced in TABLE II, where M1-M6 represent the 6 models in the curriculum training, and T1-T6 correspond to the matching training terrains for M1-M6. As observed, the diagonal shape in the table indicates that M1-M6 only perform well in their respective training scenarios but are hard to adapt to untrained environments. T7 and T8 are two entirely new test environments beyond the two training phases. It can be seen that neither M1-M6 nor DR can perform well in the new environments, whereas our approach is capable of extracting terrain characteristics from tactile information and adopting adaptive gaits.

We compared the results of several baselines in navigating through the five caves (Fig. 11). The comparison of their runtime is shown in Fig.13. The action space of method ”RJ” is the robot’s target joint angles [17], while the action space of method ”CPG” consists of parameters for the CPG modules. The ”DR” method introduces Domain Randomization on top of CPG. The baselines in the figure do not utilize tactile information. It can be observed that our method achieved the most efficient navigation results. We found that similar to the Tac results in Fig. 12, directly incorporating tactile information into the state space, regardless of using RJ, CPG, or DR in the action space, failed to complete the navigation task within a reasonable timeframe, and therefore, the results are not depicted in the figure. We speculate that the reason might be the inherent difficulty of simultaneously learning both gait and terrain adaptability from scratch. In contrast, our approach, through curriculum learning, divides the training into two phases, each focusing on learning gait and terrain adaptability, respectively. This approach simplifies the problem by decoupling the two tasks.

The centroid motion trajectory of the robot in one of the caves is shown in Fig. 14, and it can be observed that the centroid motion trajectory closely aligns with the path planned by the high-level controller. By observing the robot’s motion at a closer distance, we found that when tactile information is not utilized, i.e., RL directly determines the parameters of the CPG module based on the robot’s state, the terrain adaptability is compromised. As shown in Fig. 15, when the robot uses the sidewinding gait on an uphill without the slope information, it is prone to turning over. We provide a detailed demonstration of this comparison in the supplementary video, showcasing the motion of various baselines within the caves.