VINet: Visual and Inertial-based Terrain Classification and Adaptive Navigation over Unknown Terrain

In [14], Xue et al. proposed an orderless-texture-based and spatial information-targeted neural network. [15] used both 3D LiDAR and camera to classify terrain by computing a 3D-scan feature that overcomes the issue in different lighting conditions. Similarly, Nguyen [16] introduced a structure-based method as well as estimating the surface smoothness to classify different roughness surface objects. [17] proposed an angular imaging method to distinguish different material types.

In addition, some methods try to solve the classification problem with inertial sensing. The earlier method like [18] utilizes a dynamic model to estimate the terrain profile and further label the terrain based on the spatial frequency part of the profile predicted. Likewise, [19] adopts the same frequency analysis to classify the terrain by filtering out the terrain impulses from the identified terrain profile. [20] tries to reveal the terrain types using the inertial data collected from the human walk. Recent work [21] proposes several deep learning methods for classifying the time-series IMU data on top of the engineered features.

Indoor navigation has been well developed in the past decade [22, 23, 24], but it still remains a challenge in the outdoor environment [25, 26, 27], especially on different terrains [28, 29]. Kumar et al. [28] investigate adaptive control strategies to enable a quadruped robot to walk on different material terrains, such as sand, mud, grass, and dirt. The wheeled robot or tracked robot [30] is another type of platform that is used in terrain navigation. [31, 32] and [33] cover the topics of unstructured environment modeling and controller design from the angle of the tracked robot. As mentioned in [28], the adaptive control policy enables the legged robot to quickly recover stability from falling, which poses a key factor towards the success of navigating in dynamics-changing environments of different terrain. Such related learning-based adaptive controller methods have been discussed in [34, 35, 36].

The inputs of the network consist of an image patch $I_{patch}\in\mathbb{R}^{3\times l\times l}$ taken along the intended trajectory of the mobile robot from the RGB camera, a sequence of 6 dimensional IMU data $I_{imu}(t_{s})=[a_{x},a_{y},a_{z},w_{x},w_{y},w_{z}](t_{s})\in\mathbb{R}^{6},t_{s}=\{t_{0},...,t_{0}+D\}$ with three linear accelerations and three angular velocities for each axis, and a pulse-width modulation (PWM) signal $S(t_{s})\in[0,1]^{m}$, where $m$ is the number of motors relating to navigation and each dimension represents the command sent to a specific motor. The task of our method is to utilize visual and inertial perception sensors on a mobile robot for terrain classification. Our model will output a terrain category $t\in T=\{t|t=0,...,n-1\}$, where $n$ is the number of terrain types. In our setting, we only focus on surfaces that are navigable with different textures and physical properties and try to reduce the trajectory-following errors during navigation. Avoiding non-traversable surfaces like water or large rocks is not the point of the paper.

let $T$ be the set of known terrains, and the set of unknown terrains $T_{u}$ be the complement of $T$. Given several pre-defined trajectories on surface $i$, we define $MSE_{i}(p)$ as the mean squared error of the controller $p$ trying to follow those trajectories on surface $i$. From the set of known surfaces $T$, we can obtain a set of trained controller $P=\{p_{i}|i=0,...,k-1\}$, and we have

$$\arg\min_{x}MSE_{i}(p_{x})=i,$$

where $p_{i}$ is the controller pre-trained on the known surface $i\in T$. The ground truth category $c_{i}$ of surface $i$ is defined as follows:

$c_{i}=\begin{cases}i&if\ i\in T\\ \underset{x}{\arg\min}\ MSE_{i}(p_{x})&if\ i\in T_{u}\\ \end{cases}$

(1)

In our setting, known terrains are surfaces on which our model has training data, so the perception model and nonlinear model predictive controller (NMPC) can be trained or fine-tuned on those surfaces; unknown terrains are ones that our model has never encountered during training. The ground truth categories of the terrain are defined by the characteristics of the surfaces during navigation, or specifically, by the type of controllers that has the best performance on such terrain. For known terrain, we collected image, IMU, state of the tracked robot, planned trajectories, and actual trajectories to train the perception model and controller; for unknown terrain, we collected perception data as well as the performance of each controller pre-trained on the known surfaces according to definition 1, for the purpose of evaluation.

The proposed VINet consists of one image stream and one IMU stream, as shown in Figure 2. For the image stream, we adopted EfficientNet-B0 [37] as our image feature extractor. Given an image patch $I_{patch}$, the image branch takes a series of convolution operations with kernel size of 3 and 5, and eventually down-sampled to 1280 channels with spatial resolution of $7\times 7$. After flattening and two linear layers with batch normalization and drop-out, we can obtain an intermediate feature $F_{img}\in\mathbb{R}^{d}$, where $d$ is the shared dimension between $F_{img}$ and IMU feature $F_{imu}$.

Considering the effect of actions on the IMU pattern, our IMU stream takes an IMU sequence $I_{imu}$ and the corresponding PWM signal $S$ for some time duration $D$. Given the raw input sequence $H=[I_{imu};S]\in\mathbb{R}^{D\times(6+m)}$, the output feature $F_{attn}$ is computed as:

$$F_{attn}=softmax(k(H)^{T}\cdot q(H))^{T}\cdot v(H),$$

(2)

where $k(H),\ q(H),\ v(H)$ represent the key, query, and value feature, respectively, as in the self-attention [38] literature. Before the features are passed into the classification head, we process the feature $F_{attn}$ with the IMU denoising module.

Due to the noisy nature of IMU data, we introduce an attention-based denoising method to sample useful features in the time dimension. The attention map $A_{map}\in\mathbb{R}^{D\times D}$ is defined as $softmax(k(H)^{T}\cdot q(H))$, and we can calculate the denoised IMU feature $F_{d}$ as:

$$F_{d}=F_{attn}[\ \underset{i}{argsort}(-A_{map}[i][i])[0:f_{s}\times D]\ ],$$

(3)

where $f_{s}$ is the sampling rate, and $argsort$ sort the value in increasing order and returns the corresponding indices. This step removes the features at some specific timestamp with a low attention value for denoising purposes. After passing through a few linear layers with batch normalization and flattening, we can obtain an IMU feature $F_{imu}\in\mathbb{R}^{d}$.

After going through the image and IMU stream, we can obtain $F_{img}$ and $F_{imu}$ with dimension $d$. We used a learnable weight for each channel of the features to fuse $F_{img}$ and $F_{imu}$. After several fully connected layers and activation, the network will output the vector with a classification score.

In the classification head, we want to train the network to learn a shared embedding space for $F_{img}$ and $F_{imu}$, which is achieved by adding a similarity constraint in the network during training. The purpose of creating a shared embedding space is to alleviate the effect of one feature taking over the other. In addition, we want to add some robustness to the network: if the data from one branch is corrupted and produced a terrible feature, it would not significantly affect and corrupt the classification head because the other branch could have some effect of averaging the corrupted features back to the shared embedding space.

In this subsection, we give more details of our controller and present our adaptive scheduling control framework with VINet on known and unknown terrain.

The output of the NMPC, $u$, is then mapped to the PWM signals $S$ controlling the two motors driving each track of the robot. The terrain track interaction, caused by terrain resistance and deformation, is captured by the GP model: $S=gp\left(u\right)$. The GP models provide the required $S$ for each terrain, so that the actual linear and angular speeds match with the desired $u$ calculated by the NMPC. Since $S$ is two-dimensional, corresponding to the left and right track motors, two one-dimensional GP models are fitted for each known terrain. The Radial Basis Function (RBF) kernel is used in the GP models:

$$\kappa(u_{i},u_{j})=\sigma^{2}_{f}exp\left(-\frac{1}{2}\left(u_{i}-u_{j}\right)^{T}L^{-2}\left(u_{i}-u_{j}\right)\right)+\sigma^{2}_{n}$$

(8)

where $L$ is the diagonal length scale matrix and $\sigma_{f}$, $\sigma_{n}$ represent the data and prior noise variance, and $u_{i}$, $u_{j}$ represent data features.

For unknown terrain, our classifier would still choose one of the categories in the original label set since our classifier is only trained on known surfaces. Because of the introduction of IMU as inertial feedback and the terrain label assignment measured by the navigation feedback from the controller, VINet can make a prediction based more on generalized navigation features instead of purely semantic labels, leading to a better choice of the controller on an unknown surface. For evaluation of perception, we find a few unknown terrains and test our pre-trained controller to assign those terrains to one of our pre-trained controllers, as defined in Equation 1.

While the image-based classifier can achieve a good result on standardized classification benchmarks, it does not directly translate to good performance in navigation. First, the categories in standard data sets are usually semantic-based; on the other hand, it’s less intuitive to give a fair traversability or material label that is useful for navigation, and most existing practices are usually heuristic-based [12, 13] for benchmarking purposes.

Second, a purely vision-based method could easily overfit the known data and makes it harder to generalize to more unknown terrains. For example, grass and artificial grass might be visually similar, but they might have different hardness and slipperiness. Because of such difference, the controller fitted with data navigating over grass might lead to worse navigation on the artificial grass because the classifier could not distinguish their navigation features by only relying on the visual difference.

To solve those issues, we eliminate the heuristics in the labeling by using some pre-trained controllers as labels for the terrain, on which the controller has the best performance. The labeling becomes navigation-based instead of semantic-based: once our model is trained according to this labeling scheme and the model can predict correctly, the output would choose the best controller to use while navigating on the corresponding surface, which could directly benefit the navigation. Through navigation-based labeling, the unknown terrain will be predicted as one of the known terrains and the interpretation is that those two types of terrains could have similar navigation features due to some visual textures. The output could still be random and meaningless sometimes unless we have some feedback during the navigation. To improve the Interpretability and generalizability of the output, we add an IMU sequence as input during navigation, in addition to the visual input.

By introducing the IMU, we want to improve the model’s ability to generalize on unseen terrains. The model can build connections between the extracted features from inertial sensing of IMU and the navigation features of the terrain, so the network can have less bias due to over-fitting on the images. Because all of the terrain labels are based on navigation data from known terrains, the prediction output is optimized with respect to the IMU and Image input. In Section IV, we validate our statement by showing comparisons with and without IMU as input to the network.

We collected 4923 images of size $1280\times 720$ among 11 different terrain types and the corresponding inertial sequences at 30 Hz. The 11 terrain types include black carpet, green carpet, smooth tiles, concrete, grass, soil, mulch, wood rocks, rocky surface, asphalt, and small rocks.

Gaussian Process models used in the tracking controller are fitted by the collected data on each known terrain, including the PWM signal $S$ and the corresponding robot’s linear and angular speeds $u=[v,\omega]^{T}$ calculated based on the robot localization output. For each terrain, about 1500 data pairs $\{S,u\}$ were collected and used to train the GP model. Then, the NMPC controller is tuned manually on each terrain type to further optimize the trajectory tracking performance on each known type. For the purpose of evaluation, we find some other surfaces as unknown terrains and evaluate those pre-trained controllers on such terrains to get the best controller as its navigation-based label. Due to limited space, we show the details of each terrain class and evaluated trajectories in the full report [39].

We evaluate our network with the collected data as described in IV-A. We use the Adam optimizer with a learning rate of 0.0003 and exponential decay of 0.95. All experiments are conducted on a single GeForce RTX 2080 Ti GPU and trained with a batch size of 128 for 200 epochs. We sample a patch of size $128\times 128$ along the expected trajectory based on the current velocity and heading, and extract the corresponding IMU while the robot navigates near that image patch region. For all experiments, we set the length of the IMU sequence $D$ to be 60, and the sampling rate of the IMU sequence $f_{s}$ to be $5/6$. We choose the dimension of the shared embedding space to be 512 and 256 in our experiments. For evaluation, we use the averaged accuracy, which is calculated as:

$$Acc=\frac{\sum_{i}1(T(I_{img},I_{imu})=c)}{\sum_{i}1}$$

(9)

In this section, we show some quantitative comparisons under different settings. For comparison methods, we used a recent method [40], which is based on SVM with principle component analysis. We implement this method according to the paper since code is not available.

We use a tracked robot to evaluate the performance of the method, as shown in Figure 3. The external dimension of the robot is $390\times 270\times 145\ mm$, and the track width is $40\ mm$. The tracked robot has a 12V DC motor with 5 kg-cm rated torque specs on each side of the robot and it can carry a maximum payload of 30kg. We equipped the robot with a NVIDIA Jetson AGX Orin for processing, a single-line LiDAR (RPLiDAR-A3) for navigation, a 1080p camera (Logitech C920) for terrain image capturing, and an IMU (3DM-GX5-25) for motion data recording. All computations including data processing, inference and control are computed on the edge.

As shown in Figure 3, the adaptive tracking controller receives the terrain type from the VINet and switches to the corresponding controller in real-time. We conducted multiple test cases with the adaptive controller with the same trajectories covering three different terrains, including mulch, soil as known surfaces, and artificial grass as unknown surfaces, as shown in Figure 4. For comparison, the baseline controller does not receive the terrain type output from VINet and thus uses only one controller on all three types of terrains. In the test, we choose the carpet and mulch controllers as the baseline controllers. Figure 4 shows a sample trajectory tracking test case with the adaptive controller.

The tracking error of the adaptive and the baseline controllers are shown in Table IV. Our proposed VINet-based adaptive control framework has the best performance in terms of the root-mean-square error (RMSE) between the actual and target trajectories.