Verti-Selector: Automatic Curriculum Learning for
Wheeled Mobility on Vertically Challenging Terrain

Off-road mobility is a challenging domain for autonomous robots, as they must navigate complex, unstructured environments with varying terrain conditions. Traditional approaches to off-road navigation often rely on hand-crafted perception [17], planning [18], modeling [19], and control [20] methods with human heuristics. Those classical methods require extensive engineering effort, are affected by cascading errors from upstream modules, and struggle at adapting to new environments [21].

Considering the limitations of classical approaches, learning off-road mobility has emerged as a promising alternative avenue [22], such as learning end-to-end policies [23], semantic perception [24, 25, 26, 27, 28], kinodynamic models [29, 30, 31, 32, 33, 34, 35], parameter adaptation [36, 37, 38, 39, 40], and cost functions [41, 42, 43, 44, 45, 46, 47, 48]. Learning methods, such as RL, can alleviate engineering effort and allow emergent and adaptive behaviors. However, those methods are often data-hungry, either requiring extensive expert demonstration or labeled data for imitation learning or millions of trial-and-error exploration steps using RL. How to generalize learning results to unseen deployment enviornments is also difficult. To tackle those challenges of learning methods, curriculum learning has the potential to improve sample efficiency and generalization by presenting the robot with a sequence of tasks that gradually increase in difficulty. VS is based on RL guided by an efficient curriculum for wheeled mobility on vertically challenging terrain.

Curriculum learning is a concept inspired by the structured nature of human learning [49]. This idea was expanded upon by Bengio et al. [50], who proposed a learning paradigm where training examples are presented in a meaningful order, gradually increasing in complexity. Over the following years, curriculum learning found applications in various supervised learning settings, such as natural language processing [51] and computer vision [52]. Building on these foundational ideas, the community developed a set of mechanisms collectively known as Automatic Curriculum Learning (ACL) [53].

ACL techniques automatically adjust the distribution of training data by selecting learning situations that match the evolving capabilities of the learning agents. While ACL has been successfully applied to various domains, most applications have been limited to simple tasks or simulated environments. For instance, in supervised learning, ACL has been employed to improve performance on static benchmark datasets, such as image classification [50]. Similarly, in RL, ACL has been primarily studied in the context of simple gridworld environments [54] and Atari games [55].

Despite ACL’s potential in improving sample efficiency and asymptotic performance in these simplified settings [56], few works have explored its application to real-world mobility tasks, where robots must learn to navigate complex, unstructured, off-road environments. The challenges posed by off-road terrain require sophisticated approaches to curriculum learning. In this work, we investigate how to automatically select an appropriate task sequence as a curriculum based on real-world data to enhance sample efficiency and policy generalization for wheeled robots navigating vertically challenging terrain, while considering their evolving capabilities.

To create a realistic simulation environment for vertically challenging terrain, we first collect elevation map data [57] using our physical Verti-4-Wheeler (V4W) on a custom-built indoor testbed. This testbed consists of hundreds of randomly distributed and stacked rocks and boulders, with an average size of 30cm, matching the scale of the V4W. The test course measures 3.1$\times$1.3m, with the highest elevation reaching up to 0.5m, more than twice the height of the vehicle (Fig. 1 middle).

Within the Chrono multi-physics simulation engine, we generate a triangular mesh by assigning a vertex to each pixel of the elevation map. The mesh vertices are then vertically adjusted to align with the specified elevation values in the elevation map. Finally, the mesh is scaled to match the given spatial extents (Fig. 1 around). To simulate the interaction between the terrain and the vehicle’s wheels, we set the friction coefficient of the terrain material using Chrono’s built-in material properties. The friction coefficient is set to 0.9, and the restitution coefficient is set to 0.01. These values are carefully chosen to represent a realistic off-road terrain surface, considering factors such as tire grip and surface deformation. By following this methodology, we create a highly accurate and realistic simulation environment that closely mimics the vertically challenging terrain encountered by the V4W in real-world scenarios.

The diversity of PCG environments makes them valuable testbeds for evaluating the robustness and generalization ability of RL agents. To maintain the PCG principle, we assign a fixed identifier (index) to each terrain. We generate a sequence of elevation maps by linearly interpolating between a starting map $I_{0}$ (flat terrain) and an ending map $I_{N}$ (real-world rugged terrain) using a weighted average. The intermediate map $I_{i}$ at index $i$ out of $N+1$ indices can be calculated using the following equation:

The $N+1$ elevation maps generated by PCG serve as individual tasks that, when ordered appropriately, comprise our curriculum to learn wheeled mobility on vertically challenging terrain.

In this work, we formulate the off-road navigation task for wheeled robots as a Markov Decision Process (MDP) characterized by a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\gamma,r)$, where $\mathcal{S}$ represents the state space, $\mathcal{A}$ denotes the action space, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\to\mathbb{P}(\mathcal{S})$ signifies the transition probability function, $\gamma\in[0,1)$ is the discount factor, and $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the reward function. We employ RL to learn a policy $\pi:\mathcal{S}\to\mathcal{A}$ that maps states $s\in\mathcal{S}$ to actions $a\in\mathcal{A}$, enabling the robot to navigate vertically challenging terrain while avoiding pitfalls such as getting stuck or rolling over, ultimately guiding it to reach the designated goal. The objective is to maximize the expected cumulative discounted reward:

where $\tau=(s_{0},a_{0},s_{1},a_{1},\ldots)$ represents a trajectory sampled from the policy $\pi$.

The state space $\mathcal{S}$ includes the angular difference between the vehicle and goal heading, current vehicle velocity, and a low-dimensional representation of the elevation map patch underneath the robot obtained using a Sliced-Wasserstein Autoencoder (SWAE) [58]. The action space $\mathcal{A}$ consists of the desired linear speed and steering angle, which will be tracked by a low-level PID controller. We employ Proximal Policy Optimization (PPO) [59] as the RL algorithm to learn the policy, considering the continuous action space.

The reward function $r$ is designed to incentivize the robot’s progress toward the goal while penalizing immobilization due to excessive roll and pitch angles, as well as timeouts. It consists of three key components: (1) a progress reward that encourages the robot to move toward the goal, (2) an instability penalty that discourages excessive roll and pitch angles, and (3) a timeout penalty that penalizes the robot for not reaching the goal within a specified time limit.

By carefully designing the MDP and reward function, we aim to learn a robust policy that enables a wheeled robot to navigate vertically challenging terrain efficiently and safely. The learned policy is then used as a foundation for our ACL approach, which further enhances RL sample efficiency and the robot’s performance and generalization capabilities.

Verti-Selector (VS), depicted in Fig. 2, maintains a dynamic, non-parametric sampling distribution $q(\mathcal{T}|\mathcal{D}_{\text{train}})$ over the set of PCG-generated training terrain $\mathcal{D}_{\text{train}}$, favoring terrain with higher learning potential. Specifically, throughout the training process, VS updates $q(\mathcal{T}|\mathcal{D}_{\text{train}})$ according to a heuristic score that assigns greater weight to terrain with higher estimated learning potential based on the robot’s past experiences. VS maintains two arrays, $\mathbf{u}\in\mathbb{R}^{|\mathcal{D}_{\text{train}}|}$ and $\mathbf{v}\in\mathbb{N}^{|\mathcal{D}_{\text{train}}|}$, where $u_{i}$ stores the score for terrain $\mathcal{T}_{i}$ and $v_{i}$ keeps track of the episode count at which $\mathcal{T}_{i}$ was last sampled. After each episode, VS updates $q(\mathcal{T}|\mathcal{D}_{\text{train}})$ by computing a mixture of two distributions: $q_{u}(\mathcal{T}|\mathcal{D}_{\text{train}})$, based on the terrain scores, and $q_{v}(\mathcal{T}|\mathcal{D}_{\text{train}})$, based on the elapsed time since each terrain was last sampled:

where $\alpha\in[0,1]$ is a hyperparameter regulating the equilibrium between the two distributions. The mixture distribution ensures that the sampling process considers both the estimated learning potential and the time elapsed since each terrain was last encountered, mitigating the risk of catastrophic forgetting in neural networks.

An overview of the simulation environment is shown in Fig. 3. We generate 100 distinct synthetic terrain for training based on the real-world rock testbed [16] and each terrain has a fixed identifier (index) by the PCG principle. To show the generalization of VS, the test terrain are more uneven than the training terrain, as visualized in Fig. 3. We use a mobile robot with reduced double wishbone suspensions and rack-pinion steering in VW-Chrono. For RL training, an episode terminates if the robot reaches the designated goal or exceeds the maximum time (20s). To learn wheeled mobility on vertically challenging terrain, the following design choices are made:

To evaluate the performance of the learned policy, we deploy our model on the V4W platform ($0.863\text{m}\times 0.249\text{m}\times 0.2\text{m}$, Fig. 1 middle), a four-wheeled vehicle based on an off-the-shelf, two-axle, four-wheel-drive, off-road platform from Traxxas. The onboard computation is handled by an NVIDIA Jetson Xavier NX module. A Microsoft Azure Kinect RGB-D camera produces depth images to construct real-time elevation maps [57]. We use low-gear and lock both front and rear differentials to improve mobility on vertically challenging terrain. For the controlled environment, we shuffle our indoor rock testbed to achieve varying levels of difficulty: easy, medium, and hard. The testbed is designed to mimic vertically challenging terrain encountered in outdoor off-road environments with controllable complexity.

VS is compared against four baseline methods: Optimistic Planner (OP), Naive Planner (NP), Vanilla RL (VR), and a Manually-designed Curriculum (MC) [61].

OP minimizes the angular difference between the vehicle’s current and desired heading, optimistically assuming a flat terrain. However, this assumption often struggles with steep slopes and rugged boulders, leading to suboptimal performance on vertically challenging terrain.

To address the limitations of OP, NP incorporates a heuristic based on the elevation map of the surrounding terrain. This planner divides the $64\times 64$ surrounding elevation map into regions and selects the most traversable direction based on the mean and variance of the elevation values. Although more effective than OP, NP still relies on fixed rules and may not adapt well to diverse terrain conditions.

VR takes a more flexible approach by training the policy on randomly selected terrain from the PCG-produced training set, without any explicit curriculum. While this allows the robot to experience a wide range of terrain conditions, the lack of structure in the training process may lead to suboptimal sample efficiency, as the robot may spend too much time on terrain that is either too simple or too challenging for its current skill level.

MC addresses this issue by following a curriculum that gradually increases the difficulty of the training terrain. The curriculum consists of five stages, each with a specific success rate threshold $\{1,1,0.8,0.6\}$ that the robot must reach before progressing to the next one. However, MC may not always align with the robot’s actual learning progress, potentially leading to inefficiencies in the training process.

Our main findings are that (i) VS with rank prioritization ($\alpha=0.1,\beta=0.1$) significantly improves both sample efficiency during training and generalization on the test terrain, attaining the highest success rate in 50 trials out of all baselines evaluated; (ii) The relatively low average roll and pitch angles indicate that VS learns a stable policy that can effectively handle the uneven test terrain.

We conduct ten trials each on three configurations of our physical testbed (Fig. 5 left) with the V4W, recording the same set of metrics. The results are presented in Table III. The learned VS policy demonstrates a high success rate across all difficulty levels, with a slight decrease in performance as the terrain complexity increases. The average traversal time also shows a consistent trend, with longer times required for more challenging courses. MC fails more on the Medium course and fails all trials on the Hard one. It mostly suffers from longer traversal time and larger roll/pitch angles. These results validate VS’s generalizability from simulation to a real-world vertically challenging testbed.

To further demonstrate the generalizability and applicability of the learned VS policy, we deploy it on the V4W in a real-world outdoor environment. We select a challenging off-road location with diverse terrain features, including steep slopes, various rocks, and uneven surfaces (Fig. 5 right). The platform exhibits stable and efficient navigation by effectively making appropriate steering and throttle decisions based on the perceived outdoor terrain features.