Achieving Stable High-Speed Locomotion for Humanoid Robots with Deep Reinforcement Learning

In recent years, there has been a significant increase in research on legged robot locomotion utilizing DRL methods. Given the complex structure and high manufacturing costs of humanoid robots, conducting trial-and-error experiments directly in the real world is both expensive and risky. Compared to quadrupedal robots, humanoid robots face additional challenges due to their higher center-of-mass (CoM) and greater number of joints, which complicates DRL-based control.

Radosavovic et al. [6] proposed a learning-based approach for controlling humanoid robots using a causal transformer model that predicts the robot’s next action based on its history of proprioceptive observations and actions. Zhang et al. [11] tackled the complexity of humanoid locomotion by introducing an adversarial motion prior-based imitation learning framework. Cheng et al. [5] developed an approach that enhances upper-body control by closely following reference motions, while relaxing constraints on the lower body to improve robustness and adaptability. Taylor et al. [12] presented a DRL method to teach bipedal robots human-like movements directly from motion capture data, bridging the gap between simulation and reality. Their method incorporates motion re-targeting during training and employs domain randomization techniques to address differences in joint configurations and physical systems.

However, to the best of our knowledge, existing methods have not adequately addressed the impact of upper-body movements on whole-body kinodynamics in achieving accurate and stable locomotion control in humanoid robots. Furthermore, the ability to achieve precise high-velocity tracking in humanoid robot locomotion has not yet been reported. Drawing inspiration from Raibert [10], we analyze how humans maintain stability while moving at high speeds and propose a DRL method that incorporates kinodynamic priors for humanoid robot locomotion control, utilizing only a limited number of joints.

The angular momentum-based control method was introduced decades ago to enhance walking efficiency and performance, specifically by minimizing angular momentum around the CoM during the support phase [10]. Although methods such as DRL and MPC have become mainstream, it is still valuable to consider real-world physical effects. Several studies have analyzed angular momentum and other physical quantities during human motion from a biomechanical perspective [13, 14, 15]. Following these biomechanical studies, some methods combining MPC with angular momentum analysis have been developed. Kojio et al. [16] proposed a gait modification method that considers not only the position of the steps but also their timing and angular momentum. Specifically, this method represented the walkable area as a convex hull and calculated the robot’s foot placement, step timing, and angular momentum for each step. Ding et al. [17] introduced a Nonlinear Model Predictive Control (NMPC) framework integrating multiple balance strategies, including step location adjustment, CoM height variation, and angular momentum adaptation, for maintaining stability under dynamic disturbances. However, methods that combine the angular momentum-based control method with DRL have not been explored.

The primary objective of this work is to achieve a precise control of locomotion velocity in humanoid robots, particularly in high-speed scenarios. Specifically, we aim to find an optimal control policy $\pi^{\ast}$ that minimizes the difference between the robot’s actual velocity and the commanded velocity, as defined by (1). Here, $v^{\pi}$ represents the velocity of the humanoid robot under the control of policy $\pi$, and $v^{\text{cmd}}$ is the commanded velocity.

This problem can be formulated as a sequential decision-making problem for achieving the precise velocity control. At time $t$, the input $x_{t}\in\mathcal{X}$ of the agent contains the current and history information of the humanoid robot, such as periodic clock signals, proprioceptive sensor data, and commanded velocities. The action $a_{t}\in\mathcal{A}$ refers to the target joint positions of the robot. Given $x_{t}$, the humanoid robot executes the action $a_{t}$ based on the current control policy $\pi$. Upon receiving a new observation at time $t+1$, the agent incorporates this information into the history, updates the input to $x_{t+1}$, and receives a reward $r_{t}$ from the reward function.

We employ DRL to train a deep neural network (DNN), $\pi_{\theta}$, to approximate the optimal control policy $\pi^{\ast}$, where $\theta$ represents the parameters of the model. These parameters are optimized by maximizing the expected total rewards during training. The DRL algorithm used in this study is Proximal Policy Optimization (PPO) [18], chosen for its efficiency and suitability for parallel computation.

Humanoid robots have numerous movable joints, which, if left unrestricted, can often result in instability during motion. In kinodynamics, maintaining stability requires minimizing the overall angular momentum of the robot to prevent such instability. Rather than fixing the upper-body joints, we encourage their use to counterbalance the angular momentum generated by the legs. Specifically, we allow the shoulder pitch and elbow pitch joints of XBot-L, as shown in Fig. 2, to generate angular momentum that counteracts the leg swing.

According to the kinodynamics of humanoid robot locomotion, the angular momentum generated by the swinging motion of the robot’s legs is significant and cannot be ignored. Moreover, this angular momentum increases with the robot’s velocity, especially in the yaw direction [9]. The total angular momentum is the sum of the angular momentum produced by the robot’s individual links during locomotion, as described in (3),

where $\mathbf{L}_{\text{total}}$ represents the total angular momentum, $n$ is the number of links, $\mathbf{I}_{i}$ is the inertia tensor of the $i^{th}$ link’s CoM, $\mathbf{\omega}_{i}$ denotes the angular velocity, and $\mathbf{c}_{i}$ and $\dot{\mathbf{c}}_{i}$ represent the position and velocity of the $i^{th}$ link’s CoM relative to the whole-body CoM, respectively..

The substantial angular momentum generated during high-velocity locomotion presents a considerable challenge to precise control, especially in dynamic movements, where high stability is essential. To maintain postural equilibrium, we designed a reward function to minimize angular momentum for more stable and efficient locomotion. The angular momentum-based reward is defined in (4),

where $\|A^{a}_{t}\|_{2}$ represents the $L_{2}$ norm of the angular momentum at time $t$. The function $\text{clip}(\psi,c_{1},c_{2})$ constrains the output of the function $\psi$ to mitigate extreme values. Here, $\psi$ signifies any function that necessitates the clipping operation, with $c_{1}$ and $c_{2}$ indicating the lower and upper bounds of the output value, respectively.

In locomotion tasks, reward functions play a critical role in shaping the behavior of humanoid robots. Previous studies often rely on fixed reward functions, which limits the humanoid robots’ ability to accurately track velocities across a broad range of commanded velocities. To enhance the generalization capabilities of humanoid robots, we propose velocity-related reward functions that promote the learning of more dynamically adaptive behaviors. These functions enable precise control even over a wide range of commanded velocities. The rewards are categorized as follows: 1) base height reward, 2) feet clearance reward, and 3) joint position reward. In our approach, the coefficients $\alpha^{b}$, $\alpha^{f}$, and $\alpha^{p}$ are set to 0.2, 1.0, and 1.6, respectively.

To further evaluate the performance of the KSLC, we conducted sim-to-sim experiments, as access to a physical humanoid robot for sim-to-real testing was unavailable. The disparity between low-fidelity and high-fidelity simulators can be considered analogous to the gap between simulation and real-world conditions. Transferring a policy learned in a low-fidelity simulator to a high-fidelity simulator is known as sim-to-sim transfer [7]. As demonstrated in [7], agents that perform well in sim-to-sim transfer often keep satisfactory performance in sim-to-real transfer. Following this approach, MuJoCo was employed as the high-fidelity simulator in our sim-to-sim experiments. A video of the experiment is available at https://youtu.be/uVT8Up8vAKc and in the supplementary materials.

The sim-to-sim transfer results demonstrate that the policy trained with the KSLC in the Humanoid-Gym maintained consistent or even superior performance in the MuJoCo. The XBot-L successfully tracked a commanded velocity of 4.0 m/s without falling. As illustrated in Fig. 6, the arm movement patterns remained consistent with those observed in the Humanoid-Gym. Furthermore, the humanoid robot utilizing the KSLC method exhibited distinct behaviors at different commanded velocities, supporting its anticipated stability during high-speed locomotion.