Variable-Frequency Model Learning and Predictive Control for Jumping Maneuvers on Legged Robots

Aggressive jumping maneuvers with legged robots have received significant attention recently, and have been successfully demonstrated using trajectory optimization [QuannICRA19, chuongjump3D, matthew_mit2021_1, ChignoliICRA2021], model-based control [YanranDingTRO, continuous_jump_bipedal, GabrielICRA2021, park2017high, ZhitaoIROS22, fullbody_MPC], and learning-based control [zhuang2023robot, RL_jump_bipedal, yang2023cajun, vassil_drl, jumpingrl, lokesh_ogmp]. Unlike walking or running, aggressive motions are particularly challenging due to (1) the extreme underactuation in the mid-air phase (the robot mainly relies on the force control during contact to regulate its global position and orientation), (2) significant dynamics model error and uncertainty that are inherently hard to model accurately, especially with contact and hardware load during extreme motions, and (3) the trade-off between model accuracy and efficient computation for real-time execution. Achieving both target accuracy and robustness for long-flight maneuvers, such as high or long jumps, therefore still presents an open challenge. In this work, we address this challenge by developing a real-time MPC for quadruped jumping using a robot dynamics model learned from experiments.

Many control and optimization techniques have been developed for jumping motions. Trajectory optimization (TO) with full-body nonlinear dynamics is normally utilized to generate long-flight trajectories in an offline fashion (e.g., [QuannICRA19, chuongjump3D]). Many MPC approaches sacrifice model accuracy to achieve robust maneuvers by using simplified models that treat the trunk and legs as a unified body [NMPC_3D_hopping, ZhitaoIROS22, GabrielICRA2021, park2017high, YanranDingTRO]. Our recent iterative learning control (ILC) work [chuong_ilc_jump] handles model uncertainty to realize long-flight jumps via multi-stage optimization that optimizes the control policy after each jump offline until reaching a target accurately. It also relies on a simplified model for computational efficiency. However, this work focuses on target jumping rather than robustness, e.g. requiring the same initial condition for all trials. Different from existing works, we learn a robot dynamics model from experiments and develop real-time MPC using the learned dynamics to achieve both target accuracy and robustness in continuous quadruped jumping.

Learning robot models from experiments to capture complex dynamics effects and account for model errors with real hardware has become a popular approach [bauersfeld2021neurobem, hewing2019cautious, saviolo2022physics, duong23porthamiltonian]. Many frameworks learn residual dynamics using neural networks [bauersfeld2021neurobem, salzmann2023real] or Gaussian processes [hewing2019cautious, cao2017gaussian] and use MPC to control the learned systems [salzmann2023real, pohlodek2022hilompc] to improve tracking errors. Most existing works, however, primarily investigate dynamical systems without contact. Recently, Sun et al. [sun2021online] proposed a notable method to learn a linear residual model, addressing unknown dynamics during walking stabilization. Pandala et al. [Pandala_robustMPC] propose to close the gap between reduced- and full-order models by using deep reinforcement learning to learn unmodeled dynamics, which are used in MPC to realize robust walking on various terrains. Meanwhile, we use supervised learning to learn unmodelled dynamics with real data from hardware. Also, we learn dynamics for maneuvers with long-flight periods to tackle (1) the switching between multiple dynamic models due to contact, including flight phase where control action has very little effect on the body dynamics, (2) disturbances and uncertainty in dynamic modeling due to hard impact in jumping, and (3) the effect of intermittent control at contact transitions on state predictions.

In this letter, we propose a residual learning model that uses a variable-frequency scheme, i.e., varying the coarseness of integration time step, to address the aforementioned challenges and enhance long-term trajectory predictions over different jumping phases. MPC has been commonly used for jumping by optimizing the control inputs during the contact phase based on future state prediction during the flight phase (e.g.,[GabrielICRA2021, fullbody_MPC, park2017high, YanranDingTRO, continuous_jump]). A major challenge in MPC for long flight maneuvers is to utilize a limited number of prediction steps yet still effectively cover the entire flight phase and especially the final jumping target. Another challenge is to obtain an accurate model for complex dynamic maneuvers, unknown dynamics, and model mismatch with the real hardware to improve the jumping accuracy, while still ensuring real-time performance. Some recent works have addressed these challenges partially. Many methods use conventional single-rigid-body dynamic (SRBD) models, ignoring the leg dynamics, to achieve real-time execution [GabrielICRA2021, continuous_jump_bipedal, YanranDingTRO]. Using the SRBD model in the contact phase can lead to inaccurate predictions of the robot’s state at take-off, which is the initial condition of the projectile motion in the flight phase. Thus, the trajectory prediction error can accumulate significantly over long flight periods. Some methods account for leg inertia [ZiyiZhouRAL, He2024, fullbody_MPC], however, disturbance and uncertainty in dynamic modeling have not been considered. Recently, planning with multi-fidelity models and multi-resolution time steps has emerged as an effective strategy to enhance target accuracy and robustness [heli_cafempc, Norby_adaptivempc, Heli_hierarchympc]. Li et al. [Heli_hierarchympc] adopt a less accurate model in the far horizon and a more accurate but expensive model in the near future. Norby et al. [Norby_adaptivempc] adapt model complexity based on the task complexity along the prediction horizon. Li et al. [Heli_hierarchympc] also design multi-resolution time steps to cover the whole flight phase. While we vary the coarseness of the time step as [Heli_hierarchympc], we design a novel residual model learning approach combined with variable-frequency MPC to address the aforementioned challenges.

Contributions: The contributions of this letter are summarized as follows.

• •

  We learn a residual dynamics model directly from a small real experiment dataset. The model accounts for nonlinear leg dynamics, modeling errors, and unknown dynamics mismatch in hardware and during contact.

• •

  We propose learning the model in an variable-frequency scheme that leverages different time resolutions to capture the entire flight phase, the jumping target, and the contact transitions over a few-step horizon, thereby significantly improving the accuracy of long-term trajectory prediction.

• •

  We develop variable-frequency MPC using the learned model to synthesize controls that improve both target accuracy and robustness in dynamic robot maneuvers.

• •

  Extensive hardware experiments validate the effectiveness and robustness of our approach with single and consecutive jumps on uneven terrains. Comparisons with other MPC techniques using a nominal model or fixed-frequency are also provided.

Consider a legged robot modelled by state $\mathbf{x}_{k}$, consisting of the pose and velocity of the body’s center of mass (CoM), foot positions $\mathbf{r}_{k}$ relative to the body’s CoM, and ground force control input $\mathbf{u}_{k}$ at the legs, sampled at time $t_{k}$. We augment the nominal SRBD model (e.g., [GabrielICRA2021, park2017high, YanranDingTRO]) with a learned residual term to account for leg dynamics, model mismatch with the real robot hardware, as well as capture complex dynamic effects in contact and hardware:

where $f$ represents the nominal SRBD model, $\delta\mbox{\boldmath$f$}_{\bm{\theta}}$ with parameters $\bm{\theta}$ approximates the residual dynamics, and $\Delta t_{k}$ is the sampling interval. Our first objective is to learn the residual dynamics $\mbox{\boldmath$f$}_{\bm{\theta}}$ using a dataset of jumping trajectories.

After learning to improve the accuracy of model prediction, our second objective is to use the learned model (1) for MPC to track a desired state trajectory $\mathbf{x}_{0:K}^{*}$.

Achieving accurate target jumping with MPC requires not only an accurate model but also the coverage of the final state upon landing, which represents the accuracy of the jumping target. Using a fixed-frequency has shown to be efficient for locomotion tasks with limited aerial phases; however, this can face two challenges in long-flight maneuvers. On the one hand, using a fine time-step dirscretization enhances the model prediction accuracy but requires a large number of steps to capture the entire flight phase, thereby increasing the optimization size and computational cost. On the other hand, using a coarse time-step discretization allows capturing the entire flight phase efficiently but can sacrifice model prediction accuracy. For jumping tasks, different phases may require different model resolutions, e.g., fine time resolution during the contact phase but coarser time resolution during the flight phase as the model complexity is reduced due to unavailable force control and contact. Therefore, we propose to learn a model for dynamic maneuvers (Problem 1) with an variable-frequency scheme that uses a coarse time discretization in the flight phase and a fine time discretization in the contact phase. Importantly, this variable-frequency scheme is also synchronized with the MPC control of the learned model (Problem 2) by utilizing the same time steps for both fitting the learned model and performing the MPC optimization. This synchronization ensures that the same discretization errors are leveraged when using the model for MPC predictions. Thus, in our formulation $t_{0:K}$ does not have equal time steps and is a mixture of fine and coarse discretization capturing the contact and flight phases. Note that we use a constant time step $\Delta t_{k}$ during the different jumping phases. It is possible to use $\Delta t_{k}$ as input to the residual model neural networks. However, we need to collect a substantial amount of data for a wide range of $\Delta t_{k}$ to avoid overfitting, which would also increase the size of the neural network and computational cost.

Fig. 2 presents an overview of our system architecture. Our approach consists of two stages: variable-frequency model learning (Sec. IV) and variable-frequency MPC (Sec. V), which solve Problem 1 and Problem 2, respectively.

We synchronize the variable-frequency scheme for both model learning and MPC execution with the same (1) variable prediction timesteps $\Delta t_{{\cal C}}$ and $\Delta t_{{\cal F}}$ for contact and flight phase, respectively, (2) the same horizon length $K$ for all data collection, training, and MPC, and (3) the same contact schedule. Full-body trajectory optimization (TO) [QuannICRA19] is utilized to generate jumping references for various targets, including body states $\mathbf{x}^{*}$, joint states $\{\mathbf{q}_{\mathbf{J}}^{*},\dot{\mathbf{q}}_{\mathbf{J}}^{*}\}$, ground contact force $\mathbf{u}^{*}$, and foot positions $\mathbf{r}^{*}$. For data collection, we combine a baseline MPC using a nominal SRBD model and joint PD controller, generating diverse motions under disturbances. For training, we design a neural network to learn the discretized residual model $\delta\mbox{\boldmath$f$}_{\bm{\theta}}$ with variable sampling time via supervised learning. For control, we design an variable-frequency MPC using the learned dynamics to track a desired reference trajectory obtained from full-body TO. The feedback states from the robot include global body’s CoM $\mathbf{x}$, and joint states $\mathbf{q}_{\mathbf{J}},\dot{\mathbf{q}}_{\mathbf{J}}\in\mathbb{R}^{4}$, and foot positions $\mathbf{r}\in\mathbb{R}^{4}$.

In this section, we describe how to learn the residual dynamics $\delta\mbox{\boldmath$f$}_{\bm{\theta}}$ from data with a variable-frequency scheme that can cover the entire flight phase, the final state upon landing, and the contact transitions between jumping phases.

In this section, we design a variable-frequency MPC controller for the learned dynamics (6) to track a desired jumping reference trajectory obtained from full-body TO. For a given robot state $\mathbf{x}_{0}$, we formulate MPC as:

where (8d) represents input constraints related to friction cone and force limits and (8e) aims to nullify the force on the swing legs based on the contact schedule.

With the MPC horizon including $K_{{\cal C}}$ steps in the contact phase and $K_{{\cal F}}=K-K_{{\cal C}}$ steps in the flight phase, we define $\tilde{\mathbf{U}}_{{\cal C}}=[\tilde{\mathbf{u}}_{0}^{\top},\tilde{\mathbf{u}}_{1}^{\top},...,\tilde{\mathbf{u}}_{K_{{\cal C}}-1}^{\top}]^{\top}\in\mathbb{R}^{K_{{\cal C}}\times 4}$ as a concatenation of the control inputs in the contact phase. With this notation, the predicted trajectory is:

where $\tilde{\mathbf{X}}_{{\cal C}}=[\tilde{\mathbf{x}}_{1}^{\top},\tilde{\mathbf{x}}_{2}^{\top},...,\tilde{\mathbf{x}}_{K_{c}}^{\top}]^{\top}$ and $\tilde{\mathbf{X}}_{{\cal F}}=[\tilde{\mathbf{x}}_{K_{c}+1}^{\top},\tilde{\mathbf{x}}_{K_{c}+2}^{\top},...,\tilde{\mathbf{x}}_{K}^{\top}]^{\top}$ denote the concatenation of predicted states belonging to the contact and flight phase, respectively. The matrices ${\cal A}_{{\cal C}}$ and ${\cal A}_{{\cal F}}$ are computed as

While we use the current robot state for training the neural networks $\{\mathbf{G}_{\bm{\theta}}$, $\mathbf{h}_{\bm{\theta}}\}$ as presented in Sec. IV, we utilize the reference trajectory ($\mathbf{x}_{k}^{*},\mathbf{r}_{k}^{*}$) in future MPC horizons for guiding the MPC since the future robot states are not available at each MPC update ([continuous_jump_bipedal, YanranDingTRO, GabrielICRA2021, ZhitaoIROS22, park2017high]). The reference will be used to compute the output of the neural networks $\mathbf{G}_{\bm{\theta}}$ and $\mathbf{h}_{\bm{\theta}}$ in future horizons. In particular, we still use the current robot state and foot position $(\mathbf{x}_{0},\mathbf{r}_{0})$ to evaluate the residual term $\mathbf{B}_{\bm{\theta},0}=\mathbf{G}_{\bm{\theta}}(\mathbf{x}_{0},\mathbf{r}_{0})\Delta t_{0}$ for each MPC update, then define $\mathbf{B}_{\bm{\theta},k}=\mathbf{G}_{\bm{\theta}}(\mathbf{x}_{k}^{*},\mathbf{r}_{k}^{*})\Delta t_{k}$, $k=1,...,K_{{\cal C}}$. In (9a), we obtain $[{\cal B}_{{\cal C}}]_{\{m,n\}}=\mathbf{A}_{{\cal C}}^{m-n}\mathbf{B}_{n-1}$ and its residual $\left[{\cal B}_{{\cal C},\bm{\theta}}\right]_{\{m,n\}}=\mathbf{A}_{{\cal C}}^{m-n}\mathbf{B}_{\bm{\theta},n-1}~{}\textit{if}~{}m\geq n,\textit{and}~{}\bm{0}~{}\textit{otherwise}.$

For the sake of notation, we define $\chi_{k}=(\mathbf{x}_{k},\mathbf{r}_{k})$, $\chi_{k}^{*}=(\mathbf{x}_{k}^{*},\mathbf{r}_{k}^{*})$ as the actual and reference robot state and foot trajectory. The residual matrix $\mathbf{H}_{{\cal C},\bm{\theta}}$ in (9) is $\mathbf{H}_{{\cal C},\bm{\theta}}=\Delta t_{{\cal C}}$ $\mathbf{S}_{{\cal C}}\left[\mathbf{h}_{\bm{\theta}}(\chi_{0})^{\top}~{}\mathbf{h}_{\bm{\theta}}(\chi_{1}^{*})^{\top}~{}...~{}\mathbf{h}_{\bm{\theta}}(\chi_{K_{c}-1}^{*})^{\top}\right]^{\top}$, where $\mathbf{S}_{{\cal C}}$ is lower triangular matrix with each entry $[\mathbf{S}_{{\cal C}}]_{\{m,n\}}=\mathbf{A}_{{\cal C}}^{m-n}\in\mathbb{R}^{7\times 7}$, $m\geq n$. The residual matrix $\mathbf{H}_{{\cal F},\bm{\theta}}=\Delta t_{{\cal F}}\mathbf{S}_{{\cal F}}\left[\mathbf{h}_{\bm{\theta}}(\chi_{K_{c}}^{*})^{\top}\quad\ldots\quad\mathbf{h}_{\bm{\theta}}(\chi_{K}^{*})^{\top}\right]^{\top}$ where $\mathbf{S}_{{\cal F}}$ is lower triangular matrix with each entry defined $[\mathbf{S}_{{\cal F}}]_{\{m,n\}}=\mathbf{A}_{{\cal F}}^{m-n}\in\mathbb{R}^{7\times 7}$, $m\geq n$. The state prediction at the end of $K_{{\cal C}}$ steps can be computed as

where $\mathbf{S}=\left[\mathbf{A}_{{\cal C}}^{K_{c}-1}\mathbf{B}_{0}\quad\mathbf{A}_{{\cal C}}^{K_{c}-2}\mathbf{B}_{1}\quad...\quad\mathbf{B}_{K_{c}-1}\right]$, $\mathbf{S}_{\bm{\theta}}=\left[\mathbf{A}_{{\cal C}}^{K_{c}-1}\mathbf{B}_{\bm{\theta},0}\quad\mathbf{A}_{{\cal C}}^{K_{c}-2}\mathbf{B}_{\bm{\theta},1}\quad...\quad\mathbf{B}_{\bm{\theta},K_{c}-1}\right]$, and $\bm{\phi}_{\bm{\theta}}=\left[\mathbf{A}_{{\cal C}}^{K_{c}-1}\mathbf{h}_{\bm{\theta},0}\quad\mathbf{A}_{{\cal C}}^{K_{c}-2}\mathbf{h}_{\bm{\theta},1}\quad...\quad\mathbf{h}_{\bm{\theta},K_{c}-1}\right]$. Substituting (10) into (9) yields the state prediction as

where, $\mathbf{A}_{\text{qp}}=\begin{bmatrix}{\cal A}_{{\cal C}}\\ {\cal A}_{{\cal F}}\mathbf{A}_{{\cal C}}^{K_{c}}\end{bmatrix}$, $\mathbf{B}_{\text{qp}}=\begin{bmatrix}{\cal B}_{{\cal C}}&\bf 0\\ {\cal A}_{{\cal F}}\mathbf{S}&\bf 0\end{bmatrix}$, $\mathbf{B}_{\bm{\theta},\text{qp}}=\begin{bmatrix}{\cal B}_{{\cal C},\bm{\theta}}&\bf 0\\ {\cal A}_{{\cal F}}\mathbf{S}_{\bm{\theta}}&\bf 0\end{bmatrix}$, $\mathbf{H}_{\bm{\theta}}=\begin{bmatrix}\mathbf{H}_{{\cal C},\bm{\theta}}\\ \mathbf{H}_{{\cal F},\bm{\theta}}+{\cal A}_{{\cal F}}\bm{\phi}_{\bm{\theta}}\end{bmatrix}$, and $\tilde{\mathbf{U}}=\begin{bmatrix}\tilde{\mathbf{U}}_{{\cal C}}\\ \tilde{\mathbf{U}}_{{\cal F}}\end{bmatrix}$.

The objective of the MPC in (8) can be rewritten as:

leading to a quadratic program (QP):

where

In this letter, we developed a learned-model MPC approach that enables both target accuracy and robustness for aggressive planar jumping motions. We learned a residual model from a limited dataset to account for the effect of leg dynamics and model mismatch with real hardware. Given the learned model, we designed a learning-based variable-frequency MPC that considers the jumping transitions, the entire flight phase, and the jumping target during the optimization process. We demonstrated the scalability of our approach to handle new jumping targets with unseen reference trajectories. While allowing real-time computation, our approach uses the reference trajectory to evaluate the neural network residual model and ensure a linear model for MPC, which sacrifices a certain degree of accuracy if the system does not operate around the reference under substantial disturbances. To overcome this limitation, our future work will explore the use of nonlinear MPC and address its associated higher computational cost. We will also generalize the model learning and control framework with varying contact schedules. Further, we will incorporate line-foot, rolling, and soft contact in learning aggressive legged locomotion maneuvers.