Learning Multimodal Bipedal Locomotion and
Implicit Transitions: A Versatile Policy Approach

To preserve the scalability of our approach to diverse behaviors while also being agnostic to the robot’s morphology, we use rough reference motions, which are not required to be either kinematically or dynamically feasible but merely be a visual proxy of the desired motion. We only need the keyframes of the base pose (position and orientation) and, optionally, the contact sequence (for well-defined contact patterns) in a desired motion’s reference. As shown in Fig 3 (top), we generate the rough reference motion by linearly interpolating between user-defined keyframes. We can optionally specify the reference contact sequence through a binary signal for each leg to obtain locomotion modes with well-defined contact patterns. For instance, a reference for walking mode defined as constant base height with increasing position value along the heading direction seldom results in a distinct left-right marching motion, making it hard to distinguish between walking and hopping. Note that there are no reference trajectories for the actuated states (joint DOFs), thus allowing the policy to learn the required actions to realize without overspecified constraints.

Unlike prior works, [19, 13, 11] which have a one-to-one correspondence between the agent’s state and latent trajectories, we propose to encode a complete reference motion into a single point in the latent space before the agent’s training. By decoupling the temporal correspondence, the policy has direct access to the compressed past and future intent, which is agnostic to its current state. As shown later in the paper, a state-agnostic mode command helps us generate complex behaviors (as in Fig. 1) with discrete switching through simple mode planning methods. Since the reference motions are time-series data with a strong temporal relationship between each point in the trajectory, we use a Long Short-Term Memory (LSTM) Autoencoder to learn our mode embeddings. Additionally, the architecture of an LSTM network is independent of the input trajectory’s length, which is of practical significance. Unlike periodic (walk) or steady state (stay idle) modes, where we can generate reference motions of arbitrary length, transient motions such as jumping on a block are time-critical, thus having a defined trajectory length. As shown in Fig 2 (top) , we train an LSTM Encoder, $E_{\theta}$ and Decoder, $D_{\psi}$ pair based on the reconstruction loss below

where $X$ is a reference motion of length T and $\hat{X}$ is its reconstruction. Thus, $E_{\theta}$ recurrently takes in the entire trajectory one timestep at a time and generates a single latent vector $z$ as output. A repeat vector is then constructed, making T identical copies of $z$ and sent as an input to the decoder along with the hidden state being propagated internally in a recurrent fashion, thus outputting a reconstruction of the $X$ trajectory, $\hat{X}$ of the same length. We use the open-source implementation sequitur¹¹1 https://github.com/shobrook/sequitur for training the autoencoder, with identical encoder and decoder networks with a single hidden layer of 32 neurons and no activations. As seen Fig. 4 the mode encoder effectively encodes a complete trajectory into a single latent vector thereby leading to a compact representation of the desired behaviors.

In contrast to traditional locomotion policies for legged robots that are command-conditioned on the base’s lateral, longitudinal, and yaw velocities, we use a latent-conditioned policy, $\pi(a|s,z)$ where $z\in\mathbb{Z}$, is a latent space of behaviors/modes. Prior approaches simultaneously discover and learn such skills from scratch [20] or morphologically accurate motion capture data [11, 21, 13], which are unsuitable for robot control. Thus we decoupled the training first to learn the latent mode encodings (as explained in Sec. II-B, II-A) and then trained a policy conditioned to those encodings to imitate the corresponding locomotion mode. We control the policy only through mode commands and relax the need to accurately track velocity commands as it is often merely a surrogate objective used to satisfy high-level task objectives. An indirect way to modulate the robot velocity and a method to plan compositional locomotion tasks like parkour is discussed in Sec III. Described below are the specific details pertaining to the policy.

With the structure of our policy extablished, we now explain our proposed adaptive sampling-based training in Algo. 1 to realize a consistent performance across modes and transitions. Unlike prior approaches of curriculum learning to gradually learn the increasing difficulty of a single behavior [2], comparing the complexities across different locomotion modes and defining an optimal learning order is not straightforward. However, there is indeed an inherent difference in complexity that requires more exposure to difficult instances while training. Hence, we propose to adaptively sample the mode commands in an episode such that the mode and the transition in which the policy performs the poorest are chosen more frequently. As shown in Algo. 1, during each episode, we choose an initial mode $\text{m}_{\text{i}}$ and a final mode $\text{m}_{\text{f}}$, based on skewed distributions $\text{p}_{\text{i}}$ and $\text{p}_{\text{f}}$ respectively. $\text{p}_{\text{i}}$, $\text{p}_{\text{f}}$ are constructed based on the records of the policy’s returns across each mode ($\text{R}_{\text{i}}$) and each transition ($\text{R}_{\text{f}}$). Note that while $\text{p}_{\text{i}}$ provides the marginal probability of choosing $\text{m}_{\text{i}}$, $\text{p}_{\text{f}}$ provides the conditional probability of choosing $\text{m}_{\text{f}}$ given $\text{m}_{\text{i}}$. We compute the switching timestep $\text{t}_{\text{s}}$, based on a randomly chosen clip²²2a single cycle of a locomotion mode: $\text{clip}_{\text{s}}$ and a phase: $\phi_{\text{s}}$. The above sampling technique introduces two parameters, $\gamma$ and $\epsilon$, for each distribution, $\text{p}_{\text{i}}$ and $\text{p}_{\text{f}}$. $\gamma$ prevents the distributions from being myopic (only dependent on the previous episode’s returns) and rather be computed based on the history of returns corresponding to a given mode/transition, preventing the policy from alternatively learning and unlearning. Unlearning or catastrophic forgetting can also arise due to a learnt mode not being sampled further down the training. To alleviate this, we use $\epsilon$ to add a small offset, thereby ensuring the mode with the best performance has a non-zero, yet the smallest probability of being chosen. For the results below, the values of $\epsilon$ = $0.2$ and $\gamma_{i},\gamma_{f}$ varies as $0.2,0.4,0.8$ across different training.

Having learnt a policy that can successfully realize a multitude of locomotion modes and transitions, we now showcase a method to rapidly compose mode plans to solve high-level tasks such as traversing a non-trivial terrain that calls for parkour-like locomotion. In this work, we use a rudimentary task-based mode planner as a proof of concept to show the ease of composing complex locomotion maneuvers with a trained multimodal policy. We parameterize the mode plan as a zero-order spline (as in Fig. 9) and formulate a collocation problem by defining knots of the spline at discrete time intervals. Thus, the planning problem is to find the optimal sequence of modes to maximize a task objective using the multimodal policy. We consider the reaching task where the objective is for the robot to reach a fixed goal position in the world frame. To solve this optimization, we use tabular Q-learning to obtain an optimal policy: the mode plan. The task MDP has the knot index as the state and the mode index at that knot as the action, with the reward function defined as

where, $p_{x}^{goal},p_{x}(t)$ are the goal and current base position, $k$ is the total number of knots and $dt$ is the discretisation period for the planner. The optimal plan is obtained through Monte Carlo rollouts with a given terrain type in the environment. Note that the mode planner has neither the exteroceptive feedback of the underlying terrain nor the proprioceptive feedback of the robot states. Thus the planner only generates an open-loop time-dependent plan of modes to be performed by the multimodal policy at the low level. A summary of the experiments conducted is tabulated in table III, where we ran five ideal trials of 100 episodes in parallel, and the best plan was chosen for each task.

In order to highlight our key contributions, we train three different multimodal policies, namely $\pi_{1}\text{, }\pi_{2}\text{, and }\pi_{3}$ whose training configurations are tabulated in table I. For comparison, we conduct each of the experiments with both uniform sampling ($\text{m}_{\text{i,f}}\sim\text{U}_{\text{i,f}}$) and our proposed adaptive sampling of modes and transitions ($\text{m}_{\text{i,f}}\sim\text{p}_{\text{i,f}}$). The attached video shows that the proposed framework successfully realizes the different locomotion modes, such as periodic: walk, leap, transient: launch, and steady-state: idle, from the rough reference demonstrations. Quantitatively the mean performance across modes is computed as the mean normalized returns over the rollouts for each mode, as shown in table II. We conduct the following comparisons to study the implications of our proposed design choices.

Choice of behavior encoding: Similar to [20, 10], for a baseline, we train by uniformly sampling the different locomotion modes in each episode that were parametrized as latent vectors with one-hot encoding. We compare our approach with the baseline by keeping everything the same and swapping the parameterization for learnt latent encodings. A key observation was that the policies learnt with one-hot encoding resulted in indistinguishable locomotion modes, i.e., mode aliasing. The attached video and Fig. 6 (left) show that the baseline policy results in a similar base height profile for both walk and leap modes. On the other hand, policy trained with learnt latent encoding Fig. 7 (left), results in a distinct motion for each mode where there is a well-defined rise and fall of the base height for the leap mode while it is kept constant at the nominal value for the walk mode, as desired. We hypothesize that the features in our latent encoding reflect crucial attributes of the locomotion mode that aids the policy in realizing discernible behaviors successfully.

Choice of mode sampling: As mentioned, all three experiments were conducted and trained with uniform (baseline) and adaptive (proposed) sampling. Pertaining to the modes, Fig. 5 (top) shows that the policies trained with adaptive mode sampling have a higher mean return across all modes as compared to sampling modes uniformly along the training. This effect is particularly significant, for a higher number of modes, as in the case shown in Fig. 5, which was of $\pi_{1}$ with $9$ modes in training. Furthermore, uniform sampling often leads to mode collapse, where the policy only succeeds in a subset of modes while failing or performing poorly in the rest, which can be seen in Fig. 6(center). $\pi_{1}$ trained with uniform sampling was successful in every other mode in the training but leap-left mode where the resulting base height profile fails to be anywhere close to the reference motion shown in blue in Fig. 7 (center). In contrast, the adaptive sampling variant successfully imitates the reference to the best extent permitted by the robot’s dynamics across all modes, as seen in the attached video and Fig. 7 (center).

Trained with and without contact sequence: From $\pi_{1}$, $\pi_{2}$, we can observe that a rough demonstration with just the keyframes for the base pose is sufficient to realize the locomotion modes without needing a reference of either the joint or contact states. The attached video shows that this training leads to asymmetric yet natural-looking gaits for periodic modes like walk and leap. However, we often need well-defined contact sequences to locomote on constrained terrains such as stepping stones. To this end, we illustrate the flexibility of our framework by integrating a rough demonstration of the contact state along with the base pose, in $\pi_{3}$. Particularly, with the same base pose reference of constant base height, we construct two modes: hop and walk, by only changing the reference contact states between the legs to be in and out of synchronization, respectively. The video results of $\pi_{3}$ show the resultant symmetric well-defined walking and hopping behavior upon using a contact sequence reference.

Frequency of motion in modes: The current framework with mode-conditioned policies lacks granular control over certain states ( e.g., walking velocity), unlike the current SOTA command-conditioned policies and MPCs. Though we justify this trade-off by realizing fruitful high-level tasks through mode planning, preliminary investigations show that we can crudely control the frequency or rate of motion by modulating the clock rate of the policy’s clock input. Specifically, in the leap mode, increasing the clock signal frequency led to a direct increase in the heading speed of the robot. In Fig. 7 (right), the heading velocity is double ($1.0$ m/s) the nominal value of $0.5$ m/s when we increase the clock rate by a factor of $2$. This effect saturates at thrice the nominal clock rate at a velocity of $\sim 1.2$ m/s, which we believe is due to the physical limits of the robot’s form factor and the nature of the given leap mode.

As stated before, the major advantage of learning a single policy to operate on multiple locomotion modes lies in inherently realizing smooth inter-mode transition maneuvers. We elucidate the above hypothesis and the benefit of training with adaptive transition sampling below

Emergent non-trivial transitions: We first showcase the significance of learning a single multimodal policy with implicit transitions as opposed to having multiple unimodal policies and explicit transitions. For the baseline akin to current SOTA, we train $4$ different unimodal policies that specialize in a single mode, namely: walk, leap, launch, and idle, respectively, and verified that these policies successfully realized their respective modes when tested without any transitions. However, when we try to compose motions with transitions such as pacing and jumping on top of a box, the optimal transition time and maneuver are not straightforward to obtain. Hence, when we discreetly switch between the walking and launch unimodal policies, the baseline fails as the states faced during the transition are out of distribution for the launch policy, as seen in the attached video and Fig. 6 (right). In contrast, Fig. 8 shows that the multimodal policy results in a feasible and emergent transition maneuver (zoomed-in view) to perform the parkour-like task with the transitions: walk $\rightarrow$ launch, launch $\rightarrow$ leap, and leap $\rightarrow$ idle. Thus, the multimodal policy successfully stitches together a smooth sequence of distinct locomotion modes even with discrete switching commands.

Choice of transitions sampling: Similar to the case of mode sampling, uniformly sampling transitions in training leads to transition collapse, thus failing in a subset of all possible transitions. To circumvent this, extending the adaptive sampling to transitions, we see higher mean returns over all the transitions than uniform sampling along training in Fig. 5 (bottom). Furthermore, we can also see that uniform sampling falls into a local optimum where it is successful in specific transitions but sub-optimal in the rest leading to high returns early on in training (point denoted with a star in Fig. 5 (bottom) ). However, it never recovers from this local optima and thus results in transition collapse. In contrast, adaptive sampling of transitions ensures consistent performance across all transitions and increases along the training, leading to compelling and reactive transition maneuvers, as seen in the supplementary video and table II.

As presented in the attached video, we can successfully traverse challenging terrains (as in Fig. 1) with discreetly-switched open-loop plans (as in Fig. 9) generated by the mode planner. The planner generates agile parkour-like motions by exploiting the multimodal policy’s different locomotion modes to overcome gaps of width $0.45$ m, plateaus of height $0.2$ m, and blocks of height $0.4$. The experiment shown in Fig. 1 demonstrates a leap of width $102\%$ the maximum leg length and a jump of height $80\%$ the nominal COM height of our mini-biped platform. As seen in table III, thanks to the low-dimensional and compact Q-table, we can rapidly compose these complex behavior compositions (as in Fig. 1 with the corresponding plan in Fig. 9) in a few minutes, thus being capable of planning in real-time.