Expanding Versatility of Agile Locomotion through Policy Transitions Using Latent State Representation

Each policy in our skill repertoire learns a locomotion gait from a reference clip using a motion imitation framework in simulation[36, 38, 5]. Specifically, a policy $\pi$ is learned by maximizing the expected return,

To transfer to the real-world, we apply extensive domain randomization (DR) of the simulation physics parameters and add other disturbances during the training process. This improves the inherent robustness of each policy and minimizes the performance gap between the simulation and the real-world setting[39, 40, 32]. With careful selection and application of the DR parameters described in Table I, we can reliably perform sim-to-real deployment of the independent locomotion policies with no failures (see Figure 1). For each gait, the training process is realized independently and from scratch. At this point, we have a collection of independent policies that are robust enough to be deployed to real-world robots. However, each policy only enables the robot to perform one specific gait, and a method to transition between them has to be established.

Each policy is considered a robust periodic controller capable of recovery from unstable states within an unspecified tolerance. For example, when the agent stumbles due to external perturbations, the policy acts to prevent its collapse and resumes its normal periodic motion afterward. Given this behavior, when a policy from the library is executing, it is possible to instantaneously switch the execution to a different policy at a particular destination phase. With proper timing of the switch and a good choice of the destination phase, the new active policy takes control, corrects unstable behavior due to the initial configuration, and resumes its normal periodic execution.

Our method provides a way to consistently identify transition configurations that yield successful outcomes, i.e., where the agent remains stable after a transition. It takes advantage of the rich latent representations of each policy, obtained from the last hidden layer. We formulate a transition function $f_{\theta}(\cdot)$ that maps the transition configurations to their outcome, where $\theta$ represents the weights of a feed-forward neural network, called the transition-net. The transition outcome is denoted by the binary variable $\alpha\in\{0,1\}$, and the transition configuration $\mathcal{C}$ is represented as a tuple of four elements,

To train the transition-net, we collect millions of transition samples in simulation as detailed in Section IV. Using this transition dataset, we train the transition-net in a supervised manner solving for a binary classification problem, where it aims to predict whether a transition configuration $\mathcal{C}$ would result in a successful outcome $\alpha=1$ or a failure $\alpha=0$. It optimizes for the binary cross-entropy (BCE) loss with $\alpha$ as the classification label $y$,

Next, to coherently unify all policies during deployment, we construct a meta-controller using the transition-net estimates as transition scores. The meta-controller queries the transition-net to identify the best transition configurations. It is responsible for choosing, scheduling, and executing the policies deployed on the robot using these elements:

• •

  an active policy, which controls the robot by generating joint target angles actuated via PD control;

• •

  a queued policy, to be switched for the active policy as soon as possible; and

• •

  a transition function $f_{\theta}(\mathcal{C})$ that provides a score given the current configuration $\mathcal{C}$.

During runtime, we start by defining an initial active policy that controls the robot. The active policy can be initialized to any policy available in the library. At some point in time, a request for a policy change happens, and a different policy from the library is queued. Once a policy is queued, the meta-controller recognizes that a switch should happen. At every timestep (30Hz), it queries the transition function $f_{\theta}(\mathcal{C})$ and computes the transition score of switching the active policy for the queued policy. Note that we search over multiple destination phases $\phi_{n}$ when querying and choose the highest-scoring one. When the transition score crosses a predefined threshold, the queued policy becomes active and takes control of the robot. The meta-controller pipeline is presented in Algorithm 1.

In our experiments we use the Unitree A1 quadruped robot, which has 12 joints actuated via a PD controller. In the training stage, the simulated agent matches the configurations and properties of the real robot. The observation space of the policy is composed of the state $\boldsymbol{s}_{t}$ and the reference motion $\boldsymbol{g}_{t}$. The state $\boldsymbol{s}_{t}\in\mathbb{R}^{102}$ includes state information from the current and past two timesteps of the agent. A single state is composed of 12 joint angles, the orientation and angular velocities (6), a binary indicator of the contact for each foot (4), and the previous actions of the policy (12). The reference motion $\boldsymbol{g}_{t}\in\mathbb{R}^{56}$ contains the target poses from the motion capture data at 4 future timesteps, up to one second in the future [32]. It functions as an implicit phase variable by modulating the near future targets of the agent. When searching for the best destination phase of the queued policy, we shift the queued reference motion data in the time axis.

The gait policies are trained following the imitation learning objective from [3], with the PPO clip loss parameters described in Table II. We developed the RL training environment using the simulator Isaac Gym [41], which can accelerate the training by instancing several parallel environments (4096) in a single physics scene and exposing the simulation states via a PyTorch-like API. With our implementation, the training process of a single locomotion policy takes less than one hour of wall clock time on a system equipped with an Intel i7-11800H and an RTX 3070 8GB. The policies can be deployed with the real robot in a zero-shot manner.

Our proposed transition-net is trained in a supervised manner with millions of transition samples collected from simulation with domain randomization as described in Table I. The samples contain paired labels of transition configuration and their corresponding outcome, where the source and destination policies are uniformly sampled from the library. Since the random switching strategy used to collect the samples introduce an imbalanced number of failure and successful cases, we sub-sample transition samples such that the number of success and failure samples are balanced.

The transition-net is implemented as a feed-forward network with $128-64-32$ neurons as the intermediate layers, with dropout ($p=0.4$) and ReLU activation functions applied after each layer except for the output layer, which uses a sigmoid. The neural network is trained for 100 epochs using a mini-batch of 128 samples, and the AdamW optimizer [42] with a learning rate of $5\mathrm{e}{-4}$. During deployment, we use the output of the transition-net as a scoring function, and only perform transitions when the score crosses the threshold $f_{\theta}(\mathcal{C})>th$; where $th=0.95$.

To evaluate the robustness of the transitions generated by the transition-net (Ours), we compare it to four other transition strategies: random, value function [36], transition motion tensor (TMT) [6], and TMT-value. TMT-value is an extension of TMT we developed by integrating value function estimates into the querying function of TMT. We first evaluate the success rate of the transition strategies under increasing disturbance forces in simulation. Each strategy performs every possible transition pair while disturbance forces are applied at random directions and intervals (0.1 to 0.3 seconds). The disturbance magnitude starts at 0 N and increases up to 500 N in increments of 50 N. We collect 5000 transition samples for every combination of transition strategy, transition pair, and increment of disturbance magnitude.

Figure 4 (top) summarizes the success rate of each strategy and transition pair. As expected, randomly switching between policies without any heuristics yields suboptimal and unstable transitions. By employing the timing heuristics from the phase variable, TMT achieves a success rate of 95% for $pace$–$trot$. While the phase variable is a considerable indicator of when to perform a transition, it still fails to disambiguate between different states with similar phase values, e.g., from external disturbances or other motion inconsistencies. Other strategies such as value function, TMT-value, and transition-net achieve a higher degree of robustness, highlighted by the higher success rate under the more intense disturbances. This is because more dynamic representations can better capture the state of the robot in relation to its environment dynamics. However, since the policy’s value function is not specifically designed to facilitate transitions, value function and TMT-value strategies suffer from inconsistencies with lower success rates compared to our approach.

Next, we deploy the meta-controller with the real A1 quadruped robot and once again evaluate the success rate of each transition strategy. We collect 50 transition samples for every possible combination of transition strategy and pair from the library. From Figure 4 (bottom), we can observe that the robustness achieved in the simulation translates fairly well to the real-world setting, albeit with a lower overall success rate. The value function strategy produces inconsistent transitions. This is expected as the value function is trained in the simulation setting and might produce noisy value estimates in the real-world setting. In this experiment we do not introduce external disturbances to the robot, allowing TMT and TMT-value to achieve high success rates and showing that phase serves as a reasonably accurate proxy. Finally, using the policy’s latent representation with our transition-net yields the most robust transitions. It can accurately identify the most appropriate configuration to perform the transitions for every pair in the real-world.

Jointly analyzing the results from the simulation and real-world settings, we observe that the most challenging transition are dynamic-to-static transitions, i.e., when the robot performs a full-stop after a moving gait, such as $pace$–$stand$ and $trot$–$stand$. This is evidenced by the steeper curves in Figure 4 (top) and the low success rate of the random approach in the real-world in Figure 4 (bottom). The transition-net achieves 19% higher average success rate compared to the second best strategy, TMT-value.

To investigate the effectiveness of using the latent state representations $\boldsymbol{\psi}_{m}$ as the input of transition-net, we experiment with two alternative representations: phase variable $\phi_{m}$ and the explicit state of the agent, a subset of $\boldsymbol{s}_{t}$. For each representation, we measure the accuracy of the transition-net in predicting successful transition configurations given a balanced validation set. The dataset contains 5500 samples for each pair, with a total of 33000 samples. The experiment results shown in Table III illustrate the strengths of each representation.

The phase variable remains a solid low-dimensional proxy for the agent’s state when the source policy is not very dynamic. This behavior is highlighted by the high accuracy in transitions where $stand$ is the source policy, but poor accuracy for all other transitions. A more descriptive state representation is to explicitly use the recent history of the agent’s joint positions, orientation, angular velocity, and feet contacts. With this input, the transition-net can internally estimate the phase information while having a more comprehensive state representation, resulting in higher accuracy than simply using the phase variable. In the end, the latent state representation encapsulates all of these desirable properties with the addition of implicit encoding of the environment obtained during the training process in simulation. Therefore it yields the highest prediction accuracy in transitions involving more dynamic states, such as the ones originating from $pace$ and $trot$.