Understanding the Stability of Deep Control Policies for Biped Locomotion

The design of biped controllers that produce realistic human walking has been a challenging subject in computer graphics. The key challenge is designing a balancing mechanism, which is usually implemented as a feedback loop that adjusts the controller output (joint torques or PD target poses) based on its input (body state and environment information). A variety of control approaches has been explored to generate responsive and realistic human locomotion with diverse feedback mechanisms. Finite state machines equipped with manually-designed, intuitive feedback rules have been an effective approach in the early biped controller design [HWBO95, YLvdP07]. To mitigate the complexity of full-body dynamics, simplified dynamics models such as inverted pendulums have been used in controller design [KH10, CBvdP10, TLC^∗09, MdLH10, KH17].

Data-driven (a.k.a. example-guided) approaches have been frequently used to improve the naturalness of simulated animations by adopting motion capture data as a reference to track [SKL07, LKL10, LYvdPG12, LvdPY16]. The balance mechanisms for data-driven controllers can be manually-crafted [LKL10], learned from a collection of example motions using a regression method [SKL07], or derived from a linear feedback policy using stochastic optimization and/or linear regression [LYvdPG12, LvdPY16]. Model predictive control (MPC) is used for synthesizing the full-body character animations in a physically plausible manner using reference motion data [DSAP08, HHC^∗19]. Trajectory optimization is also employed to fulfill given tasks [ABDLH12, PM18, LH18]. Many studies have utilized nonlinear optimization methods to improve the robustness of controllers, or to explore control schemes for given tasks  [YL10, SKL07, LYvdPG12, dLMH10, WFH10, WHDK12].

Recently, deep reinforcement learning has received significant attention and shown impressive improvements in the biped control problem. The control policy represented by a deep neural network can effectively achieve a feedback balancing mechanism in learning-based control. A variety of DRL algorithms has been proposed to learn control policies for biped locomotion [HTS^∗17, SML^∗15, YTL18]. Yu et al. \shortciteyu2018learning proposed a mirror symmetry loss with curriculum learning to learn a locomotion policy without any reference data.

DRL algorithms can also take advantage of using motion capture data. Liu et al. \shortciteliu_learning_2017 used deep Q-learning to learn a scheduler that reorders short control fragments which are in charge of reproducing short motion segments. Peng et al. \shortcitepeng_deeploco:_2017 presented a two-level hierarchical DRL-based control framework learned from short reference motion clips. They also presented a DRL method in their follow-up study to learn a control policy that imitates a given reference motion clip [PALvdP18]. Lee et al. \shortcitelee2019scalable proposed a DRL-based controller for a muscle-actuated anatomical model. Learning a control policy that exploits a set of reference motion data can be facilitated by recurrent neural networks [PRL^∗19] and a motion matching technique [BCHF19]. Although the adoption of DRL in biped control has been very successful, few in-depth analysis of the characteristics of DRL-based controllers has been presented.

Gait and postural stability has been measured quantitatively using a waist-pull system [RHJ^∗01] and a movable platform [BWSC01], which can apply quantified perturbations to human subjects. In biomechanics, gait stability has been estimated using measures derived from nonlinear time-series analysis, such as Lyapunov exponents and Floquet multipliers [DCCS00]. The correlation of foot placement with balancing capability has been investigated in biomechanics and robotics [MMK12]. Wight et al. \shortcite10.1115/1.2815334 advocated Foot Placement Estimator (FPE) as a measure of balance for bipedal robots. In computer graphics, measuring the response to unexpected external pushes is a common criterion for estimating the resilience and robustness of controllers [YLvdP07, MdLH10, KH17, LKL10, LvdPY16].

Lee et al. \shortcitelee2015push statistically analyzed the push-recovery stability of humans and that of simulated bipeds controlled by a hand-crafted feedback controller \shortcitelee2010data. Using maximum detour distance as a stability measure, they identified key gait factors (walking speed, level of crouching, push magnitude and timing) that affected the stability of human walking through statistical analysis. Their experimental setups are adopted in our simulation experiments and their human experiments serve as a reference of human vs simulation comparisons in our study.

This study begins with a question about how the characteristics of DRL-based controllers are similar or different compared to human walking and previous biped controllers. We aim to gain a deeper understanding and insight into how DRL achieves better stability in notoriously-challenging control problems.

The state of the biped $s=[p,\dot{p},\phi]$ includes the positions and velocities of the body links relative to the body coordinate system attached to the skeletal root. $\phi\in[0,1]$ is a normalized phase that matches the temporal span of the reference motion. The output of the control policy is a PD (Proportional Derivative) target that generates joint torques through PD control. The reward function is

where ${r}_{q}$, ${r}_{v}$, and ${r}_{e}$ are the reward for tracking reference poses, joint velocities, and positions of end-effectors, respectively. The end-effector and position/velocity tracking rewards are multiplied as suggested by Lee et al. \shortcitelee2019scalable, since they are reinforcing each other. ${r}_{g}$ encourages the biped to walk along a straight line and thus come back to the line after an external push. We set ${w}_{q}=0.8$, ${w}_{v}=0.1$, and ${w}_{g}=0.1$.

Here, $\mathbf{q}$ is an aggregated joint angle vector, $\dot{\mathbf{q}}$ is a time derivative of $\mathbf{q}$, and $\mathbf{p}_{e}$ is an aggregated vector of end-effector positions. The hat symbol indicates values from the reference motion. $\mathrm{dist}(\mathbf{d},\mathbf{p}_{c})$ is the distance from the center of mass (CoM) of the character $\mathbf{p}_{c}$ to the straight line $\mathbf{d}$.

We used Proximal Policy Optimization [SWD^∗17] and generalized advantage estimation [SML^∗15] to learn a policy function that maximizes the expected cumulative reward. The learning process is episodic. Many experience tuples are collected stochastically from episodic simulations. In each episode, experience tuples are generated by sampling actions from the policy at every time step and the policy is updated systematically by a batch of experience tuples. We refer the readers to the work of Peng et al. \shortcite2018-TOG-deepMimic for implementation details.

Even if the control policy is learned with a reference motion clip, the exploration strategy of reinforcement learning examines unseen states around the reference trajectory to achieve a certain level of resilience to withstand small unexpected disturbances. The control policy can be even more robust if it learns how to cope with disturbances and uncertainty in the learning process. It has been reported that randomly pushing the biped in the episodic simulations would result in improved robustness [WFH10, PRL^∗19]. In each episode, we applied random force for 0.2 seconds to push the biped sideways from left or right. The detailed values of push magnitude and timing used in the learning process will be described in section 5.

Human locomotion can be characterized by a family of parameters. The flexibility and representation power of deep neural networks allow parametric variations in gaits to be learned in a single network-based policy, which takes those parameters as state input. The state of the gait-conditioned policy $s=[p,\dot{p},\phi,c,l,v]$ includes three gait parameters $c,l,v$, which are normalized crouch angle, stride length, and walking speed, respectively. The stance knee is supposed to be straight at the middle of the stance phase in normal gait. The crouch gait has its knee flexed throughout the stance phase (see Figure 2). The crouch angle indicates the level of crouching normalized to $[0,1]$. The stride length and walking speed are normalized to have zero mean and one standard deviation.

Because a gait-conditioned policy should deal with gait parameters as state input, learning a gait-conditioned policy requires a collection of example motions that span a target parametric domain. We generate example motions by kinematically varying a single reference motion clip, which represents a normal gait with average walking speed and stride length. Given parameter values $(c,l,v)$, the use of hierarchical displacement mapping and time warping edits the reference motion clip to have the desired stride length $l$, crouch angle $c$, and walking speed $v$ [LS99].

During episodic learning, the initial state of each episode is taken from the state space containing the target parametric domain. A common way to determine the initial state is to sample uniformly in the state space. This naïve learning of a policy often suffers from a biased exploration problem. Many DRL algorithms tend to explore successful regions in the target domain more aggressively while leaving less successful regions unexplored in starvation. For example, human walking is more stable when it crouches to lower down its CoM [Sch87]. The control policy parameterized by a crouch angle would explore crouch walking more frequently than normal (straight-knee-at-stance) walking. Consequently, the learned normal gait in the policy would be less robust.

Recently, Won et al. \shortciteWon:2019 proposed an adaptive sampling method to deal with parametric body shape variations. We adopt their sampling idea to learn our gait-conditioned policies. The key idea is to give more opportunities to less successful regions in the target domain. The measure of success is a marginal value function $V_{m}(s_{\alpha})$ that estimates the sum of expected rewards for each gait parameter $s_{\alpha}=(c,l,v)$.

where $V(s)=V(s_{\alpha},s_{\beta})$ is a value function which approximately measures the cumulative reward when the controller follows current policy from state $s$, $s_{\beta}$ is a state vector excluding gait parameters, $S_{\beta}$ is the domain of $s_{\beta}$, and $p_{s}$ is a density function which is assumed a constant. The probability of exploring $P({s}_{\alpha})$ is

Here, $\mu$ is the expectation of $V_{m}$, which is updated along with $V_{m}$, $Z$ is a scaling factor to normalize $P$, and $k$ is the value that decides the degree of uniformity of $P$ over gait parameter space. MCMC (Markov Chain Monte Carlo) sampling with the probability aims to make the marginal value function $V_{m}(s_{\alpha})$ near-uniform across the domain of $s_{\alpha}$. We chose $k=1$ for all controllers.

The statistics of gait factors in human experiments are shown in Table 2. The trials in human data are classified into two groups: Group 1 and Group 2. The participants in Group 1 trials recovered their balance in a single step, and thus the detour distance peaked in the step. Most of the participants recovered their balance within three balance-correcting steps except for a few outliers. They took more than one step when they experienced mild difficulty. In the outliers, participants panicked by unexpected pushes and failed to return to the line. The Group 2 includes all trials except for the outliers.

In Figure 5, we compare the maximum detour distances of Human, DDC, DRL-B*, and DRL-A*. The results of Human and DDC experiments were taken from the work of Lee et al. \shortcitelee2015push. The simulation experiments were designed to reproduce the human experiments with setups and data distributions carefully tuned to match the target experiments. The only difference is that we can collect far more trial data by simulation. The Group 1 of Human, DDC, DRL-B*, and DRL-A* include 228, 3858, 4851, and 16036 trials, respectively, and the Group 2 includes 450, 13707, 20534, and 27993 trials, respectively. The comparison graph shows that DRL-B* and DRL-A* are clearly superior to DDC and comparable to human participants.

It was empirically verified in the previous study that crouch walking is more stable than normal walking in Human and DDC experiments. In particular, $30^{\circ}$-crouch walking was the most stable. This postulation agrees with our intuition that lowering down the CoM improves the gait stability. Although the detour distance measurements of DRL-B* and DRL-A* do not follow this trend, the success rate experiments agree with the postulation. Graph 6 shows that both DRL-B* and DRL-A* are more robust with 20^∘/30^∘-crouch walking than normal and 60^∘-crouch walking. $30^{\circ}$-crouch walking was the most robust in terms of the success rate.

Table  3 shows the results of Type 3 tests, which shows the fixed effects of level of crouch, walking speed, magnitude/timing of push on the detour distance. The tests confirm that all factors that were proven to be significant in the human experiments are also significant ($p<0.05$) in the DRL experiments.

Gait-specific policies DRL-specific* supposedly outperform gait-conditioned policies DRL-A* and DRL-B* since the scope of the gait-specific policy focuses on a single reference trajectory, while the gait-conditioned policies have to cope with a continuous spectrum of parametric domains. Specifically, the domain $[\mu_{s}-1.5\sigma_{s},\mu_{s}+2\sigma_{s}]\times[\mu_{l}-1.5\sigma_{l},\mu_{l}+2\sigma_{l}]$ is explored in our experiments, where $\mu_{s}$ and $\mu_{l}$ are the average walking speed and stride length, respectively, in the human experiments. $\sigma_{s}$ and $\sigma_{l}$ are their standard derivations. The use of an asymmetric domain is related to the perception plausibility of motion editing. Edited character animations that walk slower than a reference are more likely to look unnatural than animations that walk faster than the reference [VHBO14].

We conducted push-recovery experiments of 1000 trials for each of the three policies. Push forces are drawn from a normal distribution with a mean of 200N and a standard derivation of 35N (see Figure 7). A family of motion clips were generated to learn DRL-specific* polices. The stability of each gait-specific policy learned for a particular walking speed and a stride length is comparable to the stability of DRL-B* near the mean values, but clearly outperforms when the samples are away from the means. It means that naïve parametric learning can deal with only a narrow range of the parametric domain. Adaptive sampling of DRL-A* improves the stability at the corners of the domain and consequently learns a policy practically usable over the entire domain within a modest computational time. Figure 8 depicts the parametric coverage of successful episodes. The coverage of DRL-A* is wider than the coverage of DRL-B*. It means that DRL-A* can better deal with slow walking (narrow strides) and fast walking (wide strides), while DRL-B* is effective only at mid ranges.

Deciding the number of hidden layers of the policy network depends on the dimension and size of the action, state, and parameter spaces. In practice, the number of layers and the number of nodes in each layer are important hyper-parameters to tune empirically. In our experiments, we tested with two, three, and four hidden layers and found that learning was the most successful with three layers.

It has been previously postulated that balancing strategies based on foot placements play a dominant role in dynamic walking. We hypothesized that deep policies learn to smartly choose post-push footholds to recover balance within a few steps. We conducted push-recovery experiments with four deep policies DRL-A, DRL-B, DRL-A*, and DRL-B*. In each trial, the simulated biped was pushed from the left during its left foot in stance and took the next step on the right to recover its balance. Figure 9 shows that the stability of control policies are characterized by the pattern of post-push foot placements, which are plotted with respect to the reference footprint to step on if there are no disturbances. The push magnitude and timing are randomly sampled, while the walking speed, stride length, and crouch angle are fixed. Push magnitudes are sampled from a normal distribution with a mean of $200N$ and a standard derivation of $35N$. Push timings are sampled from another normal distribution of a mean of $34\%$ and a standard deviation of $21\%$.

We used 158 normal gait data, and 22 30^∘-crouch gait data . Note that the human data were collected from many participants under varied conditions and thus necessarily noisy, while the simulation data were collected in a controlled simulation setup. Therefore, the side-by-side direct comparison is meaningless, but the data should be interpreted qualitatively. In the human data (Figure 9(a), 9(d)), the disturbed swing foot tends to land behind (in the y-axis) the undisturbed step position. It means that human participants tend to slow down temporarily to cope with lateral disturbances. The success rate of DRL-A is very low ($3.0\%$ for normal walking and $10.1\%$ for 30^∘-crouch walking), and the post-push footholds of successful trials are all at nearly the same y-coordinate as the undisturbed reference step position since DRL-A did not learn how to cope with disturbances. Unlike DRL-A, DRL-A* exploited uncertainty in the learning phase and learned to adjust its step length (shorter in the y-axis) similarly to human balance strategies.

The foot placement graphs in Figure 9 also show that post-push step positions are closely correlated with push timing. The push may occur in either a double-stance phase $[0\%,20\%]$ or a swing phase $[20\%,100\%]$. The push at the early swing phase allows the walker to have a sufficient time to move its leg to an appropriate position for balance recovery, while the push at the late swing phase forces the walker to put the swing foot down rapidly with little time for control. Therefore, control policies are more resilient to early-swing pushes than late-swing pushes. Note that maximum detour distances are shorter (can be interpreted as more resilient) with later-swing pushes, which seems contradictory to the proposition based on the success rate. As we discussed before, there is no ultimate criterion for resilience and gait stability. The notion of stability can be characterized richly by a mixture of measures. For example, Figure 9 shows that the post-push step positions tend to be farther from the undisturbed reference step position as the timing of the push is earlier (the 20% curve is farther than the 60% curve from the reference position), but the success rate for early pushes are higher than late pushes (the ratio of blue and green dots in the <20% push timing region is higher than that of the 40%–60% region).

As shown in Figure 9, the successful post-push steps of normal gaits are mostly placed less in the y-axis than the reference stepping position, as in the human data. 30^∘-crouch gaits are not necessarily the case, but the y values of more successful post-push steps are smaller than that of the reference position. This means that perturbation during the learning process allows learning policies that are more similar to the actual human balancing mechanism. The significant fatigue experienced by the human participants during crouch walking is not modeled in our simulation model, which might result in the slightly different tendency of post-push steps for 30^∘-crouch gaits.

The plots in Figure 10 depict the correlation between post-push foot placements and push magnitude/timing. DRL-A* learned its balance strategies based on foot placements, which are strongly correlated with how strong the disturbance is and when it occurs. DRL-A* is more robust than DRL-A at all push magnitudes and timings. DRL-A* adjusts its post-push foothold not only in the push (lateral) direction but also in the moving direction to better respond to the pushes. We also observed the clear correlation of the number of detour steps with push magnitude. As push magnitude increases, the character needs one step (Group 1 marked in blue) to three steps (Group 2 - Group 1 marked in green) to recover balance after the push.

In an interactive control problem, it is critical to process the user input on-the-fly. A sudden change of the reference motion during the simulation can act as an impulse in the system. Since the gait-conditioned policies learn from diverse walking motions under disturbances, a simulated character controlled by these policies is robust to the change of input motions, and thus can transit from a gait pattern to another smoothly without any additional technique such as motion blending.

We demonstrate an interactive control to show the robustness of our gait-conditioned policies (Figure 11). In this example, we compare the four deep control policies: DRL-A*, DRL-B*, DRL-A, and DRL-B. Each character is controlled by a single control policy, and the reference motion is changed whenever the user input consisting of walking speed, stride length, and crouch angle is given. Despite diverse types of sabotaging with the obstacle, push forces, and repeated changes of input motion, the character controlled by DRL-A* survives alone to the last. Please refer the supplemental video at (05:24).