Learning to Locomote: Understanding How
Environment Design Matters for Deep Reinforcement Learning

The skilled control of movement, via learned control policies, is an important problem of shared interest across physics-based character animation, robotics, and machine learning. In particular, the past few years have seen an explosion of shared interest in learning to locomote using deep reinforcement learning (RL). From the machine learning perspective, this interest stems in part from the multiple locomotion tasks that found in RL benchmark suites, such as the OpenAI Gym. This allows for a focus on the development of new RL algorithms, which can then be benchmarked against problems that exist as predefined RL environments. In their basic form, these environments are innocuous encapsulations of the simulated world and the task rewards: at every time step they accept an action as input, and provide a next-state observation and reward as output. Therefore, this standardized encapsulation is general in nature and is well suited for benchmark-based comparisons.

However, there exist a number of issues and decisions that arise when attempting to translate a given design intent into the canonical form of an RL environment. These translational issues naturally arise in the context of animation problems, where design intent is the starting point, rather than a predefined RL environment such as found in common RL benchmarks. In this paper, we examine and discuss the roles of the following issues related to the design of RL environments for continuous-action problems: (a) initial-state distribution and its impact on performance and learning; (b) choices of state representation; (c) control frequency or “action repeat”; (d) episode termination and the ”infinite bootstrap” trick; (e) curriculum learning; (f) choice of action space; (g) survival bonus rewards; and (h) torque limits. In the absence of further understanding, these RL-environment issues contribute to the reputation of RL as yielding brittle and unpredictable results. Some of these issues are mentioned in an incidental fashion in work in animation and RL, which serves to motivate the more comprehensive synthesis and experiments presented in this paper. We document these use cases in the sections dedicated to individual issues.

Our work is complementary to recent RL work that examines the effect of algorithm hyperparameters, biases, and implementation details, e.g., (Zhang et al., 2018a; Henderson et al., 2018; Rajeswaran et al., 2017; Packer et al., 2018; Ponsen et al., 2009; Hessel et al., 2019; Andrychowicz et al., 2020). Henderson et al. (2018) discusses the problem of reproducibility in RL, due to extrinsic factors, such as hyperparameter selections and different code bases, and intrinsic factors, such as random seeds and implicit characteristics of environments. Hessel et al. (2019) analyze the importance of multiple inductive biases and their impact on the policy performance, including discount, reward scaling and action repetitions. Engstrom et al. (2020) points to certain code-level optimizations that sometimes contribute the bulk of performance improvements. Some of these configuration decisions are more impactful than others. The performance of RL policies can be sensitive to the change of hyperparameters which indicates the brittleness of RL algorithms. Andrychowicz et al. (2020), concurrent to our work, performs a large scale ablation of hyperparameters and implementation choices for on-policy RL, including a mix of algorithmic and environment choices.

We consider the traditional RL setting, in which a Markov Decision Process (MDP) is defined by a set of states $\mathcal{S}$, a set of actions $\mathcal{A}$, a transition probability function $p\colon\mathcal{S}\times\mathcal{A}\to\mathcal{P}(\mathcal{S})$, which assigns to every pair $(s,a)\in\mathcal{S}\times\mathcal{A}$ a probability distribution $p(\cdot|s,a)$, representing the probability of entering a state from state $s$ using action $a$, a reward function $R\colon\mathcal{S}\times\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, which describes the reward $R(s_{t+1},s_{t},a_{t})$, associated with entering state $s_{t+1}$ from state $s_{t}$, using action $a_{t}$, and a future discount factor $\gamma\in[0,1]$ representing the importance of future rewards. The solution of an MDP is a policy $\pi_{\phi}\colon\mathcal{S}\to\mathcal{A}$, parameterized by parameters $\phi$, that for every $s_{0}\in\mathcal{S}$ maximises: $J(\phi)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t+1},s_{t},a_{t})\right]$, where the expectation is taken over states $s_{t+1}$ sampled according to $p(s_{t+1}|s_{t},a_{t})$ and $a_{t}$ is the action sampled from $\pi_{\phi}(s_{t})$.

Our algorithm of choice for the following experiments is TD3 (Fujimoto et al., 2018), a state of the art, off-policy, model free, actor-critic, RL algorithm. In TD3, an actor represents the policy $\pi_{\phi}$ where $\phi$ are the weights of a neural network optimized by taking the gradient of the expected return $\nabla_{\phi}J(\phi)$, through the deterministic policy gradient algorithm (Silver et al., 2014): $\nabla_{\phi}J(\phi)=\mathbb{E}\left[\nabla_{a}Q^{\pi}(s,a)\nabla_{\phi}\pi_{\phi}(s)\right]$, and a critic is used to approximate the expected return $Q^{\pi}(s,a)$ which corresponds to taking an action $a$ in state $s$ and following policy $\pi$ thereafter. The critic is also a neural network parameterized by $\theta$ and optimized using temporal difference learning using target networks (Mnih et al., 2015) to maintain a fixed objective $\nabla_{\theta}(Q^{\pi}_{\theta}(s,a)-y)^{2}$ where $y$ is the target and is defined as $y=r_{t}+\gamma\left[Q_{\theta target}(s_{t+1},\pi_{\phi}(s_{t+1}))\right]$.

We explore, evaluate, and discuss the impact of each defining component of the environment, each potentially affecting the final performance and learning efficiency of policies in locomotion environments. For each of these experiments, unless otherwise stated, we use an implementation of TD3 based on the original code from the authors and adapted for more experimental flexibility. Hyperparameters and network architectures are given in appendix A. All of our experiments are based on the Bullet physics simulator (Coumans and Bai, 2019) and its default Roboschool locomotion environments available through Gym (Brockman et al., 2016). Each experiment is performed a total of 10 times, and averaged across seeds. The set of seeds is always the same. In a few cases, we rely on existing published results, using them to provide relevant insights in our context.

We now present each RL environment design issue in turn. For clarity, we combine the explanation, related work, and results for each.

The distribution from which the initial states of simulation episodes are sampled from plays an important role in learning and in exploration in RL. In many reinforcement learning locomotion benchmarks (Todorov et al., 2012; Coumans and Bai, 2019), the default initial state is sampled from a very narrow distribution, nearly deterministic, at the beginning of a new episode. Recent work (Ghosh et al., 2018) addresses the problem of solving harder tasks by partitioning the initial state space into slices and train an ensemble of policies, one on each slice. Packer et al. (2018) looks at a similar problem, but instead of modifying the initial-state distribution, changes the environment dynamics, by changing mass, forces and lengths of links at every trajectory. In our experiments, changing the initial-state distribution does not affect the underlying environment dynamics, which remain the same. It instead affects the way the state-space is presented to the RL algorithm.

Default PyBullet locomotion environments create a new initial state by sampling the joint angles of each link from a uniform distribution $\mathcal{U}(-0.1,0.1)$. Instead, we sample each joint angle from a new distribution $\mathcal{U}(\kappa\theta_{\min},\kappa\theta_{\max})$, where $\theta_{\min}$ and $\theta_{\max}$ are the specified lower and upper limit of the joint, predefined in the robot description files (i.e. usually URDF or XML files). The parameter $\kappa$ quantifies the width of the initial-state distribution to investigate how its variance affects the policy.

We run experiments with the following environments: Ant, HalfCheetah, Hopper and Walker2D, and illustrate the test episode return in Figure 1. For Ant and HalfCheetah environments, results for the original narrow initial-state distribution and the broad initial-state distribution with $\kappa=1$ are shown. For the Hopper and Walker environment, we show additional training curves with different values of $\kappa$ to investigate the influence of $\kappa$ at finer scales. A broader initial-state distribution leads to worse sampling efficiency and a lower episode return in the end for the majority of environments. We believe that for Ant and HalfCheetah, the largely invariant nature of the learning to the initial state distributions stems from the large degree of natural static stability for these systems after falling to the ground. This stability leads to a rapid convergence to similar states from a wide range of initial states.

The difference in performance caused by broadening the initial-state distribution reflects its impact, which is often neglected, in policy training. A narrow initial-state distribution increases learning efficiency compared with tasks with same underlying mechanics $P(s_{t+1}|s_{t},a_{t})$ but a broader initial-state distribution $p(s_{0})$. Intuitively, having a restricted range of initial states allows the agent to focus on those initial states, from which it learns how to act. If the agent is always dropped in the environment in a very different state, it is more difficult for the policy to initially learn to act, given that the experiences will represent very disjoint regions of the state space. However, learning on a broader initial-state distribution results in a more robust and general final policy. Table 1 shows results for policies trained on the default environments, with narrow initial-state distribution, when tested with different initial-state distributions. The policies trained on narrow initial-state distributions deliver significant worse run-time performances as $\kappa$ increases, showing a failure to generalize with respect to much of the state space. Thus, training with a broader initial-state distribution lead to a more robust policy that covers a wider range of the state and action space.

Summary: RL results can be strongly impacted by the choice of initial state distribution. A policy trained with the narrow initial distribution in commonly used environments often fails to generalize to regions of the state space that likely remain unseen during training. The broader the initial state distribution $p_{0}(s)$, the more difficult the RL problem can become. As a result, there is a drop in both sample-efficiency and reward with a wider initial-state distribution. However, the learned policies are more generalizable and more robust.

The state, at any given time, captures the information needed by the policy in order to understand the current condition of the environment and decide what action to take next. Despite the success of end-to-end learning, the choice or availability in the state representation can still affect the problem difficulty. In this section, we investigate how adding, removing and modifying information in the state affects the RL benchmarks. We discuss the following modifications: (1) adding a phase variable for cyclic motions, (2) augmenting the state with the Cartesian joint position in Cartesian coordinate, (3) removing contact boolean variables, (4) using the initial layers of a pre-trained policy as state representation for a new policy.

The choice of action repeat (AR), also called frame skip, has been fundamental to early work in DQN (Mnih et al., 2015) and it has been overlooked as a simple hyperparameter in many other algorithms. This relevant domain knowledge consists of repeating each action selected by the agent a fixed number of times. This effectively lowers the frequency at which the agent operates, compared to the environment frequency. The impact of the action repeat parameter has been studied by (Hessel et al., 2019), in which they train an adaptive agent that tunes the action repetition. In (Lakshminarayanan et al., 2017), the authors propose a simple-but-limited approach that improves DQN. The policy outputs both an action and an action-repeat value, which is selected from one of two possible values. Tuning these (hyperparameter) values is highly beneficial to the learning. More recently, Metelli et al. (2020) also explore the choice of control frequency and action persistency for low-D continuous environments, such as CartPole and MountainCar.

We systematically examine the effect of action repeat in common PyBullet locomotion environments. The environment frequency is fixed and given by the simulator. With $AR=1$, the control frequency and the environment frequency are equal, and a gradient step is computed after each control step. By choosing a different action repeat $AR$ value, we are scaling down the control frequency and taking a gradient step after $AR$ environment steps. The learning curves are shown in Figure 6. As expected, the choice of action repeat has a significant impact on the learning. For the Walker and Hopper, AR=1 produces the best performance, with larger values producing worse performance. This also leaves open the possibility that a higher frequency control rate could improve learning. Humanoid learns best with AR=3 or AR=4, and Ant with AR=2. An exception is HalfCheetah, which is not particularly sensitive to the choice of action repeat.

Summary: A good choice of control frequency, usually implemented as action repeat parameter for RL, is essential for good learning performance. Controlling the actions at frequencies that are too high or too low is usually harmful.

Locomotion environments such as HalfCheetah are constrained with a time limit that defines the maximum number of steps per episode. The characters continue their exploration until this time limit is reached. In both PyBullet (Coumans and Bai, 2019) and Mujoco (Todorov et al., 2012) versions of these environments, the default time limit is 1000. In some environments, such as Ant, Hopper, Walker2D, and Humanoid, an additional natural terminal condition is defined when the character falls. In either case, the motivation for limiting the duration of potentially infinite-duration trajectories is to allow for diversified experience.

It has been previously shown by Pardo et al. (2018) that terminations due to time limits should not be treated as natural terminal transitions and that these transitions should be bootstrapped instead. If we consider a transition at time-step $t$ with starting state $s_{t}$, action $a_{t}$, transition reward $r_{t}$, and next state state $s_{t+1}$, with an action-value estimate $\hat{Q}(s_{t+1})$, the target value $y$, when bootstrapping, becomes:

Here $I_{\mathrm{term}}\in\{0,1\}$ is an indicator variable, set to $0$ for any terminal transition, and is otherwise $1$, i.e., for non-terminal transitions. With infinite bootstrapping, we use $I_{\mathrm{term}}=1$ for time limit terminations, which corresponds to considering the time limit termination as being non-terminal.

Our experiments with TD3 are shown in Figure 7. These show significant improvements for this infinite bootstrapping trick, in particular for Ant and HalfCheetah, where the time limit termination is reached more readily. This finding is consistent with Pardo et al. (2018) experiments with PPO (Schulman et al., 2017). Given that it plays an important role in the final policy and the return value, and that it is often overlooked or left undocumented, we list it as one of the factors that matters for learning a locomotion policy. Like the choice of discount factor, small changes can play an important role in the many environments that do not have a natural limited duration.

Summary: Infinite bootstrap is critical for accurately optimizing the action-value function, which usually corresponds to high-quality motion.

Curriculum-based learning can enable faster learning by progressively increasing the task difficulty throughout the learning process. These strategies have been applied to supervised learning (Bengio et al., 2009) and are becoming increasingly popular in the RL setting, e.g.,  (Florensa et al., 2017; Yu et al., 2018; Narvekar and Stone, 2019; Xie et al., 2020) and many of which are summarized in a recent survey (Narvekar et al., 2020). The curriculum therefore serves as a critical source of inductive bias, particularly as the task complexity grows. The important problem of curriculum generation is being tackled from many directions, including the use of RL to learn curriculum-generating policies. These still face the problem that learning a full curriculum policy can take significantly more experience data than learning the target policy from scratch (Narvekar and Stone, 2019).

Summary: By training on task environments with progressively increasing difficulty, curriculum learning enables learning locomotion skills that would otherwise be very difficult to learn.

In most locomotion benchmarks, torque is the dominant actuation model used to drive the articulated character. In contrast, low-level stable Proportional-Derivative(PD) controller are used in a variety of recent results from animation and robotics, e.g., (Peng et al., 2017; Xie et al., 2019; Tan et al., 2011). The PD controller takes in a target joint angle $\bar{q}$ as input and outputs torque $\tau$ according to:

where $q$ and $\dot{q}$ represent the current joint angles and velocities, and $k_{p}$ and $k_{d}$ are manually defined. With a low-level PD controller, the policy produces joint angles rather than torques. In previous motion imitation work, e.g.,  (Peng et al., 2018; Xie et al., 2019; Chentanez et al., 2018), the reference trajectory provides default values for the PD-target angles, and the policy then learns a residual offset. However, this PD-residual policy is infeasible for non-imitation tasks since a reference trajectory is not available. Here we study the impact of using (non-residual) PD-action spaces for purely objective-driven (vs imitation-driven) locomotion tasks. For implementation and training details on the PD controller, we direct the reader to appendix B.

Training with low-level PD controllers is more prone to local minimum than the same environment with torque-based RL policy. This is because it is relatively easy to learn a constant set of PD-targets which maintain a static standing posture, which is rewarded via the survival bonus. To address this local-minimum issue, we first train with a broader initial state distribution, and then, after mastering a basic gait, the agent is trained with the original environment with a nearly-deterministic initial-state. Among the PyBullet locomotion benchmarks, Walker2D suffers more severely from the problem of a local minimum.

As shown in Figure 8, using PD target angle as action space often leads to a faster early learning rate. However, it may converge to lower-reward solutions than the torque action space, and it may hinder quick adaption to rapid changes in the state as the character moves faster (Chentanez et al., 2018).

Previous work (Peng and van de Panne, 2017) shows that PD-control-targets outperform torques as a choice of action space in terms of both final reward and sampling efficiency. The PD-control action space is far better at motion imitation tasks than learning locomotion from scratch. We believe this arises for two reasons. First. in motion imitation, the reference trajectory already serves as a good default, and the policy only needs to fine-tune the PD target, which is simpler than learning the PD target from the scratch. Second, current locomotion benchmarks are designed and optimized for direct torque control rather than low-level PD control, including the state initialization and the reward function.

Summary: For non-imitation locomotion tasks, PD-control-targets actions appears to have a limited advantage over a torque-based control strategy. However, PD-control-targets often learn a basic gait more efficiently during the early stages of training.

The reward function plays an obvious and important role in learning natural and fluid motion. In PyBullet locomotion benchmarks, the character is usually rewarded for moving forward with a positive velocity, and penalized for control costs, unnecessary collisions, and approaching the joint limit. Additionally, the reward function also contains a survival bonus term, which is a positive constant when the character has not fallen; otherwise it is negative or zero. In Henderson et al. (2018) and Mania et al. (2018), the existence of the survival bonus terms can lead to getting stuck in local minimum for certain types of algorithms. Here we investigate the impact of the value of the survival bonus in the reward function.

We experiment with 3 different values of survival bonus: 0, 1 and 5, and train the agent with TD3 algorithm. To allow for a fair comparison among policies trained with different survival bonus value, the policies are always tested on the environments without the survival bonus reward. The default PyBullet environment set the value of survival bonus to 1, for which the episodic test reward significantly outperforms the other two tested values for the survival bonus, as shown in Figure 9. Including the survival bonus term in the reward function is necessary; setting it to 0 makes the discovery of a basic walking gait too difficult. If the survival bonus term is too large, however, the algorithm exploits the survival bonus reward while neglecting other reward terms. This results in a character that balances but never steps forward.

Summary: The survival bonus value provides a critical form of reward shaping when learning to locomote. Values that are too small or too large leads to local minima corresponding to falling-forward and standing still, respectively.

The capabilities of characters are defined in part by their torque limits. Both high and low torque limits can be harmful to learning locomotion in simulation or actual robots. A high torque limit is prone to unnatural behavior. On the other hand, a low torque limit is also problematic since it can make it much more difficult to discover the solution modes that yield efficient and natural locomotion. In Abdolhosseini (2019), the authors experiment with the impact of torque limits by multiplying the default limits by a scalar multiplier $\lambda\in[0.6,1.6]$. Policies are trained using PPO (Schulman et al., 2017). The observed result is that higher torque limits obtain higher episodic rewards. As $\lambda$ decreases, the agent is more likely to get stuck at local minimum, and the agent fails to walk completely when the multiplier is lower than a threshold (Abdolhosseini, 2019). Such results motivate the idea of apply a torque limit curriculum during training, as a form of continuation method. This allows for large torques early in the learning process, to allow for the discovery of good solution modes, followed by progressively decreasing torque limits, which then allows the optimization to find more natural, low torque motions within the good solution modes.

Summary: Large torque limits benefit the exploration needed to find good locomotion modes, while low torque limits benefit natural behavior. A torque limit curriculum can allow for both.

Reinforcement learning has enormous potential as means of developing physics-based movement skills for animation, robotics, and beyond. Successful application of RL requires not only efficient RL algorithms, but also the design of suitable RL environments. In this paper we have investigated a number of issues and design choices related to RL environments, and we evaluate these on locomotion tasks. As a caveat, we note that these results have been applied to fairly simple benchmark systems, and thus more realistic and complex environments may yield different results. Furthermore, we evaluate these environment design choices using a specific RL algorithm (TD3). Other RL algorithms may be impacted differently, and we leave that as future work. However, our work provides a better understanding of the often brittle-and-unpredictable nature of RL solutions, as bad choices made with regard to defining RL environments quickly become problematic. Efficient learning of locomotion skills in humans and animals can arguably be attributed in large part to their “RL environment”. For example, Underlying reflex based movements and central pattern generators help constrain the state distribution, provide a suitable action space, and are tuned to control at a particular time scale.

Many environment design issues can be viewed as a form of inductive bias, e.g., the survival bonus for staying upright, or a given choice of action space. We anticipate that much of the progress needed for efficient learning of motion skills will require leveraging the many aspects of RL that are currently precluded by the canonical environment-and-task structure that is reflected in common RL benchmarks. As such, state-of-the-art RL-based approaches for animation should be inspired by common RL algorithms and benchmarks, but should not be constrained by them.