Constrained Reinforcement Learning for Dexterous Manipulation

TRPO is a policy gradient method to solve Markov Decision Processes, that avoids parameter updates that change the policy too much with a KL divergence constraint on the size of the policy update at each iteration Schulman et al. (2015).

We train TRPO on-policy i.e., the policy for collecting data is same as the policy that we want to optimize. The objective function $J(\theta)$ measures the total advantage $\hat{A}_{\theta_{old}}$ over the state visitation distribution $p^{\pi_{\theta_{old}}}$ and actions from $\pi_{\theta_{old}}$, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator. TRPO aims to maximize the objective function subject to a trust region constraint which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter $\delta$. The same can be summarized in Eq. 1.

CPO Achiam et al. (2017) is built on top of the TRPO algorithm to solve Constrained Markov Decision Processes (CMDPs) which include a cost function, $C$ and a cost discount factor $\gamma_{c}$ along with the standard MDP learning problem $(S,A,T,R,p_{0},\gamma)$. In a local policy search for CMDPs, on top of the TRPO optimization, we additionally require policy updates to be feasible for the CMDP. Our objective function thus adds another condition to limit the expected discounted cost under a cost limit, $cl$ for each constraint. The same can be summarized in Eq. 2.

We consider an object relocation task where the agent, a 24-DOF Adroit hand, learns a policy to grasp and relocate a blue ball from a tabletop to a green region (Fig. 1). We formulate this task as a Constrained MDP, $(S,A,T,R,p_{0},\gamma,C,\gamma_{c})$, where $S$ is the state space, $A$ is the action space, $T$ is the transition function, $R$ is the reward function, $p_{0}$ is the initial state distribution, $\gamma$ is a discount factor, $C$ is the cost function and $\gamma_{c}$ is a cost discount factor.

In a typical episode, at each time $t$, the agent receives an observation $o_{t}$ based on the current state. After the agent takes an action $a_{t}\sim\pi_{\theta}(o_{t})$ based on the observation, it gets reward $R(s_{t})$ from the environment, incurs a penalty cost $C(s_{t})$ and arrives at a new state with observation $s_{t+1}=T(s_{t},a_{t})$. Based on the RL algorithm, CPO or TRPO, the agent optimizes the corresponding objective function.

Observation space: The observation space is 39-dimensional including 24 hand joint angles, hand translation (3-D), hand orientation (3-D), relative position of the hand with respect to the object (3-D), relative position of the hand with respect to the goal (3-D) and relative position of the object with respect to the goal (3-D).

Action space: Each action for the relocation task is 30-dimensional, including 24 hand joint angles, 3-D hand translation and 3-D hand rotation.

Reward: The agent is rewarded for getting closer to the object, lifting the object up, taking the object closer to the goal, taking the hand closer to the goal, and reaching really close to the goal.

We define boundary constraints to restrict the initial motion of the robot arm in the direction of the object for relocation. Considering $\textbf{x}_{h}$ as the initial hand position, $\textbf{x}_{b}$ as the initial object position, x as the current hand position, the constraints are defined in Eq. 3, where $r$ is the boundary radius. The same can be visualized in Fig. 2. The derivation of the constraints can be found in Appendix A.

We penalize the agent with a fixed penalty cost whenever the agent violates any of the formulated constraints. If it violates both, it receives twice the penalty cost (set to 0.01). For practical purposes, we relax the second constraint to range between -0.1 and 1.1. A tight second constraint does not allow the robot to learn grasping the ball.

We use the MuJoCo physics simulator as our testing platform. All our RL policies are pre-trained using behavior cloning via 25 demonstrations collected via CyberGlove III Rajeswaran et al. (2017). Our policy network is a Gaussian Multi-Layer Perceptron (MLP) with two hidden layers of 32 neurons each. We also train a value network and a cost value network (only for CPO), both MLPs with two hidden layers of 128 neurons each. The learning rate for behavior cloning on our policy network, value network and cost value network is 0.001. For training via CPO and TRPO algorithms, our reward and cost discount factors are both 0.995 and the GAE parameter is 0.97. The constraint configurations for our different experiments are detailed in Appendix B.

We evaluate both CPO and TRPO algorithms on the relocation task to verify the effect of optimizing for constraints on sample efficiency and average cost during training. We evaluate two variants of the TRPO algorithm – one where the reward is penalized with the incurred cost (TRPO-RP), and one without the penalty (TRPO).

We see that CPO has a lower average number of violations than both the TRPO policies for three different intensities of the boundary constraint. From Fig. 3 (bottom), we see that CPO learns to satisfy the constraints better throughout the training process, making it potentially safer to train on real robots. Qualitative results showing the behavior for all three algorithms during early, mid and late training can be found in the video at this link. CPO also has lower sample efficiency for the three constraint cases, which is a small trade-off to ensure safer learning (Fig. 3 (top)).

We also evaluate the average number of violations for the CPO and TRPO policies after training. We find that the CPO policy continues to maintain fewer violations and is thus safer to roll out for real-world applications than the TRPO policies even though all the policies can perform the task successfully $\geq 95\%$ of the times (Fig. 5).

We evaluate the effect of changing the boundary radius in our constraint formulation on training a CPO policy. We find that a tighter radius takes significantly more samples to train, whereas a relaxed radius is more sample efficient (Fig. 4 (left top)). This behavior can also be reinforced by the average number of violations reducing more quickly for a relaxed constrained than a tighter one, although it is obviously lower to begin with (Fig. 4 (left bottom)).

We also evaluate how changing the overall cost limit affects the way CPO policies are trained. We see that as the cost limit decreases, the sample efficiency also decreases (Fig. 4 (center top)). From the average number of violations plot (Fig. 4 (center bottom)), we see that CPO optimizes perfectly for the respective limits and maintains the allowed cost throughout most of the training.

We evaluate how changing the scale of penalties impacts the way CPO trains. We linearly scale the cost limits in this case to maintain the same number of allowed constraint violations. We observe that the higher the penalty cost per violation, the longer it takes for the policy to train (Fig. 4 (right top)). However, scaling of the penalty costs does not really impact how the average number of violations reduce during training (Fig. 4 (right bottom)).