Proximal Policy Gradient: PPO with Policy Gradient

Policy gradient algorithms introduced in (Williams 1992; Sutton et al. 2000a, b) compute the gradient of the expected return with respect to policy parameters. An action-independent term, typically the state value function, can be subtracted from this gradient. Then, the policy gradient is in the form of gradient of log probability multiplied by the advantage function. The resulting policy gradient algorithm respectively boosts (reduces), the probability of advantageous (disadvantageous) actions. This policy gradient algorithm is too slow to be used as a modern day baseline. Therefore, a reasonable choice of recent advances are applied to the classic policy gradient theorem based methods. In (Achiam 2018), the generalized advantage estimate (GAE) (Schulman et al. 2015b) and the advantage normalization are applied. The resulting algorithm is called the Vanilla Policy Gradient (VPG) (Achiam 2018; Dhariwal et al. 2017).

Recently, the trust region policy optimization (TRPO) (Schulman et al. 2015a) and the proximal policy optimization (PPO) (Schulman et al. 2017) present significant performance improvement over VPG by using the proposed surrogate objective function. In particular, PPO is very popular due to the learning stability and implementation simplicity. PPO provides a novel solution to the problem of increasing the number of policy updates by introducing the clipping idea. In practice, the number of iterations cannot be indefinitely large. Instead, the iteration continues until a target KL divergence is reached. Thus a much larger number of policy update iterations per each roll out is possible, greatly increasing the sample efficiency while maintaining the steady improvement on the average return. This is the baseline PPO algorithm we use to perform experiments. Note that there are variety of variations of PPO (Schulman et al. 2017) for various enhancements.

We explore a slight modification to both VPG and PPO, devise an objective that appears to be in between, and study the performance. The objective without clipping has the gradient exactly the same as VPG’s gradient. This objective can be considered as a partial variation of the VPG objective, i.e., PPG optimizes the changes of the VPG objective. In contrast, PPO optimizes the expected importance-weighted advantage, and hence, the gradient of the PPO objective is different from VPG, which is based on the policy gradient theorem (Sutton et al. 2000a). To achieve performance similar to PPO, we introduce a new clipping algorithm designed for the PPG objective. The clipping is applied to samples that have their probabilities adjusted enough, so that these samples are excluded from the training batch.

We perform experiments in the popular continuous control environments in OpenAI Gym (Brockman et al. 2016) with the MuJoCo simulator: Ant, HalfCheetah, Hopper, and Walker2d, and also in the corresponding Bullet environments: AntBulletEnv, HalfCheetahBulletEnv, HopperBulletEnv, and Walker2DBulletEnv. We found that PPG and PPO are very similar with a few cases where PPG appears to have marginally higher average returns. The entropy tends to decay slower in PPG than in PPO. Therefore, PPG solutions appear to be more stochastic than PPO solutions. Thus, we show that policy gradient can be used for the PPO algorithm.

Model free reinforcement learning algorithms have evolved from the classic dynamic programming, policy and value iterations (Sutton and Barto 2018) to Q-learning (Watkins 1989) and direct policy search methods using neural networks for policy and value functions (Mnih et al. 2015, 2016; Andrychowicz et al. 2018; Lillicrap et al. 2015; Fujimoto et al. 2018; Haarnoja et al. 2018).

On-policy algorithms are popular for their stability, i.e., applicability to a wide set of problems. One of the early on-policy policy-gradient based methods is REINFORCE (Williams 1992) that directly used Monte-Carlo returns. Instead, value estimate was used in the Actor-Critic method (Sutton et al. 1999). In modern days, value estimates are frequently formulated as advantages. To reduce the noise in advantage, the generalized advantage estimation method has been proposed in (Schulman et al. 2015b).

The policy-gradient theorem (Sutton et al. 2000a) is used to compute the gradient of expected return in terms of policy parameters. This gradient is formulated as an objective function, which is handed over to an optimizer with an auto-derivative support. A reasonable implementation of this algorithm is called VPG (Achiam 2018; Dhariwal et al. 2017). On the other hand, TRPO (Schulman et al. 2015a) and PPO (Schulman et al. 2017) use the surrogate objective. TRPO uses natural policy gradient with KL divergence constraints while PPO uses a simple clipping mechanism. Despite the simplicity, PPO often outperforms TRPO.

The performance of PPO can be improved by a variety of secondary optimization techniques. For example, value function clipping and reward scaling have been studied in (Engstrom et al. 2020), where these secondary optimizations are shown to be crucial for PPO’s performance. Since the success of model-free algorithms depends on its exploration and PPO tends to not sufficiently explore, Wang et al. (2019) proposed to adjust the clipping range within the trust region to encourage the exploration. The performance of PPO is also enormously influenced by its entropy. Ahmed et al. (2019) and Liu et al. (2019) present higher entropy can be utilized as a regularizer that produces smoother training results.

In this paper, we choose a PPO implementation with clipping and KL divergence break, and then study whether the original policy gradient theorem can be used instead of the surrogate objective, and still performance similar to PPO can be obtained.

In this section, we first define the objectives of VPG, PPO, and PPG, and provide the detailed discussions in the later sections.

When training a policy using VPG, it is difficult to match the performance of PPO simply by increasing the number of iteration as shown in Fig. 2. In order to achieve similar performance of PPO, a clipping mechanism similar to PPO is needed. In this section, we develop this clipping formula $L^{\scalebox{0.55}{$PPG$}}$ in (5), which we call PPG clipping.

In a policy update iteration, if $\hat{A}_{t}>0$, the policy parameters should be updated to increase the probability $\pi_{\theta}(a_{t}|s_{t})$, and if $\hat{A}_{t}<0$, the policy parameters should be updated in the direction of decreasing $\pi_{\theta}(a_{t}|s_{t})$. As illustrated in Fig. 4, the gradient of $L^{\scalebox{0.55}{$PPG$}}$ should have a downward $y$-component in the $2^{nd}$ and the $3^{rd}$ quadrants, and a upward $y$-component in the $1^{st}$ and the $4^{th}$ quadrants.

Suppose a sample is in the $2^{nd}$ quadrant. This means that the probability has been increased even though the action is disadvantageous. To penalize this undesirable update, we do not set a clipping upper bound. Then all samples in the $2^{nd}$ quadrant are used in the gradient ascent, and their probabilities are likely to be decreased for the next policy update. Now, suppose that a sample is in the $4^{th}$ quadrant. This represents that the probability of that advantageous action is decreased. Therefore, the objective function should not clip the sample to penalize all of such cases.

Samples in the $1^{st}$ quadrant indicate that probabilities of advantageous actions are increased. These samples are updated properly unless the amount of log probability increase is too much. Therefore, we set an upper bound to prevent the policy from being too deterministic. Samples above this bound will have zero gradient, and hence are not forced to increase the probability further. This procedure also similarly works for samples in the $3^{rd}$ quadrant. The lower bound in the $3^{rd}$ quadrant prevents excessive probability decreases by forcing the gradients to be zero for all samples that the difference log probabilities is lower than the lower bound. Notice that clipped samples have zero gradient, so the samples in the desirable areas are effectively removed from the next policy update iteration. Thus the clipping makes the per iteration batch size small and only focuses on the samples that should be reflected in the next update.

In practice, as shown in Fig. 1, at the beginning of the policy iteration, all samples are in the $x$ axis, and as iteration continues, samples start spreading into all the quadrants. Although a large number of samples exist in the $2^{nd}$ and the $4^{th}$ quadrants, the objective function (the red lines top far right image in Fig. 1) indeed increases. A clip line ($u_{b}=0.2$) is sometimes observable as shown in the iteration 80. The iteration can be continued further until both of the clip lines are clearly observable, however, this typically hampers the performance.

Finally, we would like to note that one can design a variety of different clipping strategies such as slopped clip lines. In our tests, we obtained mixed results depending on environments. Thus, we left the exploration as a future work.

As a policy update iteration continues, $\pi_{\theta}$ becomes different from $\pi_{\theta_{old}}$ as demonstrated in Fig. 1. The KL divergence, or relative entropy, measures the difference between $\pi_{\theta}$ and $\pi_{\theta_{old}}$ as

$D_{KL}$ can be approximated by the following $D_{MC}$, i.e., $D_{KL}\to D_{MC}$ as $n\to\infty$ (Hershey and Olsen 2007).

Therefore, the average $y$ coordinates of all samples in the advantage-policy plane is just the approximated KL divergence $D_{MC}$. The objective $L^{\scalebox{0.55}{$PPG$}}_{\scalebox{0.55}{$NCLIP$}}=\frac{1}{\hat{N}}\sum_{i,t=1}^{N,T_{i}}d_{i,t}(\theta)\hat{A}_{t}$ has some relation to $D_{MC}$. Although $d_{i,t}(\theta)$ and $\hat{A}_{t}$ are correlated, one can easily see some trivial relations. For example, $L^{\scalebox{0.55}{$PPG$}}_{\scalebox{0.55}{$NCLIP$}}\leq\max(\hat{A}_{t})D_{MC}$ in the right half plane where $\hat{A}_{t}>0$. Since $\hat{A}_{t}$ is normalized to have the standard deviation of 1, $\max(\hat{A}_{t})$ is not large, typically less than 10.

When the iteration starts, all samples are on the x-axis. As the current policy is updated, the samples gradually spread over to all quadrants, while moving to the first and third quadrants are encouraged by the gradient. In the left half plane, since samples are driven to have decreased probabilities by the optimization, the contribution to the approximate KL divergence, $\sum_{\hat{A}_{t}<0}d_{i,t}$ is likely to be negative. In the right half plane, $\sum_{\hat{A}_{t}<0}d_{i,t}$ is likely to be positive. Since the policy network outputs log probabilities, the log probability changes in both of the half planes are often similar in magnitude. Therefore, the two KL contributions tend to cancel, yielding a small approximate KL divergence as shown in Fig. 3.

Finally, we note that we have performed preliminary tests using a number of $D_{KL}$ constraint ideas such as $D_{KL}$ penalty added to the loss $L^{\scalebox{0.55}{$PPG$}}$ with or without various dead zones designs, for example, penalize only if $d_{i,t}>c$, where $c$ is a small constant. We also penalized using the average of exact KL divergence at each sample. In general, we found $D_{KL}$ is easy to control in PPG iterations, and the results were often positive, yet a principle that guides the choice among these methods and a study on general applicability on various environments need more thorough investigations, and hence, are left as future works.

We have shown that the surrogate objective can be replaced by a partial variation of the original policy gradient objective that has exactly the same gradient with the original policy gradient theorem based algorithm, VPG. Similar to PPO’s clipping mechanism that updates a policy within the trust region, PPG formulates the clipping in the log space. PPG shows the average returns comparable to PPO, the often higher entropy than PPO, and the smaller KL divergence changes than PPO.