A Dynamic Penalty Function Approach for Constraint-Handling in Reinforcement Learning

Let us assume that we have $\upsilon$ inequality constraints including the inequality constraints on the state space boundary. The RL agent gets a penalty value for each constraint violation. A constraint aggregation is needed for unbiased constraint handling when multiple constraints exist. The KS function is a widely used constraint aggregation method for gradient-based optimization (Kreisselmeier and Steinhauser (1980)). Although RL is not a gradient-based optimization algorithm, we can effectively use this method with a high aggregation parameter, $\rho$, making the error from the aggregation as small as possible (Poon and Martins (2007)). For constraints $g_{i}(x_{t},u_{t})\leq 0,i=1,...,\nu$, the aggregate function $KS[\bm{g(x,u)}]$ is:

where $g_{max}$ is the maximum of all constraints evaluated at the given state-action pair.

Using the KS function, the reward function with DP term can be defined as follows:

where $\mu$ is the penalty factor which defines the slope of the penalty function. The right-hand side of 3b is calculated only when the aggregate constraint is violated. The DP function approach starts with a low value of $\mu$, which facilitates the initial training of the NN, and gradually increases it to a large value by multiplying the DP update parameter $c$ so that the slope becomes large to prevent constraint violations (Fig. 1 illustrates this approach). The procedure can be described as:

1. 1.

   Set $\mu:=\mu_{min}$.

2. 2.

   When the loss value of the NN is less than $(100-\alpha)\%$ of the maximum loss value after the parameter update, set $\mu:=c\mu$ where $c$ is some constant larger than 1

3. 3.

   Repeat step (2) until $\mu\geq\mu_{max}$

4. 4.

   After reaching $\mu\geq\mu_{max}$ set $\mu:=\mu_{max}$

5. 5.

   Continue the training until an optimal policy is found.

A feed-forward NN is the most widely used function approximator for the value function or Q-function approximation. The agent uses the output value or the gradient of the NN to update the optimal policy. Therefore, accuracy of the approximation is important for a fast and stable agent training. We tested the regression accuracy with a priori fixed uniform penalty and linear penalty, which are the popular choices for the penalty function against the proposed dynamically varied penalty function. The test value function $V(x)$ is chosen as follows:

We assumed that this problem has inequality constraints as $-5\leq x\leq 5$ and the penalty functions are assigned as in table 1 where the $\mu_{min}=0.1$, $\mu_{max}=50$ and $c=2$. This artificially chosen value function is not exactly in the same form as in the optimal control problem where the value should be calculated with respect to the cumulative rewards. Nevertheless, this analysis should be useful to understand the tendencies of the NN approximation during the agent training with the different penalty terms.

To examine the NN training performance within an RL environment, the training environment is set similarly as in RL. In every episode, 20 $x$ points are randomly sampled from -10 to 10 and the environment calculates the $R(x)$ value ($R(x)=l(x)+p(x)$) with the assigned objective function (4) and the penalty function (table 1). Then, 20 samples are stored in a replay buffer and the NN is trained with a batch of 64 data randomly sampled from the replay buffer up to 500 episodes.

Fig. 2 shows the output value of the NN during the training process. As shown in Fig. 2a, the NN approximation performance is poor at the boundary points, which causes a bias in the estimate inside the feasible region. The linear penalty case also gives a biased estimate inside the feasible region during the training and the maximum loss function is much higher than that of the DP case, as shown in Fig. 3. With the proposed scheme of dynamically varying the penalty during traning, the function is well approximated near the constraint boundary, which can reduce the bias and loss. Fig. 3 shows that the loss of NN with the DP function approach is increased during the penalty factor update period but is decreased drastically afterward. The final loss is the lowest in the case of DP, which implies the best approximation performance. From this simple test, we see the potential benefit of using a dynamically varied penalty in training the NN to learn the value function with a large penalty function.

To further examine the effectiveness of the DP approach, a simple vehicle control problem described in (5) is solved with RL. We modified the problem from Grüne and Pannek (2017) to reduce the size of the feasible region and make the starting point uncertain as described in (5f). The objective is to minimize the running cost (5a). $x_{1}(k)$ is the position of a vehicle at time k, $x_{2}(k)$ is its velocity and $u(k)$ is its acceleration.

We applied the Deep-Q-Network (DQN) algorithm which uses a deep NN to approximate the Q-function and chooses the $\epsilon$-greedy policy (Mnih et al. (2013)). For the training, we defined the discrete action space $u\in\{-0.25,-0.2375,-0.225,...,0.2375,0.25\}$ and each episode was set for a time duration of 20min with 1min control interval. The deep NN consisted of three hidden layers with 64 nodes of the ReLU activation function. For constraint aggregation, (5c) and (5d) are decomposed as shown in (6) and aggregated to a $KS[\bm{g}]$ function of (2) with $\rho=50$.

The DP was updated when the loss became lower than 30% of the maximum loss ($\alpha=70$) and the penalty factor was doubled in each update ($c=2$), starting from $\mu_{min}=0.05$ up to $\mu_{max}=20$. The value for the uniform penalty function and the penalty factor for the linear penalty were both set to 20 as shown in table 2. The training comprised 2000 episodes and the initial point was randomly selected inside the feasible region to start each episode. To test the agent’s behavior during training, we tested the policy after every 100 episodes without exploration. In addition, for a rigorous comparison, we trained a set of 100 agents in the environment set with 100 different random seeds, which introduced randomness to exploration, NN initialization, buffer sampling, etc.

Table 3 shows that the agents trained with the uniform penalty and the linear penalty could find sufficiently feasible policies only in 42 and 48 cases out of the 100 random cases, respectively. Here, ‘sufficient feasible policy’ means that the policy does not violate all the constraints and achieves a sufficient low cost value. The agent trained with the uniform penalty incurred the highest average cost. Also notable is that the agent could find feasible policies in 82 cases, but almost a half of them were unable to achieve sufficiently low cost. According to table 4, the uniform penalty seems favorable for a rapid training, but the final performance is not satisfactory. These results are likely due to the interference of the bias in the estimate inside the feasible region. The agent trained with the linear penalty function performed poorly due to having inaccurate approximate values for the value function during training. This consequently resulted in the policy being updated in a wrong direction, or converging to sub-optimal policies.

The use of the DP approach was effective in finding a sufficient feasible policy rapidly and consistently. The agent trained with the DP approach could find sufficient feasible policies in 83 cases, and the average cost of that cases was lower than the results of the agent trained with other penalty functions. We also calculated the average degree of violation by averaging the positive KS function values from the converged infeasible policy. As shown in Table 3, when the agent was trained with the DP approach, the degree of violation of the infeasible policies was much lower compared to the other approaches. This means that even the infeasible policies found in the 7 cases, nearly satisfied the constraints. Table 4 also shows that sufficiently feasible policies were found within 1500 episodes in 39 cases, which is comparable to the training with the uniform penalty. Fig. 4 shows the policy improvement during training with the DP function.