Curriculum Proximal Policy Optimization with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections

The problem to be solved in this work is to control the ego vehicle to traverse an unsignalized four-way intersection and reach the goal position. Furthermore, each road consists of two lanes. The task scenario is shown in Fig. 1.

We assume that the ego vehicle always starts from a random position (denoted by a solid red vehicle) in the lower zone of the intersection, and the goal position of the ego vehicle (denoted by a semi-transparent red vehicle) is also randomly generated within the left, upper, and right zones. There are several surrounding vehicles driving from other lanes towards different target lanes. They will react to the behavior of the ego vehicle. Without loss of generality, we assume that the position and velocity information of surrounding vehicles can be accessed by the ego vehicle. But the information about the driving intention of surrounding vehicles is unknown to the ego vehicle, which increases the difficulties of this task. The objective of this task is to generate a sequence of actions that enables the ego vehicle to expeditiously approach the target point while ensuring collision avoidance with surrounding vehicles and staying within the road boundaries.

In this work, we frame the agent’s learning objective as the optimal control of a Markov Decision Process by defining state space, action space, state transition dynamics, reward function, and discount factor. Then the RL problem can be represented by a tuple $\mathcal{E}=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$.

State space $\mathcal{S}$: In this scenario, the state space $\mathcal{S}$ is explicitly defined by $Highway\_Env$ environment, which consists of state matrices $\mathbf{S}_{t}$. At each timestep $t$, the agent observes the kinematic features of vehicles in the intersection, so the state matrix can be defined as follows,

where $N$ denotes the total number of vehicles in the intersection. The first row of $\mathbf{S}_{t}$, i.e., $(\mathbf{s}_{t}^{0})^{T}$, is a vector consisting of the kinematic features of the ego vehicle, while other rows of $\mathbf{S}_{t}$, i.e., $(\mathbf{s}_{t}^{i})^{T}\ (i=1,2,...,N)$, represent vectors of the kinematic features of the surrounding vehicles. The vector of the kinematic features is defined as follows,

where $x_{t}^{i}$ and $y_{t}^{i}$ are the vehicle’s current position in the world coordinate system, respectively; $v_{x,t}^{i}$ and $v_{y,t}^{i}$ are the speed of the vehicle along the X-axis and Y-axis, respectively; $\psi_{t}^{i}$ is the heading angle of the vehicle at timestep $t$.

Action space $\mathcal{A}$: Similar to the human driver’s operation in driving, the action space of the agent consists of five basic discrete actions as follows,

where $A^{0}$ and $A^{2}$ are the left and right lane-changing action, respectively; $A^{1}$ is the motion keeping action; $A^{3}$ and $A^{4}$ represent the deceleration and acceleration action, respectively.

The vehicle is controlled by two low-level controllers, the longitudinal and lateral controllers, which convert discrete actions (3) to the continuous control input $[a,\delta]^{T}$ of the vehicle, where $a$ and $\delta$ are the acceleration and steering angle, respectively. Considering the physical limitations, the maximum acceleration and steering angle are set as 8 m/s² and $45$ degrees, respectively.

State transition dynamics $\mathcal{P}(\mathbf{S}_{t+1}|\mathbf{S}_{t},a_{t})$: The transition function $\mathcal{P}$ defines the transition of environment state, which follows the Markov transition distribution. The next state generated by $\mathcal{P}$ is related to the current state $\mathbf{S}_{t}$ and the applied action $a_{t}\in\mathcal{A}$. The transition dynamics $\mathcal{P}(\mathbf{S}_{t+1}|\mathbf{S}_{t},a_{t})$ is implicitly defined by $Highway\_Env$ environment and unknown for the agent.

Reward function $\mathcal{R}$: The reward function assigns a positive reward for successfully completing an episode and for maintaining survival. It penalizes collisions, out-of-the-road, and lane-changing behaviors. When there are few vehicles on the road, the acceleration behaviors will be rewarded, and vice versa. The details about the structure of the reward function will be introduced in Section III-C.

Discount factor $\gamma\in(0,1)$: The future reward is accumulated with a discount factor $\gamma$.

PPO is a model-free RL framework to solve the sequential decision-making problem under uncertainties. It alternatively constructs a clipped surrogate objective to replace the original function, which generates a lower bound on the unclipped objective and avoids the incentive for an excessively large policy update. Therefore, PPO algorithm facilitates the learning of policies in a faster and more efficient way. Specifically, the objective function in the PPO algorithm is shown as follows,

where $\rho_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}$ is the probability ratio between the new policy and old policy, $\hat{A}_{t}$ is the estimated advantage function at timestep $t$, $\varepsilon$ is a hyperparameter.

The intuitive idea is to change the value of the hyperparameter $\varepsilon$ in the different training periods. Empirically, we set $\mathbf{\varepsilon}=\left\{0.25,0.2,0.15\right\}$. During the beginning stage of training, a large parameter $\varepsilon_{1}=0.25$ is used for the rough exploration, which is then adjusted to the second largest parameter $\varepsilon_{2}=0.2$ during the middle stage, and finally adjusted to a smaller parameter $\varepsilon_{3}=0.15$ during the later stage. However, determining when to make adjustments to the parameters is a problem.

In intersection-crossing tasks, the ego vehicle needs to interact with a variable number of interactive surrounding vehicles with different driving behaviors from the other three directions. Therefore, these scenarios are rather complex. It is difficult to get a satisfactory driving policy by directly deploying the PPO algorithm in these high-dynamic scenarios. Here, we introduce stage-based curriculum learning technology to generate a task sequence with increasing complexity for training acceleration and better generalization. Besides, the clipping parameter can be adjusted when switching the curriculum, which addresses the issue of when to change the clipping parameter mentioned before.

We generate a curriculum sequence with three stages, which is represented as $\mathbf{\Omega}=\left\{\Omega_{1},\Omega_{2},\Omega_{3}\right\}$. The curriculum sequence is designed for different goals with increasing complexity.

Curriculum 1: Intersection without surrounding vehicles. In stage 1, which is denoted as $\Omega_{1}$, there is only the ego vehicle in the intersection. The objective of this curriculum is to learn a transferable nominal policy to obtain the nominal policy $\pi_{1}$ that can find an action sequence to the goal point. The ego agent is guided with empirically designed rewards in this curriculum stage, which can decrease the exploration time in the whole action space and avoid local optimum with poor generalization. In this curriculum, the hyperparameter of the clipped function is set as $\varepsilon_{1}$.

Curriculum 2: Intersection with a few vehicles. In the second stage, we load the policy trained in Curriculum 1 as the initial policy of this stage for the following training process. In this curriculum, there are a few surrounding vehicles in the intersection. We aim to train the nominal policy to obtain a new policy $\pi_{2}$ with the preliminary obstacle avoidance ability by maximizing the intersection-crossing reward in this stage. The hyperparameter is switched to $\varepsilon_{2}$.

Curriculum 3: Intersection with numerous vehicles. In the third stage, we load the policy trained in Curriculum 2 as the initial policy. In this stage, there are numerous surrounding vehicles in the intersection. The objective of this curriculum is to train the previous policy to obtain the optimal policy $\pi^{*}$ with better obstacle avoidance ability in a more complex environment. In the preceding episodes of this stage, the parameter $\varepsilon$ remains at 0.2, and then transits to $\varepsilon_{3}$ to continue the course training.

For the RL-based method, the design of the reward is essential for the success of policy training. An inappropriate reward function not only slows down the speed of training but also leads to a trained policy with poor performance. Therefore, it is challenging to guide the agent to obtain a satisfactory driving strategy in a complex scenario. In this work, the reward is smartly designed for intersection scenarios with different densities.

Considering the complexity of the target scenario, a comprehensive reward function is designed as follows,

where $r_{succ}$ and $r_{l}$ are the reward for successfully completing tasks and surviving in the task, respectively; $r_{colli},r_{TO},r_{OfR},$ and $r_{LC}$ are the penalty of collision with surrounding vehicles, time-out, out-of-the-road boundary, and lane-changing behavior, respectively.

To sum up, the proposed CPPO framework with stage-decaying clipping is summarized in Algorithm 1.

The experiments are conducted on the Windows 11 system environment with a 3.90 GHz AMD Ryzen 5 5600G CPU. The task scenarios are constructed based on $Highway\_Env$ environment, where each road is a bi-lane carriageway. We use fully-connected networks with 1 hidden layer of 128 units (action network) and 64 units (critic network) to represent policies. The neural network is constructed in PyTorch [25] and trained with an Adam optimizer [26]. The MDP is solved using the proposed CPPO framework and the standard PPO method with two different fixed clipping parameters. The simulation frequency is set as $15$ Hz. The hyperparameters for network training are listed in Table  I. Here, we compare the proposed method, CPPO, and two baseline methods with different clipping parameters ($\varepsilon=0.15,0.25$). For the sake of fairness, other network parameters of these methods are the same.

Then we test all trained policies in intersection scenarios with different numbers $N_{sv}$ of surrounding vehicles, whose behaviors are characterized by the intelligent driver model (IDM) [27]. We record the success rate, collision rate, time-out rate, and out-of-road rate, respectively. By conducting these simulations, we can evaluate the generalization performance of these trained policies.

To illustrate the effectiveness of the proposed framework, the training time is listed in Table II for comparison. It is obvious that CPPO has a much faster training speed than the two baseline methods. Specifically, the training speed of CPPO is $47.2\%$ faster than PPO₁, and $40.2\%$ faster than PPO₂.

The change in reward during the training process is shown in Fig. 2. According to these three learning curves, we can find that the CPPO agent initially receives the least reward, which is because that the reward function is positively correlated with the number of vehicles in the environment upon successful task completion. However, as CPPO’s policy network converges to the optimal policy in the final stage, its reward curve surpasses that of the other two baseline methods. As the baseline algorithm PPO₁, with the parameter $\varepsilon=0.15$, is deployed directly in a complex environment for learning, its reward curve exhibits minor fluctuations in the later stages of training and is lower than that of the CPPO algorithm. For the agent trained by the baseline PPO₂ with $\varepsilon=0.25$, its reward curve exhibits a rapid increase initially. However, due to the large parameter $\varepsilon$, it oscillates significantly in later episodes. Although its reward curve ends up resembling that of PPO₁, the performance of its policy may be inferior to that of PPO₁. Above results illustrate that the introduction of stage learning allows for a more efficient sampling process, leading to a faster training speed and better convergence compared to baseline methods.

To further demonstrate the superiority of CPPO in unsignalized intersection scenarios, comparative simulations are conducted. We test policies obtained by three methods in intersection scenarios with different numbers of surrounding vehicles. Each method is retested 200 times in each scenario. Among all testing results attained by the proposed CPPO method at the unsignalized intersection, we pick up the results from a left-turn task for demonstration, and the details are presented in Fig. 3. In this demonstration, the ego vehicle is generated on the right lane of the lower zone, and its goal lane is the left lane in the left zone. We can observe that the ego vehicle exhibits a safe interaction behavior of decelerating and steering left when encountering a surrounding vehicle approaching from its left side in Fig. 3. Specifically, in the first snapshot, the ego vehicle drives at a constant speed from the right lane in the lower zone towards the central area of the intersection, preparing for a left turn. Then, in the second snapshot, the ego vehicle perceives that it is getting close to a surrounding vehicle ahead, and the vehicle ahead shows no signs of slowing down to yield. As a result, the ego vehicle decelerates and steers left to avoid a collision. Afterwards, in the third snapshot, the closest surrounding vehicle in the previous snapshot has moved away from the ego vehicle. The ego vehicle perceives that there are no other vehicles that could potentially collide with itself. Therefore, it adjusts its heading angle, accelerates towards the target lane, and continues driving until the completion of the unsignalized intersection task.

The success rates of the three methods in all evaluation scenarios are shown in Fig. 4. It is evident that the CPPO method achieves the highest overall task success rate. While its success rate decreases with the increasing complexity of the environment, it remains higher than that of the other two baseline methods. It indicates that the proposed method has better generalization performance. Furthermore, except for the scenario where there are no surrounding vehicles, PPO₁ achieves a slightly higher task success rate than PPO₂. This is because a smaller parameter $\varepsilon$ enables a search for better policies. Besides, it is noted that both two baseline methods have a low task success rate in the scenario where there is only the ego vehicle.

In addition, we have summarized the results of the success rate, collision rate, and timeout rate of tests with $N_{sv}=0,2,4$ surrounding vehicles in Table III. From this statistical result, we can observe that both baseline methods exhibit a large number of timeouts in testing, $44.5\%$ and $30.5\%$ for PPO₁ and PPO₂, where there is no surrounding vehicles. This suggests that directly deploying the agent in a complex environment for training may cause the policy to become stuck in a local optimum. For other task scenarios with surrounding vehicles, two baseline methods did not exhibit timeouts in testing results. Because of the integration of the curriculum sequence, there is no timeout case happening for the CPPO. Therefore, the introduction of the curriculum learning technologies enables the agent to converge to a better optimum compared to those agents trained directly.