Phase Re-service in Reinforcement Learning Traffic Signal Control

Consider traffic control at a single intersection with multiple incoming and outgoing roadways. Incoming vehicles travel on and are queued on incoming roadway lanes. Queues at signalized intersections are served by phases, each of which groups one or more non-conflicting turning movements. Phases are served with green signals in a pre-defined order known as the phase sequence, and a complete iteration of the sequence is a cycle [7].

At some intersections, particular left-turn movements face large demands during peak hours, such as the movement marked with red arrow in Fig. 1. Serving a specific left-turn movement such as a protected left turn twice in a cycle can effectively clear excessive left-turn queues. The second service, known as phase re-service, is typically pre-configured to follow the through movement in the same direction. For example, in Fig. 1, the protected movement served in the first phase may be served again in the third phase. Our work considers adding phase re-service into RL based signal controllers.

RL problems for signal control are often formulated as a Markov decision process (MDP), $\{\mathit{S},\mathit{A},r,\mathit{P},\gamma\}$. Here, $\mathit{S}\subseteq\mathbb{R}^{m}$ is the state space of the system in question with dimension $m$, $\mathit{A}\subseteq\mathbb{R}^{k}$ is the space of available actions with dimension $k$, $r:\mathit{S}\mapsto\mathbb{R}$ is a reward function, $\mathit{P}:\mathit{S}\times\mathit{A}\times\mathit{S}\mapsto\mathbb{R}$ is a probabilistic transition function, and $\gamma\in[0,1]$ is a discount factor.

Additionally, let $s_{t}\in\mathit{S}$ be the system state at time $t$, and $a_{t}\in\mathit{A}$ be the action taken at time $t$. The actions are chosen at each step from a policy $a_{t}\sim\pi_{\theta}(a_{t}|s_{t})$ parameterized by $\theta$. RL attempts to maximize the long-term discounted reward $\sum_{i=0}^{\infty}\gamma^{i}r_{i}(\cdot)$ by choosing an optimal policy, typically through the selection of optimal policy parameters. Signal control problems in which the action at each timestep is the phase to serve are naturally written as MDPs, which can then be solved with RL.

In this work we formulate a traffic signal control problem as a Semi-MDP, which is an extension of MDP which includes a varying transition time between states. The time between transitions is called the sojourn time, denoted $j_{t}$, and we incorporate this into the state transition as $\mathit{P}(s_{t+1},j_{t}|s_{t},a_{t})$, meaning that the transition probability takes both the next state and the transition time to it into account. Signal control problems in which the action is the temporal duration to serve the next phase are naturally written as SMDPs.

The maximization of the long-term discounted reward is realized by converging the state value to Bellman optimality [17, 18], which for an SMDP is written as:

Let $n$ be an index on signal cycles. A set of important speed properties associated with shock waves at intersections is as follows [16, 14]:

where $k_{m}$, $k_{j}$, and $q_{m}$ are critical density, jam density, and saturation flow. Additionally, $k_{a}^{n}$ and $q_{a}^{n}$ are the density and flow for arriving vehicles at each cycle $n$. Modern sensors can directly measure the number of vehicles over a time window as the flow, and the average speed of these counted vehicles. Then the density is obtained from dividing the flow by the average vehicle speed. Let $v_{1}^{n}$ be the queue expansion speed at the red signal, and let $v_{3}^{n}$ be the speed at which the end of queue moves forward to the stop line at the green signal, $v_{2}$ be the vehicle discharge speed from the queue at the green signal, and $v_{4}$ be the speed at which the residual of queued vehicles forms a new queue when the green signal ends (only for saturated movements). Fig. 2 shows graphically these different quantities.

The traffic control problem can be formulated as an SMDP as follows:

State $s$. The state consists of the count of queuing vehicles in each incoming lane, the count of non-stopping vehicles in each incoming lane, as well as the number of phases until service. This information is commonly available from sensor data [4].

Action $a$. The action taken by the RL agent is the duration of the next phase in a predefined cycle. To account for min/max green time constraints, actions are normalized within the range $[-1,1]$, achieved via a tanh function, and subsequently linearly mapped to actual phase duration as follows:

where $\tilde{a}_{p}$ and $a_{p}$ are the actual and normalized duration time for phase $p$, and $\sigma_{p}^{+},\sigma_{p}^{-}$ are the maximum and minimum green time also for $p$. For notational simplicity, we refer to $a$ as the actual duration time in the rest of this paper.

Sojourn time $j$. The sojourn time in this context is the time from applying one phase duration (the action) until the next point of applying another phase duration (the next action). The sojourn time is typically the phase duration plus the yellow signal transition time. However, if reservice is applied, the time when the next action is executed will be delayed by the re-service phase, which extends the sojourn time accordingly.

Maximum queue estimation. First, we give a maximum queue estimation technique for this problem setup, which is slightly altered form of the queue estimation techniques presented in [19].

We define the following time quantities related to queue length estimation:

• •

  $T^{n}_{g}$: The start time when the protected movement in the $n$-th cycle is served with a green signal¹¹1$T$ is the actual simulation time in seconds. In contrast, $t$ is the timestep for the agent’s decision-making..

• •

  $T^{n+1}_{r}$: The end time of the protected movement in the $n$-th cycle, which is also the $n+1$ cycle’s start time.

• •

  $T_{re}^{n}$: The time at which whether to re-service is decided for the $(n)$-th cycle.

Let $X(T)$ refer to the queue length at time $T$. Additionally we define:

as the time between assessing for re-service and the next regular green signal.

At a given $T^{n}_{re}$, $\Delta T^{n}$ could be used to perform queue length estimation, however it is not known in real-time. Instead, let $\Delta\tilde{T}^{n}$ be a real-time estimate. In this work we estimate $\Delta\tilde{T}^{n}$ at a given $T^{n}_{re}$ by using the running average of the true $\Delta T^{n}$ from prior cycles. In implementation we use 2 prior cycles.

Let $L_{max}^{n+1}$ be the forecasted maximum queue length in the next cycle, if no re-service is applied. We calculate this as follows:

From $L_{max}^{n+1}$, whether or not to apply phase re-service is decided. The different quantities covered here are shown graphically in Fig. 2.

Re-service duration calculation. First, whether or not to execute the re-service is decided via

where $\Theta_{X}$ is a threshold on queue length.

Next, we calculate the maximum queue length if re-service is applied as follows:

We then calculate the duration of the re-service as follows:

In Algorithm 1 our proposed training algorithm is presented. The algorithm joins queue estimation based phase re-service with our SMDP formulation and standard policy optimization techniques. In particular, proximal policy optimization with a generalized advantage estimator is used. Minor alterations to the PPO algorithm were made to adapt the algorithm to the SMDP approach.

We model the two intersection types in SUMO using the default parameters to control the vehicle dynamics. At each intersection (Fig. 3), we define five time varying demand profiles, which are shown in Appendix A.

The [min, max] green time constraints for each phase are [5, 30], [5, 40], [10, 25], [5, 45] for the freeway ramp intersection, and [5, 25], [5, 70], [5, 25], [5, 25], [5, 70] for the four-leg intersection, all in seconds. Green time constraints for the re-service phases are in Italic. The yellow signal is set at five seconds. The simulation time for all scenarios is one hour. The maximum distance for vehicle detection $\Theta_{m}$ is set at 250 m, while the threshold for re-service $\Theta_{X}$ is set at 200 m. The shock wave parameters are estimated from the simulation model parameters as: $k_{j}=133.3$ veh/km, $k_{m}=50$ veh/km, and $q_{m}=1550$ veh/h.

The actor network has a single 128-neuron hidden layer and a tanh activation function for output. Similarly, the critic network also has a hidden layer of 128 neurons but the activation function is ReLU. Both networks are independent, i.e., do not share common neurons. The sampled action value from the actor’s stochastic policy is also activated by tanh. The re-service coefficient is set as $\zeta=0.7$. For hyperparameters in PPO, we follow the guidelines in [22] and set the loss clipping hyperparameter $\epsilon=0.1$, the long-term reward discount factor $\gamma=0.995$, the multi-step weighting factor in advantage estimation $\lambda=0.99$, the learning rate at 2.5e-4, the minibatch size at 256, the update interval of 1200 transitions, and 20 epochs per update. All agents have the same hyperparameters.

The step-average training results for 500 episodes in two intersections are shown in Fig. 4. Demand 3 of both intersections are training demand profiles. It shows that adding phase re-service as part of the environmental transition does not significantly affect the speed of convergence, and in both intersections, the re-service can further maximize the rewards. The freeway ramps are more benefited from phase re-service probably due to a large percentage of re-serviced vehicles in the total vehicle demands.

For each intersection, the agent is trained five times with demand profile 3. The best-performing agent in the last episode of the training stage is used for testing. We run each testing scenario (consisting of an intersection and a demand profile) 20 times and report the statistics. Metrics including throughput per hour, vehicle delay per trip, and the number of stops experienced per trip are compared and summarized in Table I and Table II. All metrics are directly calculated within SUMO. A baseline non-RL algorithm called SOTL [21] is evaluated for comparison purposes. SOTL determines phase switching based on the count of arriving vehicles showing excellent performance in previous studies [23]. Below we analyze different traffic performance metrics.

Vehicle delay: We report the average delay per vehicle. In 9 out of 10 scenarios tested, vehicle delay is significantly reduced with re-service compared to scenarios without re-service and to the non-RL baseline. Numerically, compared to RL without re-service, the proposed approach achieves an improvement in terms of the mean and standard deviation of vehicle delay ranging from -1% to 29.95% and 9.95% to 59.21%, respectively. SOTL, in contrast, only exhibits a 4.2% mean vehicle delay improvement with 14.29% higher variance in Table I Demand 1. Further, we present the density histogram of vehicle delays for freeway ramps Demand 4 scenario which benefits most from the phase re-service in Fig. 5. RL with re-service method is able to reshape the distribution with a long tail, i.e., significantly delayed vehicles, back to a more centralized one, leading to a lower average delay and smaller standard deviation.

Number of stops: The average number of vehicle stops is reported as a measure of mobility. The phase re-service significantly reduces the average number of stops and the variance of number of stops over all scenarios. The percentage of improvement reaches as high as 26.05% and 45.77% in terms of the mean and standard deviation (Table I, Demand 4 scenario).

Percentage of re-service cycles: This metric calculates the proportion of cycles incorporating phase re-service in each scenario. The results from Table II Demand 3 suggest that the re-service cycle as low as 4.0% can reduce the average vehicle delay and number of stops by 9.75% and 5.47%, and moreover, lower the variance of both metrics by 23.23% and 13.58%. The highest re-service rate reaches 45.3% (Table I, Demand 4 scenario) which contributes to the largest improvements of both vehicle delay and number of stops.

Throughput: The average throughput is also calculated for each scenario. The RL algorithm with and without phase re-service realizes similar performance in maximizing the throughput, which is also comparable to the baseline.

We list the trip-level evaluation metrics by vehicle movements of the RL algorithm with and without phase re-service for the scenarios with the lowest and highest re-service penetration. They are summarized in Table III. Through and right-turn movements are grouped together in the four-leg intersection as they utilize the same phases.

As expected, we observe that movements only in regular phases are delayed and stop more since the phase re-service takes additional time. Nevertheless, the additional delay is mild. EE and SE movements in Table III(a) are on average more delayed by 5.6s and 13.5s; non-protected movements in Table III(b) experience more delays from 3.8s to 14.5s. In contrast, the improvement for the protected movement is substantial: 102.5s less delay and 1.454 fewer stops in Table III(a), 79.98s less delay, and 0.45 fewer stops in Table III(b).