Optimizing Traffic Lights with Multi-agent Deep Reinforcement Learning and V2X communication

Existing traffic signal management is performed: either leveraging limited real-time traffic information or by fixed periodic traffic signal timings [1]. This information is widely obtained from underlying inductive loop detectors. However, this input is processed in a limited domain to estimate better duration of red/green signals. The advances in mobile communication networks and sensors technology have made possible to obtain real-time traffic information [2]. An artificial brain can be implemented with deep reinforcement learning (DRL). DRL is based on three main components: states in the environment, action space and the scalar reward from each action [3]. A popular success of DRL is AlphaGo [4], and its successor AlphaGo Zero [5]. The main goal is to maximize the reward by choosing the best actions.

In some research works, state is defined as the number of vehicles waiting at an intersection or the waiting queue length [6], [7]. However, it is investigated by [8] that real traffic environment cannot be fully captured leveraging the number of waiting vehicles or the waiting queue length. Thanks to the rapid development of deep learning, large state problems have been addressed with deep neural networks paradigm [9]. In [10], [11] authors have proposed to resolve traffic control problem with DRL. However two limitations exist in the current studies: 1) fixed time intervals of traffic lights, which is not efficient in some studies; 2) random sequences of traffic signals, which may cause safety and comfort issues. In [12] the authors have controlled duration in a cycle based on information extracted from vehicles and sensors networks which can reduce average waiting time to 20%.

In this letter we investigate for the first time multiple experienced traffic operators to control traffic in each step at multiple traffic lights. This idea assumes that control process can be modeled as a Markov Decision Process (MDP). The system experiences the control strategy based on the MDP by trial and error. Recently, a Q-learning based method is proposed by [18] showing better performance than fixed period policy. An linear function is proposed to achieve more effective traffic flow management with a high traffic flow [19]. But neither tabular Q learning nor linear function methods could support the increasing size of traffic state space and accurate estimation of Q value in a real scenario [20]. Gao et al. [21] has proposed a DRL method with a change in cumulative staying time as a reward. A CNN is employed to map states to rewards [21]. H. Jiang has analyzed nonzero-sum games with multi-players by using adaptive dynamic programming (ADP) [22]. Li et al. proposed a single intersection control method based on DQN to focus on a local optimum [23]. Pol et al. combined DQN with a max-plus method to achieve cooperation between traffic lights [25]. Xiaoyuan Liang et al. proposed a model incorporating multiple optimization elements for traffic lights and the simulation on SUMO has shown efficiency of the model [12]. However, most of the works focused on single-agent controlling signals for an intersection.

The proposed system is shown in Fig. 1. The model for each agent is based on four items $\langle S,A,R,P\rangle$. Let $S$ is possible state space, and $s$ is a state ($s\in S$). $A$ is possible action space, and $a$ is an action ($a\in A$) and $R$ is reward space. Let $P$ is the transition probability function space from one state to next state. A series of consequent actions is policy $\pi$. Learning an optimal policy to maximize the cumulative expected reward is main goal. An agent at state $s$ takes an action $a$ to reach next state $s^{\prime}$, gets the reward $r$.

A four-tuple represents this situation as $\langle$$s$,$a$,$r$,$s^{\prime}$$\rangle$. The state transition occurs at a discrete step $n$ in $\pi$. Let $Q(s,a)$ is cumulative reward function in future by executing an action $a$ at state $s$. Let $r_{n}$ is reward at $n$th step, and (1) gives $Q^{\pi}(s,a)$ for $\pi$:

The parameter $\gamma$ is a discount factor in $[0,1)$, decides how much importance should be given to recent and future rewards. The optimal policy $\pi^{*}$ can be acquired through several episodes in the learning process. Calculation of optimal $Q(s,a)$ is based on the optimal $Q$ values of the succeeding states represented by the Bellman optimality (2):

It can be solved by dynamic programming keeping finite states for less computational burden. However, the $Q$ values can also be estimated by a function $\theta$ for larger number of states.

The scenario is a 6$\times$6 intersections scenario with 2, 4, 8 multi-agents are selected as shown in Fig. 3. There are a total of 128 vehicles that randomly enter the scenario from various intersections. The parameters used for training the deep neural network are in Table I. Figure 4 shows the average waiting cars at 8 intersections for the scenario. In Fig. 4(a) we have kept 30 s duration for each phase. Similarly in Fig. 4(b) the duration is 40 s. It is noted that Agent IDs are reference intersection ID. The agents are not taking actions in the Fig. 5. It is observed that traffic at intersection marked with ID 22 has a higher (greater than 3) average number of waiting cars in both 30 s and 40 s cases. All intersections have more than 1 average waiting cars during all episodes.

We divide the parameters for evaluation as: model parameters and traffic parameters. The model is trained iteratively in each episode for 500 s. The reward and $w_{{i_{a}}}$ are aggregated in each episode. The performance of agents are compared for $r_{a_{1}}$, $r_{a_{2}}$, $r_{a_{3}}$ and $r_{a_{4}}$. The rewards in Fig. 5(a) are for agents with IDs 8 and 10 and evaluated with $r_{a_{1}}$. Similarly Fig. 5(b) shows corresponding rewards evaluated by using $r_{a_{2}}$. The shared-rewards $r_{a_{3}}$ and $r_{a_{4}}$ are respectively presented in Fig. 5(c) and Fig. 5(d). The effects of using different reward functions are shown below for each case in Fig. 5(e)-(h). It is observed that in case 1 both agents have performed better in maximizing their rewards and minimizing the number of waiting cars in their connected lanes. Agent 10 performed better than agent 8 for the cases 2, 3 and 4. The reward for the agent 10 is higher than agent 8 and the corresponding results of average waiting cars also show less average waiting cars for the agent 10 as compared to agent 8.

Figure 6 shows results of 4 agents with IDs 8, 10, 15, and 17 by experimenting the same four cases using the similar scenario. It is observed in Fig. 6(a) that agent 10 performs better than other agents as the average number of waiting cars under agent 10 are less than others. In Fig. 6(b) all agents try to perform better by fluctuating their rewards. These fluctuations are also reflected in the number of waiting cars as shown in Fig. 6(f). Interesting similar rewards are observed in the shared-rewards cases (3,4) in Fig 6(c) and Fig 6(d).

Average number of waiting cars are also similar as shown in Fig. 6(g) and Fig. 6(h). The results for the 8 agents are shown in Fig. 7 with IDs 8,10,15, 17, 20, 22, 27, and 29. The case 1 outperforms other cases in Fig 7(a). All agents tried to maximize their rewards showing better results in Fig. 7(a)-(b) and Fig. 7(e)-(f). Peak average waiting cars in Fig. 7 (f) is 2.1 as compared to 3.9 (see Fig. 4(b)) after 15 episodes, which means a decrease of 41.5 $\%$. On the other hand all rewards become same for the agents as expected. However, the shared-reward schemes performed poorly as compare to unshared-rewards. The reason behind this low performance is failure to achieve better actions that could maximize the shared reward. The actions of one agent produces the environment conditions that could negatively disturb the rewards for other agents. Agents 8, 10, 17, and 29 relatively performed better even with reduced shared rewards.

In this article we have presented the performance of deep reinforcement learning under multi-agent V2X driven traffic control system. We have observed that multi-agents with individual rewards considering waiting number of cars is a better choice as compared to the average waiting time. On the other hand shared-rewards based cases do not perform better. Shared-rewards make the situation more competitive. This competition should be further investigated using other techniques of deep reinforcement learning. We have also observed that for larger number of agents, the reward based on waiting time is the better choice.