Learning Generalizable Multi-Lane Mixed-Autonomy Behaviors in Single Lane Representations of Traffic

Ring roads model the behavior of $n$ vehicles in a circular track of length $L$ with periodic boundary conditions. Within these models, the evolution of the state (or position $x_{i},i\in{1,\dots,n}$) of each vehicle is represented via the system of ordinary differential equations:

where $f(\cdot)$ is a car-following model that mimics the acceleration of human driver $i$ as a function of their ego speed $v_{i}(t)$, the speed of their leader, and the bumper-to-bumper headway between the two $h_{i}(t)$, and $x_{0}^{i}$ and $v_{0}^{i}$ are the initial position and speed of the vehicle, respectively. TODO: example.

Single-lane ring roads have served as a proxy for studying traffic instabilities for several decades. Within these problems, instabilities appear as a result of behaviors inherent to human driving. In particular, due to string instabilities Swaroop and Hedrick (1996) in human driving behaviors, small fluctuations in driving speeds produce higher amplitude oscillations by following vehicles as they break harder to avoid unsafe settings. This response perturbs the system from its desirable uniform flow equilibrium, in which all vehicles are equidistant and drive at a constant speed, and results in the formation of stop-and-go waves common to highway networks, thereby contributing to the reduced throughput and efficiency of the network. This effect has been empirically validated by the seminal work of Sugiyama et al. Sugiyama et al. (2008), and in simulation may be reconstructed through the use of popular car-following models (e.g Treiber et al. (2000); Bando et al. (1995)).

The presence of string instabilities in human drivers highlights a potential benefit for upcoming autonomous driving systems. In particular, by responding to string instabilities in human driving, automated vehicles (AVs) at even low penetrations can reestablish uniform flow driving, improving throughput and energy-efficiency in the process. Several studies have attempted to establish appropriate policies Sun et al. (2018); Cui et al. (2017); Wang and Horn (2019). Most relevant to the present paper, studies have also looked into the characteristics of emergent phenomena learned via deep reinforcement learning in mixed-autonomy ring roads with socially optimal objectives Wu et al. (2017a). The behaviors, however, lend little insight into the nature of desired behaviors needed to achieve socially optimal driving in real, large-scale networks. We highlight the limiting factors in Section 3.2 and provide methods for addressing them in Section 3.3 and 3.4. We then discuss the implications of these factors on learned behaviors in Section 4.2.

RL problems are studied as a Markov decision problem (MDP) Bellman (1957), defined by the tuple: $(\mathcal{S},\mathcal{A},\mathcal{P},r,\rho_{0},\gamma,T)$, where $\mathcal{S}\subseteq\mathbb{R}^{n}$ is an $n$-dimensional state space, $\mathcal{A}\subseteq\mathbb{R}^{m}$ an $m$-dimensional action space, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}_{+}$ a transition probability function, $r:\mathcal{S}\to\mathbb{R}$ a bounded reward function, $\rho_{0}:\mathcal{S}\to\mathbb{R}_{+}$ an initial state distribution, $\gamma\in(0,1]$ a discount factor, and $T$ a time horizon. For multiagent problems in particular, these problems are further expressed as Markov games Littman (1994), consisting a collection of state $\mathcal{S}_{1},\dots,\mathcal{S}_{k}$ and action sets $\mathcal{A}_{1},\dots,\mathcal{A}_{k}$ for each agent in the environment.

In a Markov game, an agent $i$ is in a state $s_{i}\in\mathcal{S}_{i}$ in the environment and interacts with this environment by performing actions $a_{i}\in\mathcal{A}_{i}$. The agent’s actions are defined by a policy $\pi_{\theta}:\mathcal{S}_{i}\times\mathcal{A}_{i}\to\mathbb{R}_{+}$ parametrized by $\theta$. The objective of the agent is to learn an optimal policy: $\theta^{*}:=\text{argmax}_{\theta}\eta(\pi_{\theta})$, where $\eta(\pi_{\theta})=\sum_{t=0}^{T}\gamma^{t}r_{t}$ is the expected discounted return. In the present article, these parameters are updated using direct policy gradient methods Williams (1992), and in particular the TRPO algorithm Schulman et al. (2015).

Transfer learning techniques in reinforcement learning provide methods of leveraging experiences acquired from training in one task to improve training on another Taylor and Stone (2009). These tasks may differ from the perspective of the agent (e.g., the observation the agent perceives or the actions it may perform) or other components of the MDP structure (e.g., the transition probability). Standard transfer learning practices include sharing policy parameters and state-action pairs between tasks. For a survey of transfer learning techniques, we refer the reader to Taylor and Stone (2009); Pan and Yang (2009).

The notion of transferring knowledge from policies learned on ring roads to more complex tasks has been explored in the past. In particular, the work of Kreidieh et al. (2018) explores the transferability of policies learned on a closed ring road network to open highway networks with instabilities arising from an on-ramp merge. It finds that policies trained on ring road networks exhibiting similar network densities as their target network performs well in single-lane merges without exposure to the network. This work, however, provides limited success in generating meaningful control strategies for multi-lane highway settings, where additional approximations for the dynamics of lane-changing are needed. We introduce methods for addressing this challenge in the following section.

We begin by establishing the training environment under which policies learn to regulate congestion in simplified ring road representations of traffic. This is an extension of the specifications provided in Wu et al. (2021), with modifications aimed at improving the performance of learning methods in multiagent settings.

The process of learning mixed-autonomy traffic regulation policies within simulated ring roads presents several notable benefits in the context of RL. For one, the simplicity of the proposed dynamics renders the problem easy to reconstruct and computationally efficient to simulate over discrete time segments. This is particularly beneficial within the field of RL, which relies largely on reproducibility and benchmarking to progress, and within which existing methods often require millions of interactions to generate meaningful behaviors. In addition, notions of stability and social optimally render the definition of the reward, or objective function, relatively straightforward. This is in contrast to RL studies in more complex mixed-autonomy settings, which rely primarily on heuristic insights for engineering reward functions that handle, for example, the presence on and off-ramps, insights that may not be transferable to other domains. The question, nevertheless, remains: Are policies learned within these settings meaningful/generalizable to complex, multifeatural networks?

To answer this question, we identify two features limiting ring road policies’ applicability to multi-lane, open highway networks. We address these limitations in the following subsections.

We begin by addressing the challenges of learning transferable open-network behaviors in environments with closed-loop / periodic boundary conditions. As mentioned in the previous subsection, the presence of this boundary condition produces strong couplings between actions by an AV and the behaviors of vehicles directly in front of it. This effect is in direct opposition to the natural convective stability of open networks, which likely results in the formation of undesirable emergent behaviors.

Larger diameter ring roads with additional automated vehicles, as noted in Orosz et al. (2009), can help dilute these boundary effects. Learning in these more complex settings, however, poses a significant challenge to existing multiagent RL algorithms. In particular, the increasingly delayed effects that individual actions have on the global efficiency of these networks, coupled with the effects of further non-stationary and credit assignment problems, hinder the progression of gradient-based methods as they attempt to traverse the space of possible solutions. As a result, features within the learning procedure as simple as policy initializations heavily dictate the nature of the converged behavior in these larger settings.

To improve the performance of policies in larger ring problems, we design a curriculum learning paradigm that exploits the extendability of the problem and the similarity in solutions between rings of similar magnitudes. This approach is depicted in Figure 2. Within this paradigm, policies to be trained in mixed-autonomy rings of length $nL$ with $n$ AVs are initially pretrained in a ring of length $L$ with $1$ AV for a total of $n_{\text{pretrain}}$ epochs. This initial problem is relatively simple to solve but does not sufficiently generalize the $n$-AV problem due to the vast difference in effects from the periodic boundary. However, the solution serves as a beneficial initialization for scaling the performance of policies learned in the $2$-AV settings, which share closer boundary effects. As such, a new pretrain policy is learned on the $2$-AV ring using the policy learned prior as a warm start, and this process is repeated for $n-1$ iterations until a proper policy initialization may be provided to the $n$-AV problem, which is then trained for a total of $n_{\text{train}}$ epochs.

In this subsection, we attempt to address the second of the challenges presented in Section 3.2, namely the shift in domains induced by the presence of lane changes in real-world tasks. To resolve this concern, we attempt to model the effects of lane changes from the perspective of individual lanes. In particular, we take inspiration from the work of Wu et al. (2017b) and model lane changes as stochastic insertions and deletions of vehicles within the ring. As seen in Figure 3, the insertions, which model lane-change-in actions, are characterized by their event probability $p_{\text{enter}}$ and the entry speed $v_{\text{enter}}$ and headway $h_{\text{enter}}$ with respect to the trailing vehicle, and deletions, which model lane-change out events, are characterized by their event probability $p_{\text{exit}}$.

In modeling the insertion/deletion event probabilities and entry conditions, we wish to design simple representations that sufficiently account for the effects of perturbations to ensure effective generalizability. Random lane change events represent the simplest approach for modeling such effects on automated vehicles. For vehicles whose headway permits a safe lane change, the lane change events are modeled as identical independent random variables of a Bernoulli distribution. Through such an approach, the components of the lane change events at a given time are defined as:

where $\mathbb{E}_{out}$ and $\mathbb{E}_{in}$ are the hyperparameters for the expected number of lane change events at each time step. $n_{t}$ is the number of vehicles who can safely experience an overtaking event at time $t$.

The use of random lane change actions, in addition to its design simplicity, absolves the modeling procedure of its dependence on data and estimation. As a result, policies learned through such a lane change model are not biased by small datasets or poor performing estimations and instead must learn to perform generally well under any potential lane change event, aggressive or otherwise. In the context of transfer learning, this approach can be considered closest to domain randomization Tobin et al. (2017). We explore the effects of the $\mathbb{E}_{out}$ and $\mathbb{E}_{in}$ terms in the following section.

Simulations of the ring-road problem are performed via a discretized variant of the model depicted in Eq. 2 with a simulation time step of $0.2$ sec/step. We consider four variants of the ring environment, which we characterized by the number of controlled agents $N_{AV}\in\{1,2,3,4\}$. In each of these tasks, we choose a range of ring circumferences that possess optimal (uniform flow) speeds in the range $[5,10]$ m/s. The corresponding range of acceptable lengths is chosen to be $[250N_{AV},360N_{AV}]$ m. At the start of every rollout, a new length is chosen from this range to ensure that any solution learned does not overfit to a specific desired speed. To simulate the effects of lane changes, we follow the process highlighted in Wu et al. (2017b) for stochastically introducing and removing vehicles while extending this approach to include arbitrary placements of vehicles within a leading gap during cut-ins. The lane-change interval upon which vehicles enter and exit is considered a hyperparameter and is further discussed in later sections. These simulations are reproducible online from: REDACTED FOR REVIEW PURPOSES.

For all RL experiments in this paper, we use the Trust Region Policy Optimization (TRPO) Schulman et al. (2015) policy gradient method for learning the control policy, discount factor $\gamma=0.99$, and step size $0.01$, and a diagonal Gaussian MLP policy with hidden layers $(64,64,64)$ and a ReLU non-linearity. The parameters of this policy are shared among all agents in the execution/evaluation procedure and jointly optimized in the training procedure.

We begin by evaluating the performance of the training procedure and the importance of utilizing curricula in solving larger problems. Figure 6 depicts the learning performance on the ring road task with and without the use of stochastic lane changes and curricula. This figure evaluates the policy on its ability to achieve uniform-flow equilibrium over the past $100$ rollout. The evaluation metric $m$ for an individual rollout consists of the ratio between driving speeds $v_{i}$ by individual vehicles $i$ and the desired uniform flow speed $V_{\text{eq}}(\cdot)$ as defined in Section 3.1. Mathematically, this can be written as:

where $T$ is the time horizon, $N$ is the number of vehicles, and $L$ is the ring’s circumference. As we can see, while the early stage problems perform well without curricula, policies trained on the larger ring problem struggle to achieve similar performance. On the other hand, when trained via a curriculum of gradually growing rings, the learned policies continue to achieve high scores close to the ideal value of $1$ and outperforming their non-curricular counterparts by up to $10$%. This improvement is particularly evident once lane changes are introduced to the environment, possibly due to the increased stochasticity and complexity that is forced onto the problem.

Figure 5 depicts the spatio-temporal performance of the different learned policies in mixed-autonomy ring roads of different lengths. To visualize these behaviors, we use time-space diagrams to visualize the evolution of vehicles across space and time and color their trajectories based on their current speeds to visualize macroscopic mobility and the presence of stop-and-go congestion. In fully human-driven settings (Fig. 5(a)), instabilities in human driving behaviors result in the formation of stop-and-go waves after a few minutes, depicted by the red diagonal lines in the figures. In larger rings, these waves span more vehicles and propagate more slowly through the network, rendering them more complex problems to solve. Standard RL methods (Fig. 5(b)) can perform well in training stop-and-go wave dissipating policies for smaller environments but fail in the larger ring settings. On the other hand, the introduction of curricula to the training procedure (Fig. 5(c)) helps significantly scale the performance of these algorithms, producing much fewer waves. Finally, when trained in the presence of stochastic lane changes (Fig. 5(d)), slightly more waves are experienced due to the increased complexity of the problem.

Next, we validate the generalizability of the learned policy by evaluating its zero-shot transfer to an intricate and calibrated model of human-driving. The target network considered (Figure 1, right) is a simulation of a $1$-mile section of the I-210 highway network in Los Angeles, California. This network has been the topic of considerable research, with various studies aiming to identify and reconstruct the source of congestion within it Gomes et al. (2004); Dion et al. (2015). We here exploit the work of Lee et al. (2021), which explores the role of mixed-autonomy traffic systems in improving the energy-efficiency of vehicles within I-210.

Simulations of the I-210 network are implemented in Flow Wu et al. (2021), an open-source framework designed to enable the integration of machine learning tools with microscopic simulations of traffic via SUMO Krajzewicz et al. (2012). These simulations are executed with step sizes of $0.4$ sec/step and are warmed-started for $3600$ sec to allow for the onset of congestion before being run within the evaluation procedure an additional $600$ sec. Once the initial warmup period is finished, $5$% of vehicles are replaced with AVs whose actions are sampled from the learned policy to mimic a penetration rate of $5$%.

Figure 9 depicts the performance of the different learned policies once transferred onto the I-210 network. In this task, we can no longer compute the performance as before, as no optimal uniform flow speed exists. Instead, we choose to compute the average time spent stopped by individual vehicles as they traverse the network. This serves as a proxy for the average length and frequency of stop-and-go waves that emerge in the network. Moreover, in choosing an estimation for the expectations in Eq. 8 and 7, we compute the average number of lane changes experienced in the I-210 in the absence of autonomy and use different magnitudes of this expectation (half, equal/same, double, and quadruple) as the parameter for different algorithms. As we can see from this figure, both the choice of network size and the presence and magnitude of lane changes significantly impact the amount of stopped delay incurred by drivers. From the perspective of magnitude, the transition from $1$ to $2$ AVs captures a significant portion of the improvement, with consecutive gains being marginal. This suggests that the vehicles themselves, when interacting with one another, also largely nullify the effects of the boundary in training time. From the perspective of the lane change frequency, a similar effect is seen as frequencies increase, with the policies learning more cautious maneuvers to avoid the increasing perturbations.

Figure 8 depicts the spatio-temporal performance of the learned policy in the $4$-AV setting once transferred to the I-210 network. As noted in the previous paragraph, in the absence of lane changes, policies learned significantly underperforms when transferred to this network. In particular, the policy does not account for the eventuality of waves forming downstream by maintaining a safe distance from its leader, thereby periodically contributing to the propagation of stop-and-go waves. Conversely, the policies trained in the presence of lane changes generalizes nearly perfectly to the new settings, effectively dissipating the vast majority of waves that appear. Similar behaviors seem to occur when the number of lane changes experienced in training is as high as quadruple those in the target task. However, in this latter setting, large gaps begin to form by the automated vehicles in largely uncongested settings. These overly cautious behaviors serve to unnecessarily degrade the mobility of the network, reducing the average speeds of vehicles and the throughput of the entire system despite, interestingly enough, reducing the amount of stopped time experienced on average. This finding suggests that, while exact estimations of lane-change frequencies may not be needed, significant overestimation should be avoided when designing a robust system that is not overly apprehensive of potential perturbations.