Safety Guaranteed Robust Multi-Agent Reinforcement Learning with Hierarchical Control for Connected and Automated Vehicles

We consider the robust cooperative policy-learning problem under uncertain state inputs for CAVs in mixed traffic environments including HDVs that do not communicate or coordinate with CAVs, and various driving scenarios such as multi-lane intersection and highway (as shown in Fig. 1&4). We assume that each CAV can get shared information from V2V and V2I communications. We consider that a CAV agent $i$ has accurate self-observation of its driving state but potentially perturbed observations of the other vehicles. The two parts collectively constitute its state $s_{i}$ in reinforcement learning explained in Sec. III-B, and also used by the MPC controller as inputs to the robust CBFs.

The problem of Multi-Agent Reinforcement Learning with State Uncertainty for CAVs is defined as a tuple $G=(\mathcal{S},\mathcal{A},P,\{r_{i}\},\tilde{o},\mathcal{G},\gamma)$ where $\mathcal{G}\coloneqq(\mathcal{N},\mathcal{E})$ is the communication network of all CAV agents. $\mathcal{S}$ is the joint state space of all agents: $\mathcal{S}:=\mathcal{S}_{1}\times\dots\times\mathcal{S}_{n}$. The state space of agent $i$: $\mathcal{S}_{i}=\{o_{i},o_{\mathcal{N}_{i}},o_{\mathcal{N}_{i}^{UV}}\}$ contains self-observation $o_{i}$, observations $o_{\mathcal{N}_{i}}=\{o_{j}|j\in\mathcal{N}_{i}\}$ being the communicated message shared by neighbor connected agents $\mathcal{N}_{i}$, observations $o_{\mathcal{N}_{i}^{UV}}$ of unconnected vehicles $\mathcal{N}_{i}^{UV}$ either observed by agent $i$ itself or shared by other agents or infrastructures. For example, self-observation $o_{i}$ and shared observations $o_{\mathcal{N}_{i}}$ can contain location, velocity, acceleration and lane-detection: $(\bm{l},\bm{v},\bm{\alpha},LD)$; observations of unconnected vehicles $\mathcal{N}_{i}^{UV}$ can contain location and velocity $(\bm{l},\bm{v})$.

In this work, we consider agent $i$ suffers from uncertain observed locations and velocities of other vehicles aside from the ego vehicle (i.e. $\tilde{s}_{i}=\{o_{i}\}\bigcup\{\tilde{o}_{j}|j\in\mathcal{N}_{i}\bigcup\mathcal{N}_{i}^{UV}\}$), where $\tilde{o}_{j}$ denotes an uncertain observation over vehicle $j$ in comparison with the true accurate self-observation $o_{i}$. The uncertainty is defined by bounded errors $(e_{l},e_{v})\coloneqq\tilde{o}=(\tilde{\bm{l}},\tilde{\bm{v}}),\tilde{\bm{l}}=\bm{l}+e_{l},\tilde{\bm{v}}=\bm{v}+e_{v}$. The implementation of state uncertainty in testing experiments is explained in Sec. V.

The joint action set is $\mathcal{A}:=\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n}$ where $\mathcal{A}_{i}=\{a_{i,1},a_{i,2},\cdots,a_{i,3+k}\}$ is the discrete finite action space for agent $i$. $a_{i,1}$: KEEP-LANE-MAX - the CAV $i$ maximize its reference throttle in the current lane. $a_{i,2}$: CHANGE-LANE-LEFT - the CAV $i$ changes to its left lane. $a_{i,3}$: CHANGE-LANE-RIGHT - the CAV $i$ changes to its right lane. In experiment, the path planner will set a target waypoint trajectory onto its left/right neighboring lane. $a_{i,4},\ldots,a_{i,3+k}$ are $k$ lane-keeping actions associated with different reference throttle values. By choosing action $a_{i,5}$, for example, reference throttle value $throttle_{5}$ will first converted into reference acceleration then it will be fed to the controller and the calculation of safe control is introduced in IV-B. The state transition function is $P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$. The reward functions $\{r_{i}\coloneqq\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}\}$ are defined as

in which $\bm{v}_{j}$ is vehicle $j$’s velocity; $\bm{d}_{j}$ is $j$’s default destination and $r^{s}_{j}$ is the safety reward. $\mu^{v},\mu^{l},\mu^{s}$ are non-negative weights balancing the proportions of individual and total achievement. As break-downs of the safety reward $r^{s}_{j}(s,a)=p^{\textit{Col}}(s,a)+p^{\textit{MPC}}(s,s_{\mathcal{N}_{i}},a)$, the collision penalty $p^{\textit{Col}}(s,a)$ penalizes collision and $p^{\textit{MPC}}(s,s_{\mathcal{N}_{i}},a)$ penalizes the infeasiblity of the low level controller.

The proposed Robust MARL algorithm (Alg 1; Fig. 2) uses centralized training and decentralized execution. We are inspired by the Worst-case-aware Robust RL framework [9] and designed the robust MAPPO. Each robust PPO agent maintains a policy network $\pi^{\boldsymbol{\theta}_{i}}$ (“actor”) with parameter $\boldsymbol{\theta}_{i}$, a value network (“critic”) $V(s)$ with parameter $\boldsymbol{\phi}_{i}$ and the second critic $\underline{Q}^{\boldsymbol{\omega}_{i}}(s_{i},a_{i})$ network approximating the worst-case action values with parameter $\boldsymbol{\omega}_{i}$. In order to learn a safe cooperative policy for the CAVs, the MARL interacts with the MPC controller (Sec. IV-B) during the Rollout process. As the algorithm starts, agent $i$’s policy takes initial state $s_{i}$ and samples an action $a_{i}$; the MPC-controller with robust CBFs given $a_{i}$ computes a safe control $\boldsymbol{u}_{i}$ for the vehicle to execute. The agent receives $r_{i}$ (1) and all agents synchronously move to the next time-step by observing the new state $s^{\prime}=\prod_{i}s^{\prime}_{i}$.

During training, both critics account for updating the policy with loss function (2) so that the trained policy can balance the goals between maximizing expectations of advantage $\hat{A}_{i}$ and worst-case return $\underline{Q}^{(\boldsymbol{\omega}_{i})}$. Through value-based state regularization $\mathcal{L}^{reg}_{i}(\boldsymbol{\theta}_{i})$, the policy is trained to be robust at crucial “vulnerable” states around which uncertainties are more likely to affect the policy [32, 10].

We adopt receding horizon control to implement the low-level controller for every agent $i$ in the road network. The low-level controller maps the high-level plans/actions $a_{i}\sim\pi_{\boldsymbol{\theta}_{i}}$ into primitive actions/control inputs for agent $i$. Firstly, a path planning function $z:\mathcal{A}\rightarrow\mathcal{X}_{i}\times\mathcal{U}_{i}$ is used to map the high-level plans/actions into state and action references, i.e. $(\boldsymbol{x}_{i}^{ref},\boldsymbol{u}_{i}^{ref})=z(\boldsymbol{x}_{i},a_{i})$, where $\mathcal{X}_{i}\subset\mathcal{S}_{i}$ is the state space and $\mathcal{U}_{i}$ is the input space of primitive actions of CAV $i$ respectively. Subsequently this information is fed to the MPC controller. To prevent collision between agents, safety constraints are incorporated to the controller using CBFs.

We consider that the dynamics for each vehicle is affine in terms of its control input $\bm{u}$ as follows:

where $f$ and $g$ are locally Lipschitz, $\boldsymbol{x}\in\mathcal{X}\subset\mathcal{S}$ denotes the state vector and $\bm{u}\in\mathcal{U}=[\boldsymbol{u}_{min},\boldsymbol{u}_{max}]$ denotes the control input. In the above equations $x(t),y(t),\psi(t),v(t)$ represent the current longitudinal position, lateral position, heading angle, and speed, respectively. $u(t)$ and $\phi(t)$ are the acceleration and steering angle of vehicle at time $t$, respectively, $g\left(x(t)\right)=$ $\left[g_{u}\left(\boldsymbol{x}(t)\right),g_{\phi}\left(\boldsymbol{x}(t)\right)\right]$.

We incorporate safety with respect to other vehicles (primarily unconnected vehicles) as follows. Let CAV $i$ needs to stay safe to a vehicle $j\in\{o_{i},o_{\mathcal{N}_{i}},o_{\mathcal{N}_{i}^{UV}}\}$ in its vicinity. To achieve that we enforce a constraint on vehicles $i$ by defining a speed dependent ellipsoidal safe region $b_{1}\left(\boldsymbol{x}_{i},\boldsymbol{x}_{j}\right)$ as follows:

where $a_{i}$, $b_{i}$ are weights adjusting the length of the major and minor axes of the ellipse as illustrated in Fig. 3. We enforce safety constraints on any CAV $i$ for vehicles in $\{o_{i},o_{\mathcal{N}_{i}},o_{\mathcal{N}_{i}^{UV}}\}$ in one of three scenarios which are: i. immediately preceding in the same lane, ii. located in the lane the ego is changing to and iii. arriving from another lane that merges to the lane of the ego vehicle.

Given a continuously differentiable function $b:\mathbb{R}^{n}\rightarrow\mathbb{R}$ and the safe set defined as $C:=\{\boldsymbol{x}\in\mathbb{R}^{n}:b(\boldsymbol{x})\geq 0\}$, $b(\boldsymbol{x})$ is a candidate control barrier function (CBF) for the system (IV-B) if there exists a class $\mathcal{K}$ function $\alpha$ and $\boldsymbol{u}$ such that

for all $\boldsymbol{x}\in C$, where $L_{f},L_{g}$ denote the Lie derivatives along $f$ and $g$, respectively. Additionally, we use CLFs to incorporate state references to the controller. A continuously differentiable function $V:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a globally and exponentially stabilizing CLF for (IV-B) if there exists constants $c_{i}\in\mathbb{R}_{>0}$, $i=1,2$, such that $c_{1}||\bm{x}||^{2}\leq V(\bm{x})\leq c_{2}||\bm{x}||^{2}$, $\boldsymbol{u}\in\mathcal{U}$, and the following inequality holds

where $e$ makes this a soft constraint.

The uncertain state measurements stemming from process noise and measurement noise denoted by $\hat{\bm{x}}(t)$ is expressed as follows:

where $\bm{w}(t)$ is bounded noise such that $\|\bm{w}(t)\|_{\infty}\leq\epsilon$. In the presence of noise, the robust CBF constraint becomes the following which has been shown to make the safety set $C$ forward invariant in [33].

Finally, the MPC with robust CBF and CLF control problem can be expressed as follows:

where $A_{x}\in\mathbb{R}^{n\times n}$, $A_{u}\in\mathbb{R}^{q\times q}$ are the matrix of weights, $\bm{B}\in\mathbb{R}^{n+q}$ is vector of weights, and $\bm{\delta}$ is the vector of weights of the penalty terms associated with the relaxation parameters of the CLF constraints.

In this section, we highlight our method’s performances in terms of safety guarantees and robustness against state uncertainty while generalizable to different driving scenarios. We demonstrate through ablation studies that the incorporation of MPC with robust CBF based controller improves the performance of MARL algorithm. Additionally, we also demonstrate that our proposed hierarchical approach with robust MARL also improves the MPC-CBF controller.

We trained three models: our Safe-RMM method, the “non-robust” Safe-MM that also adopts our framework, and the MCP method adopting MARL-PID controller with CBF safety shield [8]. Each model is trained on Intersection and Highway respectively for 200 episodes. In evaluation, aside from the three trained models, we have “MP”, MARL with PID controller, as an example of learning-based method without safety shielding; and we also implemented the benchmark “RULE” adopting a rule-based planner and a robust MPC controller. The “RULE” benchmark has been implemented based on the method proposed in [35], which introduces a safety-guaranteed rule for managing vehicle merging on roadways. This rule ensures safe interactions between vehicles arriving from different roads converging at a common point. Methods are evaluated for 50 episodes in both scenarios under four uncertainty configurations: None (uncertainty-free), random error $e^{\text{rand}}$, and two targeting errors $\mathtt{ERR}^{\mathcal{V}}$ and $\mathtt{ERR}^{T}$. Training results in Intersection are shown in Fig. 5; evaluations in both scenarios are presented in Table. I. For each entry in both tables, the left integer is the number of collisions happened during evaluation (in 50 episodes); the right number is the agents’ mean discounted return considering only the rewards related to velocity and goal-achievement in (1). We highlight the top performance across all methods with the least collision numbers and the highest efficiency return.