Multi-Agent Vulnerability Discovery for Autonomous Driving
with Hazard Arbitration Reward

Our work can be describe as a Partially Observed Markov Decision Process (POMDP) [7]. A POMDP is defined by tuple $<\mathcal{S},\mathcal{A},\mathcal{O},\emph{N},\mathcal{P},\mathcal{R},\gamma>$. $\mathcal{S}$ presents state space including all possible conditions of N agents. $\mathcal{A}$ is shared action space. $\mathcal{O}$ is observation space of each agent. $\mathcal{P}$ indicates the transition possibility model of states. $\mathcal{R}$ is the shared reward function. $\gamma\in[0,1]$ is the reward discount parameter. At each step, agent i receives an observation $\emph{$o_{i}$}=\mathcal{O}(\emph{$s_{i}$})$ where $\emph{$s_{i}$}\in\mathcal{S}$ is the local state, takes an action $a_{i}$ according to shared policy $\pi_{\theta}(\emph{$a_{i}$}|\emph{$o_{i}$})$ where $\theta$ is the shared parameters, transit to next state $s_{i}^{\prime}$ according to transition possibility $\emph{p}(\emph{$s_{i}^{\prime}$}|\emph{$s_{i}$},\emph{$a_{i}$})\in\mathcal{P}$, and finally receives a reward $\emph{$r_{i}$}=\mathcal{R}(\emph{$s_{i}$}|\emph{$a_{i}$})$. Considering homogeneous agents, each agent i optimizes discounted cumulative rewards $\emph{$R_{i}$}=\sum_{t}\gamma^{t}\emph{$r_{i}^{t}$}$.

Generally speaking, there are two types of strategies to attack decision-making policies: (1) Attacking-on-States: directly tampering with the state sequence of the target policy. These methods [8, 9] impose small disturbances on the state observations to make the policy output wrong decisions or get a smaller reward. (2) Attacking-by-Policy: learning one or more adversarial policies through interacting with the target policy. Gleave et al. [10] train an adversarial policy to compete with the under-test policy in a competitive game, and can discover flaws of the under-test policy.

We adopt the Attacking-by-Policy strategy in our work since it is more suitable for discovering vulnerabilities in autonomous driving policies. This is because that the perturbations crafted by Attacking-on-States strategies might be too noisy to occur in practice. In contrast, the constructed adversarial scenarios by Attacking-by-Policy strategies could appear naturally since the adversaries themselves are interactive agents with well-defined action spaces.

There exist some studies [11, 12, 13, 14] that adopt the Attack-by-Policy strategy to attack an autonomous driving policy. However, most of these studies aim at directly causing accidents [11, 12, 13], rather than discovering the vulnerability of the under-test policy. We will elaborate on the multi-aspect differences between our work and these studies in Section III-D.

We illustrate the overall framework of Safety Test framework by finding Av-Responsible Scenarios (STARS) in Fig. 2. We follow the terminologies used in the literature of adversarial attack methods, the under-test policy is defined as “defender”, the adversarial vehicles trained by the MARL are called “attackers”, and other background vehicles are named “NPCs”. The defender, served as the attack target, is controlled by a well-trained or well-designed autonomous driving policy. The purpose of attackers is to explore the vulnerability that can make the defender behave wrongly, such as a car crash or out of bounds. In addition, we introduce NPCs to better simulate real driving scenarios.

We adopt the Multi-Agent PPO (MAPPO) framework [15] with parameter sharing. MAPPO is an extension of Proximal Policy Optimization (PPO) [16] as an on-policy Centralized Training Decentralized Execution (CTDE) reinforcement learning algorithm. We train a actor network $\pi_{\theta}$ parameterized by $\theta$, and a value network $V_{\phi}$ as the critic network. With Generalized Advantage Estimator method [17], we get Advantage function $A_{i}^{(k)}$, and further the forms of actor and critic loss function are given as

$$L(\theta)=[\frac{1}{Bn}\sum_{i=1}^{B}\sum_{k=1}^{n}min(r_{\theta,i}^{(k)}A_{i}^{(k)},clip(r_{\theta,i}^{(k)},1-\epsilon,1+\epsilon)A_{i}^{(k)})]$$

$$+\sigma\frac{1}{Bn}\sum_{i=1}^{B}\sum_{k=1}^{n}S[\pi_{\theta}(o_{i}^{(k)})].$$

$$L(\phi)=\frac{1}{Bn}\sum_{i=1}^{B}\sum_{k=1}^{n}(max[(V_{\phi}(s_{i}^{(k)})-\hat{R}_{i})^{2},$$

$$(clip(V_{\phi}(s_{i}^{(k)}),V_{\phi_{old}}(s_{i}^{(k)})-\varepsilon,V_{\phi_{old}}(s_{i}^{(k)})+\varepsilon)-\hat{R}_{i})^{2}]),$$

where B is the batch size, n is the number of homogeneous agents, S is the policy entropy, $\sigma$ is the entropy coefficient hyperparameter, and $\hat{R}_{i}$ is the discounted reward.

In our training environment, all agents share the same form of observation and action space. The $ith$ agent’s observation $o_{i}=[k_{0}^{i},k_{1}^{i},k_{2}^{i},k_{3}^{i},k_{4}^{i}]$ describes the states of the closest five vehicles arranged by distance. $k=[p,x,y,v_{x},v_{y}]$ is a state vector, where p indicates presence of vehicle, x and y are x-coordinate and y-coordinate, and $v_{x}$ and $v_{y}$ are component velocities on x and y axis. The discrete action a takes five choices for accelerating, decelerating, changing to left lane, changing to right lane, and doing nothing.

Vulnerabilities of the driving policy cause the vehicle to take improper actions, such as unsafe lane changing or risky overtaking, which lead to accident scenarios. However, some accident scenarios might not correspond to vulnerabilities in the under-test driving policy, since they are not caused or even not avoidable by the under-test driving policy, which means they are not valuable for improving the driving policy. An example of these non-AV-responsible hazardous scenarios is shown in Fig. 1 (Left), where the attacker suddenly makes a lane change and hits the under-test vehicle. In contrast, Fig. 1 (Right) shows a typical AV-responsible scenarios, where the under-test policy should have decelerated but makes a lane change and thus makes a collision. To differentiate between AV-responsible and non-AV-responsible hazardous scenarios, we design a reward to penalize some trivial scenarios where the attackers are to blame. We name this reward design HAR, since this reward arbitrates the responsibility assignment of the accident scenes and provides behavior criteria for the attackers.

The design of HAR is as follows. The HAR is consisted of two parts: the collision reward $R_{C}$ and the aggressiveness penalty $P_{agg}$. We give a sparse collision reward $R_{C}$ when the defender makes collisions with NPC vehicles or attacker vehicles, since we expect the attackers to explore how to generate the hazardous scenarios.Also we give an aggressiveness penalty $P_{agg}$ to limit the behaviors of the attackers. Therefore, the HAR $R_{HAR}$ is

$$R_{C}=\begin{cases}\phi&if\text{ the defender makes a collision}\\ 0&otherwise\end{cases}$$

$$P_{agg}=\begin{cases}\rho&\left|\dot{v_{att}}\right|>\lambda\;or\;a_{att}=a_{lc}\;or\;a_{att}=a_{rc}\\ 0&otherwise\end{cases}$$

$$R_{HAR}=R_{C}+P_{agg},$$

where $\dot{v_{att}}$ is the acceleration of the attacker, $\lambda$ is an acceleration threshold, $a_{att}$ is the action of the attacker, $a_{lc}$ and $a_{rc}$ are left lane change action and right lane change action.

To evaluate the proposed methods, this work chooses three typical driving environments, i.e., highway driving, merging, and roundabout driving. These three scenarios contain the most basic road elements like straight lane, merging lane, circle lane, and crossroad. Meanwhile, the combination of these scenarios could cover most safety-critical driving behaviors like lane changing, lane merging, overtaking, and crossroad driving [1]. Our testbed is based on the previous work [19], and we further add three roles, i.e., attacker, defender, NPC. The details of these three scenarios are as follows:

We evaluate four representative autonomous driving methods, which can be divided into 3 categories.

• •

  Planning-based algorithm: Value Iteration [20]

• •

  Safe-enhanced algorithm: Robust Value Iteration [21]

• •

  RL-based algorithm: Dueling Double Q-Network [22] and Policy Proximal Optimization [16]

Value Iteration (VI) and Robust Value Iteration (RVI) are based on the finite MDP abstracted by the environment, and use dynamic-programming methods to optimize the Bellman Equation to get the action on each state. Dueling Double Deep Q-Network (D3QN) and Policy Proximal Optimization (PPO) defender models use the same observation and action space as the adversarial agent during the training process. Each model is trained in three scenarios respectively, aiming at collision avoidance and high-speed keeping.

We train MAPPO adversarial agents to explore the vulnerability of different driving policies. The reward function $\mathcal{R}$ of the attacker is consist of two parts: HAR $R_{HAR}$ and a distance reward $r_{d}$, i.e., $\mathcal{R}=R_{HAR}+r_{d}$. The HAR bounds the hazardous scenarios to encourage exploration of the AV-responsible scenarios, with the parameter $\phi=10$ and $\rho=-10.5$. And we design a small distance reward $r_{d}$ to speed up our training.

STARS can discover AV-responsible hazardous scenarios that correspond to vulnerabilities of the under-test policy. Video demonstrations of the discovered scenarios are in the supplementary material. And we discuss three key characteristics of the discovered scenarios as follows.

These scenarios are AV-responsible scenarios that correspond to vulnerabilities of the under-test policy. We demonstrate three example hazardous scenarios discovered by STARS in Fig. 4 and Fig. 5. It is obvious that in all these accidents, the under-test policy (red car) makes improper decisions and causes accidents, while other traffic participants do not exhibit any aggressive behavior. This indicates that the discovered hazardous scenarios are AV-responsible scenarios revealing vulnerabilities of the under-test policy. And the under-test policy should be improved to handle these scenarios.

STARS can discover diverse scenarios exhibiting various accident patterns, which is different from previous studies that conduct RL-based adversarial attacks on driving policies [11, 18, 12, 13, 14]. Most previous studies [11, 18, 12, 13] aim to get direct collision scenarios by designing direct collision rewards. Consequently, their attackers can only learn to construct one type of direct collision scenarios. These scenarios are predictable before running the experiments thus only contribute minor knowledge to the policy testing process.

In contrast, we do not plant in the concrete hazardous scenario into the design of HAR, and the design of HAR enables the framework to explore and discover more diverse hazardous scenarios. Fig. 4 demonstrates two of our discovered hazardous scenarios in the Highway environment against the D3QN defender: 1) When the under-test vehicle is driving on the side lane, two attackers learn to form a siege by driving in front of and at the side of the under-test vehicle. The under-test D3QN policy will make a risky decision to overtake between the two attackers and cause a crash. 2) When the under-test vehicle is driving on the side lane following another vehicle (could be an NPC vehicle), two attackers can choose to occupy the adjacent lane. In this scenario, the under-test policy will fail to slow down and collide with the vehicle ahead. Note that these two scenarios are constructed by the same attacker policy learned in a single experiment run, and the attackers can construct different scenarios according to different initialization of the vehicle positions. This demonstrates the ability of STARS to automatically discover diverse hazardous scenarios.

Multiple attackers exhibit collaborative behaviors to construct complex hazardous scenarios to make the under-test policy misbehave. Our application of MAPPO and design of HAR together enable the attackers to construct complex scenarios collaboratively. In another word, multiple attackers learn to collaborate to construct hazardous scenarios that cannot be constructed by a single attacker.

Fig. 5 shows an example of the collaborative behavior of the attackers in the Merge environment. After initialization, the attacker on the main road keeps a suitable distance in front of the defender, while the attacker on the downside merging lane adjusts its velocity and merges in at the suitable time. This scenario will induce the under-test policy to make a lane change. At this time, if another vehicle is driving on the adjacent lane of the under-test vehicle and falls a little behind, the defender will fail to predict the collision and collide with it. This scenario requires several attackers to appear at the right position and conduct the right action at the right time simultaneously, and it is extremely hard for a single attacker to trigger this scenario.

To demonstrate the power of using multiple attackers, we compare the attack success rates of using single attacker and multiple attackers in Tab. I. In the experiments of using single attacker, one of the attackers is changed into a non-adversarial NPC vehicle. The results show that the attack success rate of two or three attackers significantly outperforms that of single attacker.

We compare the robustness of four driving policies according to the attack success rates and the training curves. The attack success rates represent the occurrence rate of AV-responsible scenarios, which indicates the risk factor of the under-test policy. The training curves in Fig. 6 of HAR reflect the complexity and difficulty for attackers to explore.

The hazardous scenarios exposing the vulnerability of the dynamic programming-based policies (VI and RVI) are easy to discover and highly reproducible. Limited by the finite estimation of infinite state-action pairs, many of their lane change decisions are risky. Thus, the attackers only need to drive close to the defender vehicle on the adjacent lanes to trigger the hazardous scenarios. Therefore, even one single attacker can successfully attack VI and RVI driving policies with high success rates.

Overall, the RL-based driving policies, PPO and D3QN, perform better than VI and RVI. Attacks against them have lower attack success rates in all environments. Discovering the vulnerability of the D3QN policy needs a sophisticated collaboration between attackers, as shown in Figure 5. PPO driving policy’s vulnerability is even harder to discover, such that the single attacker fails to discover any vulnerability.