Social Coordination and Altruism
in Autonomous Driving

In order to formally study the social dilemmas between humans and autonomous agents in heterogeneous environments, it is crucial to quantify the social preference of an individual, e.g., whether if they will defect or cooperate in a given situation such as opening a gap in our highway merging example. The degree of an agent’s egoism or altruism with regards to its counterparts is defined as Social Value Orientation (SVO), a widely used notion in the social psychology literature which has been recently adopted in robotics research. Specifically, we borrow the angular annotation for SVO as defined by Liebrand  et al.  [34]. The SVO angular preference $\phi$, quantifies how an agent weights its own reward against the reward of others. An agent’s total utility $R_{i}$ can then be written as,

where $r_{i}$ is the agent’s individual utility, $r^{-}_{i}$ is the total utility of other agents from the perspective of the $i$th agent which in general is a function $f(.)$ of their individual utilities,

Autonomous agents require an understanding of human drivers’ social preferences and their willingness to coordinate. However, it is well-established in the behavioral decision theory that humans are heterogeneous in SVO and thus their preference is rather ambiguous and unclear [36]. Current works on social navigation for AVs often make restrictive assumptions on human drivers’ social preference and compliance [2], whereas Figure 2 indicates an spectrum of altruism among humans with heterogeneous social value orientations. Thus, due to the large spectrum of altruistic behavior observed by humans, our insight is to rely on autonomous cars instead to guide the overall system toward more socially desirable objectives. Specifically, we plan to find policies for AVs that improve the utility of the group as a whole through emerging alliances and more importantly, affecting the behavior of human drivers. In our particular driving example, the desired social outcome is achieving seamless and safe highway merging while maximizing the distance traveled by all vehicles and avoiding collisions.

We choose a highway merging scenario with a mixed group of AVs and HVs as our base experiment scenario, as illustrated in Figure 3. A merging vehicle, which can be either HV or AV, approaches the highway on the merging ramp and faces a mixed platoon of vehicles that are cruising on the highway. This configuration contains a group of AVs that hold the same SVO, as well as a group of HVs which are heterogeneous in their SVO and hence it is unclear if they are allies or foes. In this settings, it is obvious that the individual interest of the merging vehicle, i.e., seamless merging into the highway, does not align with that of the cruising vehicles, i.e., cruising with optimal speed and energy consumption. We design our case study scenario in a way that safe and seamless merging necessarily requires all AVs to work together and none of them alone can enable the merging of the mission vehicle without the cooperation of the others. Formally, the road section shown in Figure  3 is shared by a set of AVs $\mathcal{I}$ that are connected together via V2V communication and governed by a decentralized stochastic policy, a set of HVs $\mathcal{V}$ operated by humans with heterogeneous and unknown SVOs, and a human-driven or autonomous mission vehicle $M\in\mathcal{I}\cup\mathcal{V}$ that attempts to merge into the highway.

A human driver’s perception is often limited by their range of vision, occlusion, and obstacles. In contrast, CAVs share their observations to overcome these limitations. Each CAVs has a unique local observation $\tilde{\textbf{o}}_{i}([\textbf{o}_{i}];\mathcal{G})$ that is constructed using the its own local observation, as well as the local observations it receives from the neighboring CAVs. As mentioned before, graph $\mathcal{G}$ grasps this inter-agent communication. Therefore, an observer AV can detect a subset of other AVs, $\widetilde{\mathcal{I}}\subset\mathcal{I}$, and a subset of HVs $\widetilde{\mathcal{V}}\subset\mathcal{V}$. As we elaborated before, our aim is to find a decentralized control scheme that can induce altruism in the behavior of AVs. Hence, each AV must use its local observation $\textbf{o}_{i}$ to make independent decisions that optimize its utility. The value of the agent’s altruism, i.e., the SVO angular phase $\phi$, determines the social implications of an agent’s local actions. To summarize, we state our problem as deriving a utility function that enables the AVs to handle competitive driving scenarios, such as those illustrated in Figure 1, and lead them into socially-desirable outcomes that improve traffic safety and efficiency for the group of vehicles.

We employ a numeric representation for an agent’s observation that embeds the kinematics of the neighboring vehicles. Additionally, we integrate the history of vehicles’ last $h$ meta-actions to extract temporal information and their past trajectories. An ego vehicle $I_{i}\in\mathcal{I}$ observes a set of HVs and AVs in its perception range. The Kinematic observation includes the relative Frenet coordinates of the closest $|\widetilde{\mathcal{I}}\cup\widetilde{\mathcal{V}}|+1$ vehicles in addition to the absolute Frenet coordinates of the ego vehicle. Formally, agent $I_{i}$ receives a local observation $\tilde{\textbf{o}}_{i}\in\widetilde{\mathcal{O}}_{i}$,

Each row of the local observation matrix $\textbf{o}^{(i)}$ is defined as,

in which, $l_{j}$ and $d_{j}$ are the longitudinal and lateral Frenet coordinates of the $j$th vehicle, respectively. Vehicle’s yaw angle is denoted by $\rho$ and the autonomy flag is $\lambda_{j}=0$ if $I_{j}\in\widetilde{\mathcal{V}}$ and $\lambda_{j}=1$ otherwise. In case that the total number of observed vehicles is smaller than the set size of the observation matrix o, the remaining rows are filled with zeros with $p_{j}=0$. $\overline{\mathrm{H}}^{\mathcal{A}}_{j}$ is the unrolled numeric representation of the action history array $\textbf{H}^{\mathcal{A}}_{j}$ that contains the last $h$ meta-actions taken by $I_{j}$ and is defined as,

Our interest is in maneuver-level decision-making for autonomous vehicles. Thus, we define the action space $\mathcal{A}$ as the set of abstract meta-actions $\mathcal{A}_{i}=[\texttt{Lane Left}$, Idle, Lane Right, Accelerate, $\texttt{Decelerate}]^{\top}$. These meta-actions are then translated into admissible trajectories and low-level control signals that eventually govern the movement of the vehicle. The implementation details of how meta-actions render into steering and acceleration signals are discussed in Section VI. Additionally, the discrete meta-actions defined above must be translated into numeric values in Eq. (11). We experiment with three encodings and choose the one that leads to best performance after training:

• -

  Binary: A one-hot encoding with 5 bits for $a_{i}\in\mathcal{A}_{i}$.

• -

  Discrete: An integer in $(0,5]$ for $a_{i}\in\mathcal{A}_{i}$.

• -

  Frenet: Two integers in $[-1,1]$ for lateral and longitudinal actions.

Inter-agent relations in our mixed-autonomy problem can be broken down into the interactions among autonomous agents, i.e., AV-AV interactions, as well as between autonomous agents and human drivers, i.e., human-AI interactions. Decoupling the two enables us to systematically study the interactions between human drivers with ambiguous SVO and our autonomous agents. We refer to an autonomous agent’s altruism toward a human as sympathy and define cooperation as the altruistic behavior among autonomous agents. Our rationale for decoupling the components of altruism is that they differ in nature. As an instance, sympathy may not be reciprocal as humans are heterogeneous in their SVO but cooperation among autonomous agents is essentially homogeneous, assuming that they hold the same SVO. We investigate each component of altruism separately to better understand the emerging behaviors and the mechanics of inducing altruism in autonomous agents. Following this definition, we can rewrite Eq. (7) as,

where $\theta$ is the sympathy angular phase determining the cooperation-to-sympathy ratio. Parameters $R_{i}^{\mathrm{AV}}$ and $R_{i}^{\mathrm{HV}}$ denote the total utility of other autonomous and human-driven vehicles, respectively, as perceived from the $i$th agent’s perspective. We expand on this topic in Section V-C where we introduce the distributed reward structure.

Following the notions of sympathy and cooperation and the notation of Eq. (12) we decompose the decentralized reward received by agent $I_{i}\in\mathcal{I}$ as,

in which $j\in\widetilde{\mathcal{I}}\setminus\{I_{i}\}$, $k\in(\widetilde{\mathcal{V}}\cup\{M\})\setminus(\mathcal{I}\cap\{M\})$. The $r_{i}$ term denotes the ego vehicle’s driving performance derived from metrics such as distance traveled, average speed, and a negative cost for changes in acceleration to promote a smooth and efficient movement by the vehicle. The cooperative reward term, $r^{\mathrm{AV}}_{i,j}$ accounts for the utility of the ego’s allies. It is important to note that ego vehicle only requires the observation $\tilde{\textbf{o}}_{i}$ to compute $R^{\mathrm{C}}$ and not any explicit coordination or knowledge of the actions of the other agents. The sympathetic reward term, $r^{\mathrm{HV}}_{i,k}$ is defined as

where $u_{k}$ denotes an HV’s utility, e.g., its speed, $d_{i,k}$ is the distance between the observer autonomous agent and the $k$th HV, and $\eta$ and $\psi$ are dimensionless coefficients. Moreover, the sparse scenario-specific mission reward term $r^{\mathrm{M}}_{e}$ in the case of our driving scenario is representing the success or failure of the merging maneuver,

Two cascade multi-layer perceptron (MLP) networks are utilized as the feature extractor network (FEN) and the function approximator network (FAN), each with two layers of size 256 and 128 neurons, respectively, and rectified linear unit (ReLU) non-linearities. As introduced in Section V-A, the temporal information in a vehicle’s observations is captured through integrating the history of the past actions in the observations and the feature extractor network must be able to efficiently extract meaningful patterns from this information. Both networks are trained end-to-end to enforce the feature extractor network to extract the most vital information that is required for estimating the state-action value function. The policy is trained offline and deployed into all agents to be executed in a distributed and online fashion, meaning that each agent makes independent decisions based on its observation but they all follow the same stochastic policy.

As we elaborated in Section III, the non-stationarity of the environment is a major problem in concurrent training of multiple RL agents. We employ a semi-sequential training and policy dissemination algorithm to cope with this challenge and stabilize the training process. Algorithm 1 summarizes our overall methodology which is done in two stages. First, an experience replay buffer (ERB) is filled with data from simulation episodes and then, random samples drawn from this buffer is used for updating the weights of both FEN and FAN networks. For simplicity we refer to the set of all weights for both neural networks as w. We use a novel method for scoring the entries in ERB and drawing them with a probability proportional to that score.

ERB is highly skewed due to the nature of our highway merging scenario. To elaborate, each episode can be morphologically broken down into two parts, straight driving on the highway and the merging point. The former mostly provides information and training samples that are useful for learning the basics of driving and the latter contains the important information regarding the inter-agent coordination and altruistic behavior, which is of our interest. Only a few time steps of each episode contain the merging point and the rest is mostly related to highway cruising. To balance the training data drawn from the experience replay, we randomly draw samples with a probability $p_{\mathrm{ERB}}$ proportional to their spatial distance from the merging point. This method showed better performance when compared to the most common method of prioritizing the experience replay based on a sample’s last resulted reward.

After drawing a training sample from ERB, the agent $I_{i}\in\mathcal{I}$ performs $k_{\mathrm{diss}}$ iterations of training while the weights $\textbf{w}^{-}_{j}$ of all other agents $I_{j}(j\neq i)$ is frozen. The updated weights $\textbf{w}^{+}_{i}$ are then disseminated to the other agents to update their policy. This process is then repeated for all agents until convergence. Doing so enables us to stabilize the training and train all agents concurrently. The key idea is applying incremental updates and keeping the environment stationary in-between the updates so that the optimizer achieves convergence. This semi-sequential algorithm is illustrated in Figure 4 and Algorithm 1.

We modified an OpenAI Gym environment [38] to enable multi-agent training and distributed execution in a mixed-autonomy highway merging scenario. The meta-actions determined by the stochastic policy are translated to low-level steering and acceleration control signals through a closed-loop proportional–integral–derivative (PID) controller. Motion of the vehicles is then governed by a Kinematic Bicycle Model that determines the vehicles’ yaw rate and acceleration. As a common practice in robotics, road segments and the motion of the agents are expressed in Frenet-Serret coordinates and broken into lateral and longitudinal movements.

In order to ensure learning generalizable policies rather than memorizing a sequence of actions by the function approximator network, the initial state of each simulation episode is randomized. This episode initialization is particularly critical as the resulting initial states must be still meaningful and valid for our desired conflictive highway merging scenario. Trivial episodes where the merging vehicle can easily merge into the highway regardless of the AVs’ actions or the episodes where the AVs’ do not have an opportunity to enable safe merging, not only do not add valuable information to the training process but also can lead into misleading measures. The initial longitude and speed of the cruising vehicles are uniformly randomized and the initial longitude $l_{M}(t_{0})$ and speed $v_{M}(t_{0})$ of the merging vehicle are drawn from a clipped-Gaussian distribution $\widetilde{\mathcal{N}}(x;\mu,\sigma,\delta)$ defined as,

where $\mathcal{N}(x;\mu,\sigma)$ denotes a Gaussian distribution and $\mathds{1}$ is the Heaviside step function. We elaborate on initializing episodes via parameters $\mu$, $\sigma$, and $\delta$ in Section VII-E.

Lateral and longitudinal movements of HVs are mimicked by human driver models proposed by Treiber et al. and Kesting  et al.  [39, 40]. The lateral actions of HVs, i.e., the decision to perform a lane change, follow the Minimizing Overall Braking Induced by Lane changes (MOBIL) strategy [40]. MOBIL model allows a lane change only if the resulting acceleration $\mathrm{acc}_{n}>-b_{\mathrm{safe}}$ meets the safety criterion, and the incentive criterion is also satisfied,

with $\mathrm{acc}_{e}$, $\mathrm{acc}_{n}$, and $\mathrm{acc}_{o}$ being the acceleration of the ego HV, the following vehicle in the target lane, and the following vehicle in the current lane, respectively, and $\mathrm{acc}^{\prime}_{e}$, $\mathrm{acc}^{\prime}_{n}$, and $\mathrm{acc}^{\prime}_{o}$ are the corresponding accelerations assuming the ego HV has performed the lane change. $\mathrm{acc}_{\mathrm{th}}$ is the threshold that determines if the ego HV shall performs the lane change. HV’s SVO angle $\phi_{e}$ is also referred to as the politeness factor in the literature and is extracted from the empirical probability distribution illustrated in Figure 2.

The longitudinal acceleration of HVs follows the Intelligent Driver Model (IDM) [39]. The longitudinal Frenet acceleration of a HV, $\ddot{l}_{\mathrm{IDM}}$, is determined by

where $\dot{l}$ denotes the longitudinal Frenet speed of the HV, and the desired Frenet distance to the leading vehicle is controlled by $d^{*}$, defined as,

in which $\Delta\dot{l}$ is the approach rate, and the model parameters $v_{\mathrm{set}}$, $T_{\mathrm{set}}$, $d_{0}$, $\mathrm{acc}_{\mathrm{max}}$, and $\mathrm{acc}_{\mathrm{des}}$ are set speed, set time gap, minimum gap distance, maximum acceleration, and the desired acceleration, respectively. Additionally, the acceleration of the vehicle is a random variable defined as,

with $\mathcal{N}(0,1)$ being a standard Gaussian random variable and $\sigma_{\mathrm{vel}}$ is the standard deviation of the velocity noise at the time step $\Delta t$ of the simulation.

The autonomous agents are trained using the semi-sequential multi-agent Q-learning algorithm that we introduced in Figure 4 and Algorithm 1 for 15,000 episodes that are generated by the procedure discussed in Section VI-A. Training process is repeated and compared across multiple runs to assure the stability of training and that it converges to the similar policies every time. The trained policies are then evaluated for 2,000 randomized novel test episodes to gauge their efficacy. Test episodes are intentionally generated with different and broader initialization range than the training episodes to demonstrate that agents actually are able to learn generalizable policies and not only memorize sequences of actions.

The two key variables in Eq. (13) are $\phi$ and $\theta$ that determine the level of altruism, which is the general term we use for both HVs and AVs, as well as level of sympathy, which is the term for altruism toward HVs only. Our experiments are done in 2$\times$6 settings with different values of $phi$ and $\theta$. Furthermore, we experiment with both autonomous, $M\in\mathcal{I}$, and human-driven, $M\in\mathcal{V}$, mission vehicle. Our experiment settings are:

• •

  HV+E. autonomous agents are egoistic ($\phi_{i}=0$ for $I_{i}\in\mathcal{I}$), and the mission vehicle is HV ($M\in\mathcal{V}$);

• •

  HV+C. autonomous vehicles are cooperative only ($\phi_{i}=\phi^{*}$ and $\theta_{i}=\pi/2$ for $I_{i}\in\mathcal{I}$), and the mission vehicle is HV ($M\in\mathcal{V}$);

• •

  HV+SC. autonomous vehicles are sympathetic and cooperative ($\phi_{i}=\phi^{*}$ and $\theta_{i}=\pi/4$ for $I_{i}\in\mathcal{I}$), and the merging vehicle is HV ($M\in\mathcal{V}$);

• •

  AV+E/C/SC. Duals of the the above cases with autonomous mission vehicle ($M\in\mathcal{I}$).

In HV+SC and AV+SC scenarios where autonomous agents have both sympathy and cooperation components, we set the sympathy angle to $\theta=\pi/4$ for the sake of fairness and to avoid imposing bias between HVs and AVs as they both carry humans or goods and neither should have a pre-assumed advantage over the other. The SVO angle $\phi$ is however tuned to reach the optimal level of altruism, we elaborate on this topic in Section VII-D and derive the optimal SVO angle $\phi^{*}$.

To gauge the impact of the aforementioned manipulated variables and other configurable parameters, 3 metrics are chosen that despite being correlated with each other, provide different insights on the efficacy of our solution. As a traffic-level metric, the average distance traveled by HVs and AVs is logged during simulation episodes. Additionally, counting the percentage of the episodes that experience a successful merge enables us to probe the overall social importance of a solution. Safety is also gauged through counting the percentage of episodes that contain at least one crash.

Social and individual performance of altruistic and purely egoistic agents are compared through the 3 key hypotheses:

• •

  H1. While egoistic AVs fail to account for a merging HV, AVs that hold both sympathy and cooperation elements explore ways to enable safe and seamless merging. Therefore, we expect HV+SC to outperform HV+E and HV+C settings.

• •

  H2. AVs with $\phi\neq 0$ are able to implicitly learn the SVO of HVs and guide them to improve the overall performance of the group.

• •

  H3. There exists a social value orientation angle ${0<\phi^{*}<\pi/2}$ for autonomous agents that can both lessen the number of crashes and improve the number of successful merges.

Figure 5 illustrates an overall comparison between the settings defined in Section VII-A. Focusing on the cases with a human-driven merging vehicle, it is evident that in the absence of the sympathy component in AVs, i.e., in HV+E and HV+C settings, merging fails in the majority of episodes. Failed merging leads to a crash in our simulator as vehicles cannot stop on the highway nor the merging ramp and the merging vehicle that fails to merge collides with the barrier at the end of the merging ramp. This assumption is made to make our simulations more realistic and avoid unfeasible solutions that require full-stop on the highway. Therefore, most of the crash cases shown in Figure 5 are due to unsuccessful merging and not the lack of basic driving skills in HVs and AVs. As an additional evidence, independent crashes that are not relevant to a failed merge are also plotted in Figure 5, which confirms the fact that the vehicles hold sufficient basic skills to maneuver on a highway and avoid collisions.

Figures 5 and 6 clarify the positive social impact that sympathy and cooperation make in terms of reducing the total number of crashes and failed merging. However, a counter-argument against this comparison can be the fact that a rather conservative model is used to mimic HVs in our simulations and this might limit their capability in merging. To investigate this claim, we repeat the comparison with an autonomous mission vehicle that is more risk-tolerant and attempts more creative ways to merge into the highway. In the AV+E setting that AVs only care about their individual utility, although the results are better compared to HV+E, even the autonomous mission vehicle still fails to safely merge in more than 1/3 of the episodes. We conclude that our test case indeed creates a competitive and conflictive scene for the vehicles and showcases how incorporating sympathy and cooperation components in the reward structure of AVs leads to socially-desirable outcomes and improves safety and traffic flow. Figure 6 provides further intuition to this comparison by depicting a sampled set of mission vehicle’s trajectories in different experiment settings. It is evident that un-sympathetic does not allow the mission vehicle to merge, causing its trajectory to end in the merging ramp.

As it was emphasized before, the autonomous agents do not have access to an explicit behavior model of human drivers and instead implicitly learn this model from experience during the training episodes. Although we employ a rather conservative model of human drivers to showcase our concept, it is expected that given sufficient training data, the autonomous agents can extract models of more complex human behaviors as well. However, sensitivity of our solution to these models and the effect of human behaviors on inter-agent coordination is a topic worthy of investigation which we leave for our future work. As a relevant observation, AVs implicitly learn to predict the behavior of HVs and the fact that HVs commonly act egoistically (refer to Figure 2) and do not slow down for the merging vehicle. Hence, they do not rely on the HVs and instead compromise on their individual reward to enable the highway merging.

Our experimental scenarios in Section VII-A are defined based on the optimal SVO angle $\phi^{*}$ of the autonomous agents. This parameter clearly has an important impact on the behavior of AVs and thus the safety and traffic-flow metrics. We trained a large set of agents with different SVO angles and tested them in our case study driving scenario. The optimal SVO angle is then defined as the angle that results the best performance metrics, i.e., least number of episodes with collisions and failed merges. We formulate this simple optimization objective as the convex combination of the two metrics,

where $f_{\mathrm{C}}$ and $f_{\mathrm{MF}}$ are the percentage of episodes with a crash and failed mission, respectively. The hyper-parameter $\xi$ determines the importance of each performance metric and we choose it to be $\xi=0.5$ as otherwise it could bias the training process by putting more emphasis on either of the metrics. Figure 8 illustrates how the two metrics change when the autonomous agents’ SVO is varied from $\phi=0$ (purely egoistic) towards $\phi=\pi/2$ (purely altruistic). It is worth mentioning that neither of the two extremes seems optimal and a point between caring about others and being selfish leads to the most socially-desirable outcome.

A fair critique to the behavior of sympathetic cooperative agents can be the fact that the Guide AV, i.e., AV3 in Figure 3, decelerates and therefore slows down the group of vehicles behind only to allow the mission vehicle to merge. In other words, the utility of a big group of vehicles is being compromised for the sake of the mission vehicle. To investigate the fairness and effectiveness of this outcome, we measure the average distance traveled by HVs and AVs. Figure 9 reveals how despite the fact that in the HV+SC setting a group of vehicles need to slow down to open up space for the mission vehicle, eventually both HVs and AVs manage to travel more distance when compared to a similar setup with egoistic agents (HV+E). It should be noted that the effect of Guide AV’s deceleration gradually propagates through the platoon of vehicles in behind and only affects a limited group of vehicles as the traffic in the platoon is not rigid and can contract and expand.