Learning Latent Traits for Simulated Cooperative Driving Tasks

Our objective is based on the premise of learning a reward function from a dataset, which we cast as an inverse reinforcement learning (IRL) problem. We adopt an adversarial formulation of IRL [27], where the reward function is learned in a generative adversarial network (GAN). This formulation is suited to data-rich applications and robust to recovering the actual reward, up to a constant.

Markov Decision Processes. A Markov Decision Process (MDP) is defined by the tuple $\langle{\mathcal{S}},{\mathcal{A}},{\mathcal{T}},{\mathcal{R}},\gamma,\rho_{0}\rangle$, where ${\mathcal{S}}$ is a state space, $\mathcal{A}$ is an action space, $\mathcal{T}:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{S}}\mapsto[0,1]$ is a stochastic state transition function, $\mathcal{R}:{\mathcal{S}}\times{\mathcal{A}}\mapsto\mathbb{R}$ is a reward function, $\gamma\in[0,1]$ is a discount factor, and $\rho_{0}(s)$ is an initial state distribution. In the IRL setup, ${\mathcal{R}}$ and ${\mathcal{T}}$ are unknown, and must be recovered.

MaxEnt IRL. In maximum-entropy IRL, the objective is to find a policy $\pi^{*}$ that is exponentially close to the set of demonstration trajectories. In other words, it maximizes the entropy-regularized $\gamma$-discounted reward

$$\pi^{*}=\underset{\pi}{\arg\max\ }\mathbb{E}_{\pi}\left[\sum_{t=0}^{T}\gamma^{t}{\mathcal{R}}(s_{t},a_{t})+{\mathcal{H}}(\pi(a_{t}\mid s_{t}))\right]$$

(1)

where ${\mathcal{H}}$ is the entropy of the policy distribution, i.e. ${\mathcal{H}}(\pi)=\mathbb{E}_{\pi}\left[-\log\pi\right]$ over a time horizon $T$. The demonstrations are given by a set of trajectories, which denote the state-action pairs $\tau=(s_{0},a_{0},\ldots,s_{T},a_{T})$. A collection of trajectories, complemented by an identifier of the human driver, $y$, forms a dataset of demonstrations ${\mathcal{D}}$. We assume that trajectories may be of different lengths.

In [27], the adversarial formulation of the MaxEnt problem is solved via a GAN. Within the GAN, the discriminator takes the form

$$D_{\theta}(s,a,s^{\prime})=\frac{\exp(f_{\theta}(s,a,s^{\prime}))}{\exp(f_{\theta}(s,a,s^{\prime}))+\pi_{\omega}(a\mid s)},$$

(2)

where $s^{\prime}\sim{\mathcal{T}}(s,a)$, $D_{\theta}$ is a discriminator network with parameters $\theta$, $f_{\theta}(\cdot)$ is factored as $f_{\theta_{1},\theta_{2}}(s,a,s^{\prime})=g_{\theta_{1}}(s,a)+\gamma h_{\theta_{2}}(s^{\prime})-h_{\theta_{2}}(s)$, $\omega$ are the policy parameters, and $g_{\theta_{1}}$, $h_{\theta_{2}}$ are the base and potential components of the reward.

Policy Optimization. We use policy gradient methods to optimize $\pi$ in (1), i.e. the generator in the IRL problem. One approach is proximal policy optimization (PPO) [28], where a policy update step applies a gradient of the objective $J$:

$$\nabla_{\omega}{J(\pi_{\omega})}=\mathbb{E}_{\omega}\left[\sum_{t=0}^{T-1}\gamma^{t}f_{\theta}(s_{t},a_{t},s_{t+1})\nabla_{\omega}\log\pi_{\omega}(a_{t}\mid s_{t})\right]$$

(3)

which induces a gradient-ascent problem.

We adopt a meta-IRL approach [29] allowing to learn a reward structure and policy for the human from data, conditioned on the characteristics of particular humans. The overall framework is depicted in Fig. 2. A latent variable $z$ is introduced that captures the differences in behaviors between humans, i.e. their traits. Batches of time-series data of the human and their observations, each pre-identified per human, are fed to the learning framework. During each epoch, training alternates between a discriminator and a generator. The discriminator combines a context encoder $q_{\psi}(h\mid\tau)$ that captures the traits of a driver and a reward function $f_{\theta}(s,a,z)$ that encodes the rewards, penalties and preferences of the human agent. Roll-outs generated from a frozen-parameter policy are used to compute the loss terms that allow for the joint optimization of context encoder and reward function. Upon convergence, the discriminator learns to distinguish between the driver-generated and policy-generated trajectories. The discriminator takes the adversarial IRL form:

$$D_{\theta}(s,a,s^{\prime},z)=\frac{\exp(f_{\theta}(s,a,s^{\prime},z))}{\exp(f_{\theta}(s,a,s^{\prime},z))+\pi_{\omega}(a\mid s,z)}$$

(4)

The generator, on the other hand, uses the frozen context encoder and reward function to train a trait-conditioned policy $\pi_{\omega}(\cdot\mid s,z)$ for the human. We describe each of these components below.

Context encoder. The context encoder is represented as a network, $q_{\psi}(z\mid\tau)$, defining the probability of $z$ given the trajectory $\tau$. We represent $q_{\psi}(z\mid\tau)$ as a long short-term memory (LSTM) network [30], taking in a trajectory history. The hidden layer $h$ is fed into two linear layers representing the mean and log-variance of the latent encoding. For all of our experiments, we adopt a single-hidden-layer LSTM with 128 units.

Pooling layer. To account for sparsity of behavior indicative of traits, we pool together examples of drivers, determined based on a unique identifier such that we capture the aggregate behavior of the human’s history of behaviors within the latent embedding. We insert this pooling layer just prior to the linear operations mapping LSTM hidden state $h$ to the statistics of $z$, i.e. $\mu_{z}$, $\log(\sigma_{z})$, and hence it is subsumed within $q_{\psi}(\cdot)$. Although we may use any aggregation operation, we choose an average-pooling operation over $h$ for our experiments. We ensure that the pools are consistently sized by maintaining a fixed pooling size, and shuffling the batches for a particular human at each training round.

Reward function. We adopt a discounted reward which maps traits, $z$, and states, actions, and next-states to a real-valued reward. To remove reward ambiguity while learning, we adopt the form $f_{\theta_{1},\theta_{2}}(s,a,s^{\prime},z)=g_{\theta_{1}}(s,a,z)+\gamma h_{\theta_{2}}(s^{\prime},z)-h_{\theta_{2}}(s,z)$ [27]. We use a two-layer multi-layer perceptron (MLP) with 32 units per layer for our base reward network $g_{\theta_{1}}(\cdot)$ and a single-layer MLP with 32 units as our potential network $h_{\theta_{2}}(\cdot)$.

Policy. Our policy is a 32-unit feed-forward network, $\pi_{\omega}(a\mid s,z)$, that encodes the probability of the human taking action $a$ given its perception of the world and their trait. We use the policy gradient algorithm proximal policy optimization (PPO) [28] to train such a policy using the empirical reward structure and context encoder.

Discriminator losses. Finally, we describe the losses used for training the discriminator. See Table I for a complete list. Loss ${\mathcal{L}}_{1}$ encourages the likelihood of the trajectories under the reward, $p_{\theta}(\cdot)$ to match those of the demonstrated trajectories, $p_{\pi_{h}^{*}}(\cdot)$. Loss ${\mathcal{L}}_{2}$ encourages high mutual information, $I_{p_{\theta}}$, between the trajectories and latent variables. The contrastive loss ${\mathcal{L}}_{3}$ [31] promotes cohesion of pairs of latent variables $z$, $z^{\prime}$ having the same label ($y_{z}=y_{z^{\prime}}$) and separation of pairs $z$, $z^{\prime\prime}$ with dissimilar labels ($y_{z}\neq y_{z^{\prime\prime}})$. Here, labels can represent different driver types, different preferences for receiving AI help, and $\ell(\cdot,\cdot)$ may be any distance measure; we choose the $L_{2}$ norm. One can soften the indicator function $\mathds{1}(\cdot)$ in order to more generally capture a continuum or partial ordering of labels. Finally, ${\mathcal{L}}_{4}$ regularizes the latent variables according to a unit Gaussian. We apply weights on the final three terms: 5.0 for ${\mathcal{L}}_{2}$, 10.0 for ${\mathcal{L}}_{3}$ and $10^{-4}$ for ${\mathcal{L}}_{4}$.

In Figure 3 (right), the controlled Markov model consists of two actions: the vehicle action $a_{t}^{H}$, and the AI’s intervention action $a_{t}^{A}\in\{alert,no\_alert\}$. We introduce a counter variable $c_{t}$ to encode how long a successful intervention by the vehicle AI remains in effect. That is, when the vehicle AI issues an alert, the effect of the alert persists for a fixed time window, during which the acceptance state remains 1.

In our model, the transitions for the acceptance state $i_{t}$ and $c_{t}$ are determined as follows:

	if	$\displaystyle\quad i_{t-1}=0\quad\text{then}\quad\begin{cases}i_{t}=1,c_{t}=0\quad&\text{if}\;a_{t}^{A}=alert\\ i_{t}=0,c_{t}=0\quad&\text{if}\;a_{t}^{A}=no\_alert\\ \end{cases}$		(5)
	if	$\displaystyle\quad i_{t-1}=1,\quad a_{t}^{A}=alert,\quad\text{then}\quad c_{t}=0\quad\text{and}\quad i_{t}=1$		(6)
	if	$\displaystyle\quad i_{t-1}=1,\quad a_{t}^{A}=no\_alert,$
		$\displaystyle\text{then}\quad c_{t}=(c_{t-1}+1)\text{mod}\;N,\begin{cases}i_{t}=1\quad&\text{if}\;c_{t-1}<N-1\\ i_{t}=0\quad&\text{if}\;c_{t-1}=N-1\end{cases}$		(7)

Relations (5) and (6) capture the behavior that, when a driver complies with an alert from the vehicle AI, their acceptance state is always set to be 1, regardless of the acceptance state at the previous time step. Additionally, $c_{t}$ is set to be 0, indicating that the vehicle AI’s intervention has been re-triggered. If the driver’s acceptance state is 0, then it continues to be 0 if no alert has been received. In (7), if the acceptance state is already 1 (which implies that the vehicle AI’s alert was already accepted by the driver at an earlier time step), then the acceptance state continues to be 1 (that is the alert continues to have an effect on the driver) as long as $c_{t}$ remains within the intervention effectiveness window. Once $c_{t}$ is greater than the window size, the acceptance state is reset to 0, if no more alerts are accepted.

Mediating distraction. The distraction variable $d_{t}$ evolves as a controlled Markov chain whose transition probabilities are modulated according to the acceptance state $i_{t}$ of the driver. The amount of modulation is controlled by the intervention effective factor, $\eta$, which captures the driver’s willingness to be influenced by the alert issued by the vehicle AI. Crucially, we assume that $\eta$ depends on the level of cautiousness of the driver. Upon accepting the alert from the vehicle AI, the baseline transition probabilities of the distraction state Markov chain are modulated in such a way the probability of becoming distracted is reduced and of becoming more attentive is increased. Specifically, if $\beta\in[0,1]$ is the baseline probability of transitioning from an attentive state ($d_{t-1}=0$) to a distracted state ($d_{t}=1$), the conditional (modified according to the acceptance state) transition probabilities for the Markov model is given by

$$p(d_{t}=1|d_{t-1}=0)=\textnormal{max}(0,\beta-\eta\mathds{1}(i_{t}=1)).$$

(8)

Likewise, the probability of transitioning from a distracted to an attentive state becomes higher when the driver is willing to accept the AI agent’s alert. Therefore,

$$p(d_{t}=0|d_{t-1}=1)=\textnormal{min}(1,\alpha+\eta\mathds{1}(i_{t}=1))$$

(9)

where $\alpha\in[0,1]$ is the baseline probability of transitioning from a distracted to an attentive state.

From the above equations, we can see that if the driver accepts an alert from the vehicle’s AI, the acceptance state will be set to 1, and as a result the transition probabilities in (8) and (9) are modulated and this modulation remains in effect for at least $N$ time steps.

In our framework, at every timestep $t$, first $c_{t}$ is updated, followed by the acceptance state $i_{t}$ and then finally the distraction variable $d_{t}$.

For $t>0$, the value of $d_{t}$, the distraction state, affects the applied vehicle action $v_{t}$ as follows,

$$v_{t}=\begin{cases}v_{t-1}\quad&\text{if}\,d_{t}=1\\ a_{t}^{H}\quad&\text{if}\,d_{t}=0\end{cases}$$

(10)

with $v_{0}=a_{0}^{H}$.

Human subject validation. We lastly validated the model using a human studies experiment conducted on 500 individuals via a crowd-sourced survey. Each individual was shown a small set of example roll-outs drawn from the model, and was asked to rate how safe, distracted, risky, or similar to their own driving the behavior was. The findings of this study revealed that the intended behavior of the model was broadly consistent with human expectation of distracted and cautious driving, although further analysis is needed to account for judgement variations over a wider set of roll-outs.

Table II shows the full list of driving-related rewards used by the joint human-vehicle AI model capturing a joint policy trained for the actions of both the human and AI system.

$R_{\text{coll}}$, $R_{\text{speed}}$, $R_{\text{right-lane}}$, $R_{\text{merging}}$, $R_{\text{lane-change}}$ are the reward components pertaining to driving performance, while $R_{\text{distraction}}$ pertains to the human and $R_{\text{alert}}$ captures the rewards the vehicle AI receives for issuing sparse alerts. $R_{\text{accept-alert}}$ is the reward term that connects the vehicle AI to the human.

For training our models, we set the coefficients related to driving rewards to values such that the vehicle favors being safe on the right lane, at higher speeds and seeks to minimize lane changes.

To generate training data, we sample from four driver types spanning levels of distraction and levels of cautiousness, as per our running example. We generate $3\times 10^{5}$ timesteps per type, according to the values in Table III. We generate episodes based on a merge scenario, pictured in Fig. 1, having three available lanes, and with a maximum of 20 randomly-placed vehicles. The actions are stepped at a rate of 5 Hz, while the simulator is updated at 15 Hz.

We consider this dataset as ground truth, and execute training based on three assumptions on what aspects of the data are labeled.

Unsupervised.

We assume no supervision on the data, and allow the latent embedding to be learned purely based on the learner’s ability to detect nuances in behavior.

Supervised on driver ID.

We train in the general case where the types of drivers are fully labeled.

Supervised on preferences.

We assume preferences of the driver are identifiable. We argue that such labels could be obtained more easily than for the case with ID supervision, via preference queries given to a subset of drivers [21].

We build training data for each case and assume that 20% of the data is labeled. We use two discriminator updates per training round with a learning rate of $5\times 10^{-4}$, and allow the generator updates for 500 steps with a learning rate of $10^{-2}$. The LSTM takes in a trajectory 20 seconds in length, and we use a pooling size of 8 for each driver ID. For purposes of comparison, we terminate training after $10^{6}$ timesteps. In all experiments, we use a two-dimensional latent space.

In Fig. 4, we notice that the ability to distinguish driver types is heavily influenced by the level of supervision. In the case of driver ID supervision, our training was successful at achieving good separation of latent variables associated with high-distraction levels from those that were lower (comparing cases 0 (Lisa) and 1 (Marge) with 2 (Bart) and 3 (Homer)). The two high-distraction cases (2 and 3) are also highly separated. In the case of ground-truth preferences, each of the cases yield latent embeddings that are relatively-well clustered; the data indicate that the two distinct preference settings separate into clusters, with more separation happening when the driver tends to be more distracted. We hypothesize that the failure to distinguish between different confidence levels in low-distraction cases is likely due to the chosen scenario.

Table IV presents the performance of (1) driver-specific HMI models (rows 1-4) (2) an HMI tuned to an ‘average‘ driver type (Infl. = 6.0, $\eta$ = 0.505, $\beta=0.5$, $\alpha=0.5$), denoted as AvgHMI assisting the four driver types (row 5) and (3) the drivers with no HMI assistance (row 6). The performance is evaluated by computing the average high speed return (Table IV, left) and the average distraction reward per episode (Table IV, right).

We observe that the distraction rewards are the highest when the drivers are assisted by driver-specific HMI models. In particular, we find that the average total distraction reward over an episode is -101.2 for personalized HMI, compared with -283.0 for AvgHMI and -470.8 for NoHMI. Additionally, Bart (less cautious, less receptive and highly distracted) becomes distracted the most across all drivers regardless of the HMI model used. We also see that when the AvgHMI assists Marge and Homer, who are both highly receptive to HMI assistance, the average model is able to do a good job in keeping the distraction rewards fairly high. Since the AvgHMI is trained to assist an ‘average’ driver who tends be more distracted than Marge, but less than Homer, it is effective in increasing Marge’s distraction reward even better than the Marge-specific HMI model. However we do see that this comes at a cost: when AvgHMI assists Marge and Homer, the high speed returns are lower than driver-specific models. The decrease is more prominent for Homer who is much more distractible.

With regards to high speed returns, we observe that in general, driver-specific HMI models are able to help the drivers achieve consistently high speed returns. In particular, we see an average return of 448 over the four individuals, compared with 423 for the AvgHMI, and 427 for NoHMI. Except for Lisa, for all other drivers, there is a decrease in the high speed return when there is no assistance.