Control-Aware Prediction Objectives for Autonomous Driving

Model-based reinforcement learning (MBRL) methods learn a dynamics model of an autonomous agent to predict which control decisions lead to states with higher objective rewards [7, 22, 21]. MBRL prediction is related to AV prediction, with the main difference being that AVs predict the trajectories of other human agents and not those of the autonomous agent. Nevertheless, several MBRL works have recently challenged the common assumption that the better a dynamics model’s predictive accuracy, the better the downstream policy will maximize reward. For example, [*]lambert2020objective [17] show task-agnostic loss functions used to train dynamics models are often uncorrelated with episode rewards, an issue termed “objective mismatch”.

Indeed, learned dynamics models need not be accurate everywhere in the state space, only in the areas that help maximize rewards [3]. Some RL works investigate training models insofar as they improve estimating the value function [11, 2], policy gradient [14, 1], or ability to reach a goal state [23]. Others optimize downstream policies directly using Bayesian optimization to search model parameters [3]. Work by [*]donti2017task [8] points out that in practice we often want a combination of training a model to optimize its likelihood as well as a downstream task term, although in the context of constrained optimisation. Similarly, [*]lambert2020objective [17] correlate both metrics by increasing the weight in the loss of data points closer to data the optimal controller generates.

Unfortunately for AV applications, such objective-aware prediction methods are often inapplicable for several reasons. First, MBRL assumes access to an objective reward function, reward samples, or goal state, but objective measures or goals of desirable driving are often difficult to define. By contrast, human-designed AV control systems are often preferable for verification and interpretability reasons. Second, a common assumption in RL is that the policy is either differentiable or stochastic (in the case of policy gradients), whereas real-world AV control systems often contain complex logic that is neither differentiable nor stochastic. Our work focuses instead on how to learn prediction given access to a safe, potentially non-differentiable controller.

Map information can help incorporate prior knowledge into prediction metric design. For example, since the ego vehicle drives on the road, pedestrian forecasting errors could be given more weight on road surfaces than otherwise. Work by [*]shridhar2020beelines [32] use maps to help focus on potential collisions with other agents by generating a set of candidate ego trajectories along known lane tracks that any controller might follow. This method does not assume a particular downstream controller, but makes an educated guess as to what a reasonable controller might do. Another map-based prediction metric is Drivable Area Compliance (DAC) [5], which counts the proportion of model samples that exit the drivable area. A conceptual difference with our method is that our prediction metrics assume access to the specific downstream controller that will be used at test-time, improving test-time performance. Since controllers already consider road information, we circumvent the need to explicitly design prediction metrics around mapping information, which can incur additional hyperparameters, such as the relative costs of predicting if an agent will traverse either road / sidewalk / building.

[*]philion2020learning [26] propose a control-aware 3D object perception metric called Planning KL-divergence (PKL) based on how perceptual errors cause distributional divergence in the ego’s distribution of planned paths, compared to a planner with perfect observations, measured by the KL divergence. In contrast, our work focuses on prediction objectives, and additionally demonstrates how the new objective empirically affects online-control using a driving simulator. We also avoid KL distance losses in our work since this assumes non-trivial stochasticity in the controller or data, which is not always the case. Other works have also used KL policy distances to investigate how observations can be compressed while preserving human-like actions had they remained uncompressed [28]. Work by [*]piazzoni2020modeling [27] also investigates how perceptual metrics affect downstream planning but are specific to perception.

Let $\mathbf{x}\in\mathcal{X}$ denote past trajectory information about all agents, used to make probabilistic predictions $\hat{\mathbf{y}}\in\mathcal{P}_{\mathcal{Y}}$ about the future multi-agent trajectories $\mathbf{y}\in\mathcal{Y}$. Trajectories are predicted up to time horizon $T$, and $\mathbf{y}_{T}$ denotes the future state at time $T$. As the intents of other agents are usually uncertain, we use a probabilistic prediction model $q_{\theta}$ with trainable parameters $\theta$ to sample the motion of others: $\hat{\mathbf{y}}\sim q_{\theta}(\mathbf{Y}|\mathbf{x})$, denoting likelihoods as $q_{\theta}(\mathbf{y}|\mathbf{x})\doteq q_{\theta}(\mathbf{Y}=\mathbf{y}|\mathbf{x})$. If multiple samples are taken, $\hat{\mathbf{y}}^{k}$ refers to the $k$th sample, and to single out the $n$th agent we overload notation using $\hat{\mathbf{y}}_{n}$, and use $\mathbf{y}_{\text{ego}}$ as the AV’s future trajectory. Given such predictions, the AV controller $\pi$ outputs ego controls $\mathbf{u}\in\mathcal{U}$ to anticipate and avoid colliding with other agents’ future trajectories: $\mathbf{u}=\pi(\mathbf{y})$.

We assume our AV stack performs behavior prediction before control, a common assumption [31]. While conditioning behavior prediction on ego’s intent provides more accurate prediction, for sake of simplicity we assume that other agents do not anticipate the AV’s future, only the AV anticipates the other agents’ future trajectories in order to avoid collisions.

Common prediction metrics in the literature and in prediction benchmarking challenges–including Argoverse Forecasting [5], Lyft Prediction [12], Waymo Open Motion [10], and AIODrive [35]–are summarized in Table I:

Most metrics compare the Euclidean distance between either the full predicted state-sequence $\hat{\mathbf{y}}$ (or final state $\hat{\mathbf{y}}_{T}$) with the true sequence $\mathbf{y}$ (or final state $\mathbf{y}_{T}$) an agent took, as recorded in data. Probabilistic models are typically trained to minimize the negative log likelihood (NLL) of the data. All such metrics are agnostic to road geometry and downstream planning, which implicitly assumes that all other agents’ forecasts are equally relevant. For example, consider two pedestrians: one walking ahead of the ego vehicle and one behind. Assuming independent pedestrian motion, the NLL objective factorizes as: $-\log q_{\theta}(\mathbf{y}^{\text{ahead}},\mathbf{y}^{\text{behind}}|\mathbf{x})=-\log q_{\theta}(\mathbf{y}^{\text{ahead}}|\mathbf{x})-\log q_{\theta}(\mathbf{y}^{\text{behind}}|\mathbf{x})$. Notice that this prediction metric is equally concerned with both $\mathbf{y}^{\text{ahead}}$ and $\mathbf{y}^{\text{behind}}$. Intuitively, accurate prediction of the pedestrian ahead of the ego vehicle is more important for safe motion planning since the ego’s planned path is more likely to intersect with $\mathbf{y}^{\text{ahead}}$ than $\mathbf{y}^{\text{behind}}$. How can prediction metrics become “aware” that errors in predicting $\mathbf{y}^{\text{ahead}}$ have greater downstream consequences than errors in $\mathbf{y}^{\text{behind}}$?

Most predictive metrics in the literature are agnostic to $\mathbf{u}$ and simply use a delta function to only score correct trajectory predictions, recovering the standard log likelihood metric: $\text{Gain}_{\mathbf{x},\mathbf{y}}(\theta)=\int\delta(\mathbf{y}-\hat{\mathbf{y}})q_{\theta}(\hat{\mathbf{y}}|\mathbf{x})\text{d}\hat{\mathbf{y}}=q_{\theta}(\mathbf{y}|\mathbf{x})$. However, we are interested in utilities that are a function of $\mathbf{u}$ in order to weight predictions by their downstream effect on the ego’s control. For instance, we could score trajectory predictions based on the resultant ego controls $\pi(\hat{\mathbf{y}})$ matching the ego’s behavior under knowledge of the true future trajectories $\pi(\mathbf{y})$: $\text{Gain}_{\mathbf{x},\mathbf{y},\pi}(\theta)=\int\delta(\pi(\mathbf{y})-\pi(\hat{\mathbf{y}}))q_{\theta}(\hat{\mathbf{y}}|\mathbf{x})\text{d}\hat{\mathbf{y}}.$ This integral is unfortunately intractable to derive or estimate, but softer utility functions can be used instead. One example is $||\pi(\hat{\mathbf{y}})-\pi(\mathbf{y})||_{1}$, which we include as a baseline in Table II. Optimizing this controller output error guides the learning process towards predicting controller inputs (predicted trajectories) accurately, insofar as they result in the correct control. Any trajectory errors that do not induce a change in the AV’s control are thus considered inconsequential and ignored.

We propose a GRU encoder-decoder architecture with an attention mechanism as introduced in [34]. Our method weights the agent predictions using attention factors between agents $\mathbf{x}$ and the AV’s future trajectory $\mathbf{y}_{\text{ego}}$. The predictive model is a function parameterized by $\theta$ noted $q_{\theta}:\mathcal{X}\to\mathcal{P}_{\mathcal{Y}\times\mathcal{Y}_{\text{ego}}}$. We note $\theta=\{\theta_{\text{ego}},\theta_{\text{agent}}\}$ where $\theta_{\text{ego}}$ is the set of parameters for the ego decoder only; $\mathcal{X}$ is the past observation space and $\mathcal{P}_{\mathcal{Y}\times\mathcal{Y}_{\text{ego}}}$ the probability spaces of future trajectories: $\mathcal{P}_{\mathcal{Y}_{\text{ego}}}$ for the ego and $\mathcal{P}_{\mathcal{Y}}$ for other agents.

We use an architecture similar to [18, 20] where we train a model with multi-head attention; the ego agent attends the other agents. The ego predictions are used as a proxy for the actual planner to compute the importance weights of other agents:

The ego-attention blocks in figure Fig. 2 are heads of a multi-head attention mechanism. The computation performed by each head is given below:

The attention vector is given by

where $Q$ is the query matrix, $K$ the key matrix, and $\sigma$ the softmax operation that normalizes the attention vector (for details see [34]). The encoded $\text{output}=\alpha V$ is a weighted mean of the value vectors over the agents (including ego).

Inspired by [20] the attention model produces outputs in the form of a sequence of Gaussian mixtures for each agent and is trained to minimize the NLL for all agents and the ego trajectory predictions. However, for our application, ego prediction is not the goal, it is only a proxy to compute the importance weights of the pedestrian contribution to the ego behavior.

We propose to use attention coefficients $\alpha$ as importance factors in a weighted sum of per-human state prediction loss (as opposed to uniform weighting). Algorithm 1 summarizes how the model is trained with importance weighting. If multiple heads are used, the attention coefficients are averaged:

The attention predictor imitates a planner that interacts with the other agents to avoid collision. The only way for the predictor to interact with the other agents is through attention. Therefore, as the model learns the correlations between the planner’s trajectories and the agents trajectories, larger attention coefficients are given to the agents that cause larger reactions from the controller. It learns this offline and does not need access to the controller nor its gradient.

Predicting jointly the ego trajectory and the other agents allows us to use the attention coefficients for concern weighting in a single run. The coefficient $\alpha_{0}$ quantifies how independent the ego is from other agents.

This method defines a concern about an agent but not about specific trajectories of that agent. It can define the concern without using the controller because it instead uses an offline-learned model that imitates the controller.

Our second proposal can also be formulated as a re-weighted maximization objective, where we weight the log likelihood of each agent’s trajectory in a scene by its individual contribution to the ego’s control decision. We do this by first enumerating through each agent in a scene, and computing counterfactual outputs from the AV’s controller if every agent traversed their individual trajectory as recorded in the replay buffer, except for agent $n$. If we resample the trajectory that the $n$th agent might otherwise have taken, $\hat{\mathbf{y}}^{k}_{n}\sim q_{\theta}(\mathbf{Y}_{n}|\mathbf{x})$, we can compute the control output that would result:

to compare against the control had no agent deviated from their recorded trajectories:

The difference in these two hypothetical controls corresponds to how much an individual agent affects the ego vehicle, and can represent the concern associated with predicting this particular agent in this particular instance accurately. If the model is probabilistic, then taking multiple samples $(K>1)$ helps ensure high importance even if the other agent only might cause a control deviation:

which we use as weights for predictive model training:

We summarize our counterfactual action discrepancy method in Algorithm 2. One benefit of this approach compared to the attention CAPO is that it is

There are various choices for utilities, or weights for traditional module metrics. In Table II we summarize the several baselines methods, including NLL and our novel proposals.

We implement our pedestrian prediction scenario in the CARLA simulator’s Town01. A single ego vehicle is commanded to drive down a road that is adjacent to sidewalks which are populated with pedestrians. Occasionally, a pedestrian will cross the street and the ego agent must slow to avoid a collision when necessary.

The ego vehicle adjusts its longitudinal control to balance the competing priorities of maintaining the current speed limit (45mph) while avoiding collisions with the pedestrians crossing the road (either their current or future-predicted distance on the road ahead). This balance is performed by an Intelligent Driver Model [33]. It uses predictions to estimate the closest collision distance and controls the vehicle to stop ahead of that point.

Pedestrians spawn at random locations on the sidewalk and are then provided a long-range navigation goal that is also uniformly sampled from the sidewalk. When the long-range goal is reached, another is sampled to replace it. To induce pseudo-random motion, a short-range goal is also generated at each time step. This goal is generated by projecting a point 4m along the path to the long-range goal, starting at the pedestrian’s location. The lateral offset $\beta_{t+1}$ of the short-range goal is generated by sampling from a normal distribution centered about the previous lateral offset $\beta_{t}$ after it has been scaled down (to drive it towards the long-range goal):

where $\sigma$ is the variance of the noise, and $\epsilon\in[0,1)$ is the commitment to the long-range goal.

When on the sidewalk, pedestrians are programmed to walk at speeds sampled about 2m/s while navigating around other pedestrians to avoid collisions and, occasionally, will pause outside of shops. Each different kind of pedestrian is defined with various noise levels, commitment, and stopping chance.

Pedestrians may also randomly decide to cross the road. The probability increases if their velocity vector points towards the road and increases greatly when the pedestrian is close to the road. While crossing, they travel at 2 m/s in the shortest path possible, i.e., perpendicular to the road direction. To increase task difficulty, the probability that pedestrians will cross the road is increased at test time.

1) Oracle distribution The pedestrian behavior is simulated with a known distribution at each time step, which is sampled to produce a trajectory. The trajectory distribution is approximated by sampling $K=5$ trajectories for each pedestrian. The planner reacts to the trajectory that would cause the closest intersection with its desired path. This method is a perfect probabilistic predictor which is only accessible with simulated data. Its predictions are not biased towards sampling the most critical trajectories, and planning with relatively few samples can yield suboptimal results.

2) Gradient weighting Recent work has also investigated weighing prediction objectives by a measure of local sensitivity of downstream costs to individual predictions [13]. This method first learns a cost function to evaluate ego controls given predicted human trajectories, using inverse optimal control. The method then weights each prediction loss by the gradient magnitude of the cost outputs w.r.t. the prediction inputs. In our work, we consider using the controller directly, forming an analogous baseline: using gradients of the controller w.r.t. the predicted trajectory $||\nabla_{\hat{\mathbf{y}}}\pi(\hat{\mathbf{y}})||_{1}$, or true trajectory $||\nabla_{\mathbf{y}}\pi(\mathbf{y})||_{1}$, included in Table II. While we do not assume differentiable controllers in our own methods, we nevertheless experiment using a differentiable controller to compare against this baseline method.

3) Attention weighting As presented in section IV-B. We train this model with our CAPO method (algorithm 1) and, as a baseline, we compare it with the unbiased prediction as the predictor [20] would produce.

4) Reparameterized Pushforward Policy (R2P2) we use the likelihood-based multi-agent prediction algorithm R2P2 [29] as baseline $\text{Gain}_{\mathbf{y}}$, and also use R2P2 as the base model for all other predictive models apart from the attention model. R2P2 is a autoregressive normalizing flow, capable of expressing multimodal agent trajectories, trained with NLL. We parameterize R2P2 to predict 30 steps with data at 10Hz, corresponding to a 3s prediction for all pedestrians. We train it with our CAPO method (algorithm 2) using $K=10$ samples and we use $K=1$ samples at test time.

The table III presents our results for 100 sequences. We track the performance of the system (prediction and planner) with the success rate and the number of collisions. Three conditions may end a sequence:

• •

  Success: vehicles traverses 200m road without incident.

• •

  Collision: a pedestrian was hurt.

• •

  Time out: the car was too slow ($>60$s).

We also score efficiency and comfort indicators by average speed and average jerk respectively. Finally, we compute the average pedestrian trajectory prediction errors and their downstream effect on the planner with the average displacement error (ADE) and the Control Error equal to $||\pi(\mathbf{y})-\pi(\hat{\mathbf{y}})||_{1}$. The control error measures the downstream effect of the prediction error on the ego’s plans.