BITS: Bi-level Imitation for Traffic Simulation

We take an agent-centric approach to traffic simulation, i.e., each agent makes decisions in a decentralized manner without explicit coordination. As mentioned previously, this allows for flexible integration with existing simulation frameworks containing other types of simulated agents and encourages the emergence of new interactive behaviors. We focus on simulating vehicle traffic in this work, but an agent can be any type of road user captured in driving logs (e.g., cyclists, pedestrians). We use $s$ and $c$ to denote the dynamic state and decision-relevant context for an agent, respectively. Specifically, state $s$ includes the position, heading, and velocity of an agent. Context $c=(I,S)$ includes a local semantic map $I$ and the $h$ previous states of an agent and its $N$ neighboring agents $S_{t-h:t}=\{s^{(0)}_{t-h:t},s^{(1)}_{t-h:t},...,s^{(N)}_{t-h:t}\}$. Given the decision context information $c_{t}$ and the current state $s_{t}$, the goal of a traffic simulation model $\pi_{\theta}$ is to generate the next state of the agent $s_{t+1}=\mathcal{T}(\pi_{\theta}(c_{t}),s_{t})$ subject to a dynamics transition function $\mathcal{T}(\cdot)$. We use a simple unicycle model with dynamics constraints as $\mathcal{T}$ and defer more realistic vehicle dynamics models to future works.

We leverage driving logs captured in the real world [16, 17] to train our traffic model. Since log data readily includes semantic maps and the trajectories of all observed agents, we can treat driving logs as a set of multi-agent expert demonstration sequences $\tau=\{c^{(i)}_{0},s^{(i)}_{0},c^{(i)}_{1},s^{(i)}_{1},...,c^{(i)}_{T},s^{(i)}_{T}\}_{i=0}^{N}$ and formulate traffic simulation as a supervised imitation learning problem. However, the nature of urban driving poses significant technical challenges. First, the decision process is partially-observed as the model does not have access to the underlying intent of the demonstrator and other decision-relevant cues such as the turn signals of other vehicles. Accordingly, action supervisions are inherently ambiguous and are usually modeled with probabilistic distributions [34, 33, 35]. Although this ambiguity complicates training, effectively modeling action distributions also enables generating diverse counterfactual traffic simulations. Second, since each agent acts without explicit coordination, their joint behaviors create a combinatorial space of possible future states. Such uncertainty makes generating stable traffic simulations extremely challenging. Below, we describe how our approach can generate multimodal simulations with a stochastic hierarchical policy and mitigate uncertainty in state evolution with a prediction-and-planning module.

The goal of our traffic simulation model is to produce diverse and plausible behaviors by learning from real-world driving logs as demonstrations. Most existing methods in trajectory prediction use deep latent variable models (e.g., VAEs) to capture the behavior distributions. However, findings in the imitation learning literature [60] suggest that learning to generate stable long-horizon behaviors requires a large amount of training data. Our method instead decomposes the learning problem into (1) training a high-level goal network that captures the spatial distribution of possible short-term goals, and (2) training a deterministic goal-conditional policy that learns to reach the predicted goal. The spatial goal network exploits the 2D birds-eye-view structure of driving motion and represents the spatial goal distribution efficiently with a 2D grid. This decomposition additionally moves the burden of modeling multi-modal trajectories to the high-level goal predictor, enabling the low-level goal-conditioned policy to reuse goal-reaching skills to improve sample efficiency.

Spatial goal network. The spatial goal network is trained to predict the distribution of the short-term goal pose (2D position and heading) $p(\tilde{s}_{t+H}|c_{t})$ of an agent given its decision context $c_{t}$. Following prior works [22, 61], we first encode the decision context into a rasterized semantic map, which includes the semantic map $I$ and past agent trajectories rasterized as 2D bounding boxes in additional channels. The model takes as input the rasterized semantic map and outputs a 2D grid of goal likelihood as well as residual components to refine the predicted goal location. The output takes the shape of a 4-channel tensor with the same spatial size as the input rasterized map. Channel $0$ is the likelihood of the coarse goal location 2D probability map. Each pixel in channel $1$ and $2$ is the $(x,y)$ scalar residual relative to the grid location. Channel $3$ is the heading prediction at each grid location. Once a location is selected based on the probability map in channel $0$, the location is corrected by the residual and transformed into a goal pose $\tilde{s}_{t+H}$ in the agent local coordinate frame. We treat the 2D location map as a joint distribution and train via cross-entropy loss across locations. The other channels are trained with masked regression losses (e.g., squared error).

Goal-conditional policy. The goal-conditional policy takes the form of a deterministic trajectory generator $s_{t:t+H}=\pi_{\theta}(c_{t},\tilde{s}_{t+H})$. Although we may further augment the policy with stochastic components, we empirically found it unnecessary as the short-term goal largely reduces the uncertainty in prediction. Inspired by prior works [34, 23], instead of directly regressing each state in an agent’s trajectory the model predicts controls (velocity, change of heading) at each future time step and forward integrates them through an agent’s dynamics model (e.g., extended unicycle dynamics [62] for vehicles). The errors between the predicted and reference trajectory are then backpropagated directly through the dynamics model. Overall, this strategy provides a strong dynamically-grounded learning signal which corrects predictions at earlier steps to reduce errors at later steps.

So far, we have described a bi-level imitation learning method that can generate plausible traffic simulations from limited data. The policy can synthesize diverse behaviors by sampling from the multi-modal spatial goal predictor. However, the performance of the policy remains bounded by the size and coverage of training data. Driving logs are biased towards nominal behaviors and contain almost no safety-critical situations such as collisions or driving off-road. The objective of generating diverse behaviors further amplifies this challenge, as agents are encouraged to enter previously-unseen regions of the map and create new interactions. As a result, to achieve stable long-horizon simulations, agents must generate reasonable behaviors even at states where guidance from training data is lacking.

To this end, we propose to augment our policy with a prediction-and-planning module to stabilize long-horizon rollouts. The module draws action samples $a_{t}$ from the stochastic bi-level policy $\pi_{\theta}$ described above and selects the action that minimizes a rule-based cost function $C$ given the predicted future states of the environment $S_{t:t+H}$, that is, $\arg\min_{a_{t}\sim\pi_{\theta}}C(a_{t},S_{t:t+H},c_{t})$. This approach is similar to the motion planning pipeline in a typical modular AV stack, with the important difference that we use our learned policy to generate human-like motion trajectory candidates. The key idea is that the policy $\pi_{\theta}$ can directly follow the data likelihood at in-distribution states, where most action samples are rule-following, and receive corrective guidance at states where the most likely actions may lead to bad consequences. In addition, the sampling module allows for agile adjustment of the simulator (e.g. level of diversity, emphasis on multiple objectives) without retraining. Below we describe the model for future state prediction and the cost function for action selection.

Future state prediction. Since we assume an analytical vehicle dynamics model and known static map, the main task of the model is to predict the future motion trajectories of nearby agents. We follow a typical trajectory prediction pipeline and featurize each agent by its local and global scene context. Specifically, we use RoIAlign [63] to crop the features extracted by the intermediate layer of a deep CNN. The per-agent features are then concatenated with a global scene context feature (extracted by the final layer of the same CNN) to make the final trajectory prediction $s^{(i)}_{t:t+H}$ for each neighboring agent $i$. The model is illustrated in Fig. 1. We use deterministic prediction in this work and defer more sophisticated probabilistic prediction and planning to future work.

Cost-based trajectory selection. We consider two rule-based costs: collision and road departure. The collision cost is computed based on distances from the corners of two bounding box rectangles using distances between the four corners. To calculate the road departure cost, we first generate a distance map that records the Manhattan distance to the drivable area in pixel space (obtained efficiently via $D$ convolution steps). The resulting distance map assigns zero to points within the drivable area, with values increasing outside the drivable area until saturating at $D$. We directly use this Manhattan distance value as a penalty. Note that both cost terms are zero for nominal trajectories, i.e., trajectories that do not result in collision and road departure, minimizing their effect on selecting among rule-following action samples.

Designing metrics for simulation is particularly difficult because of the lack of ground truth. As a result, metrics such as average displacement error (ADE) and final displacement error (FDE), commonly used to evaluate trajectory prediction, do not suit the evaluation of simulation models. To address this evaluation gap, we propose three types of simulation metrics: (i) metrics measuring how much simulated agents violate common traffic rules, such as driving off-road or causing collisions with other agents; (ii) metrics measuring the statistics of simulation rollouts, including resemblance to collected driving logs in terms of driving characteristics such as speed profile, control effort, coverage of the driving area, and behavior diversity between different simulation trials; (iii) data-driven metrics learned from real-world driving logs, such as measuring the likelihood of simulation rollouts under data-driven trajectory forecasting models. Here we describe (ii) and (iii) in details.

Coverage and diversity. To calculate how much of a scene is covered by the agents, we first compute a simulator’s trajectory distribution via Density Estimation with a Gaussian kernel over all time steps of the simulator’s rollouts, focusing on the 2D spatial distribution of the trajectory. To measure the coverage of the map, we count the number of grid points where the KDE estimate is above a threshold, separating the count between drivable areas and non-drivable areas. To measure the diversity of stochastic policies, we run multiple trials with the same initial condition and collect the density estimates for each trial. Given two different trials, we compute the Wasserstein distance between the two density profiles. In particular, all grid points with non-zero density are flattened and a distance matrix is computed containing the Euclidean distances between every pair of grid points. The density profile is then normalized to sum to 1 and the Wasserstein distance can be computed efficiently as in [64]. For $n$ trials of the same scene, we calculate the Wasserstein distances between the $n(n-1)/2$ pairs of density profiles and take the mean as a metric for diversity:

$$\mathrm{Diversity}=\frac{2}{n(n-1)}\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}\mathrm{Wass}(\rho_{i},\rho_{j}),$$

where $\mathrm{Wass(\cdot,\cdot)}$ is the Wasserstein distance and $\rho_{i}$ is the density profile for the $i$-th trial.

Learned metrics. There are many potential ways to learn a metric that evaluates whether simulated behavior is human-like. One such way is to evaluate simulated agent trajectory likelihoods using a prediction model trained from real-world driving logs. To evaluate this possibility, we develop an occupancy-based prediction model that predicts where an agent will be in future time steps. The model uses a similar structure to our spatial goal network and discretizes the position space into bins. The model is trained to minimize a cross-entropy loss function and its predictions are then used to compute the trajectory likelihoods of simulated trajectories. We include more details in the Appendix.

Datasets. The Lyft dataset [16] contains 1000 hours of driving data collected along a 6.8 mile route in Palo Alto. The dataset contains many repeated trips along each road segment. Since the annotations are auto-generated using a perception stack, there are many labeling errors including inaccurate agent positions, headings, and semantic types [18]. In contrast, nuScenes [17] contains 5.5 hours of accurate manually-labeled trajectories spanning two cities (Boston and Singapore) with more diverse scenarios and denser traffic. Through these datasets, we compare our method to baselines under different learning challenges such as noisy labels and small training sets. For both datasets, we train all models on trajectories from the train split and conduct evaluation on 100 scenes randomly sampled from the validation split. We consider only vehicle simulation in this paper and defer simulating other types of agents (e.g., pedestrians, cyclists) to future works.

Simulation environments. As stated above, we initialize our simulation environments from real driving data, leading to realistic agent placements and dynamic states. Note that all scenes are drawn from the validation split previously unseen to the trained models. Each agent in the scene is independently controlled by replicas of the same model. The simulation runs at a frequency of 10 Hz and results are reported on 20-seconds simulation episodes.

Baselines. We consider methods from both the traffic simulation and trajectory prediction literature. SimNet [22] is a deterministic behavior-cloning model for traffic simulation. TrafficSim is an agent-centric adaptation of the original traffic simulation method in [23] that features an isotropic Gaussian CVAE. We remove the scene consistency loss in training since we do not assume control over all agents. SocialGAN [30] learns to generate trajectories through adversarial imitation. TPP is adapted from Trajectron++ [34], comprised of a discrete CVAE with Gaussian trajectory decoder for each discrete mode. We also consider variants of these methods augmented with our planning-and-control module (marked with “+p”), i.e., selecting future action samples with a cost function. We also evaluate ablations of our method, BITS (max) takes the maximum-likelihood action instead of sampling and BITS (sample) samples actions for rollouts without the prediction-and-planning module described in Sec. 3.3. All methods share the same rasterized input format, ResNet-18 encoder backbone, and MLP-based trajectory decoder networks.

Metrics. As mentioned in Sec. 3.4, designing evaluation metrics for traffic simulation is challenging as there are not single quantities that can summarize the performance of a method, and we cannot easily compare with ground truth dataset trajectories as our goal is to generate new and diverse simulation rollouts. To address this problem, we consider three types of evaluation metrics.

Environment metrics measure rule violations, environment coverage, and trajectory diversity. Both coverage and diversity as described in Sec. 3.4 are calculated from 5 simulation trials with different seeds per scene. We define a critical failure as an agent colliding with other agents or driving off-road for more than 1s. Failure Rate (FR) is the average fraction of agents experiencing a critical failure in a scene. We also report raw collision rate (coll) and road departure rate (offroad).

Dataset metrics compare simulation and ground truth data statistics. They are computed using a normalized Wasserstein distance between the histograms of the driving profiles of the simulated and recorded trajectories. We focus on speed and jerk, commonly used as driver comfort metrics, in the main paper and include others in the Appendix. We also report scene Average Distance Error (sADE) which measures the average position differences between simulated and recorded trajectories. Note that sADE is not suitable for measuring simulation realism and is included only as a reference since it heavily penalizes alternative simulations (e.g., turning left vs. going straight).

Finally, learned metrics as described in Sec. 3.4 measure simulation realism based on a likelihood model trained from real-world driving log.

Table 1 and Table 2 show quantitative results of closed-loop simulation on Lyft and nuScenes datasets, respectively. Fig. 2 qualitatively visualizes trajectories generated by selected methods and Fig. 3 shows a more detailed analysis on time-to-failure caused by road departure error. We make the following core observations from these results.

Recorded driving data is noisy. As stated above, both datasets contain certain levels of labeling noise indicated by non-zero failure rates for ground truth data (labeled “Dataset"), with higher noise in Lyft than nuScenes and a majority of errors stemming from vehicle-vehicle collisions due to imprecise bounding box labels. Our method BITS is able to achieve lower failure rates than even the recorded trajectories in both datasets.

Sample efficiency. nuScenes contains far fewer training samples than Lyft, necessitating high sample efficiency in order for a policy to manage compounding errors over long simulations. As shown in Table 2, even without the prediction-and-planning module, both variants of BITS achieve low failure rates compared to other baselines. For a more direct analysis, we also report the mean time-to-failure in Fig. 3, observing that failure rates in nuScenes increase significantly over the course of simulation for the non-hierarchical policy baselines and remain low for BITS and its ablations.

BITS generates diverse and stable simulations. We observe that the baseline methods exhibit trade-offs between generating diverse rollouts and overfitting to a single mode of behaviors. For example, in Lyft, TPP suffers from mode collapse which yields low failure rates at the cost of low diversity. TrafficSim achieves relatively high diversity and coverage, but also high failure rates. This observation is corroborated by the visualizations shown in Fig. 2, where all simulation trials by TPP are visually identical and resembles the ground truth (on right, titled “Dataset"), and while the trajectories generated by TrafficSim are diverse, some suffer from collisions and road departures. In contrast, BITS simultaneously attains high diversity and coverage with a low failure rate. This contrast is more pronounced in nuScenes where the training set is small. BITS achieves a balanced performance even without the prediction-and-planning module thanks to its high sample efficiency.

Prediction-and-planning is not always effective. The prediction-and-planning module is generally effective in reducing failure rates, with two important exceptions: (1) when action samples are not diverse, and (2) when all action samples lead to failure. Case (1) is exemplified by TPP in the Lyft environment (Fig. 2), where the model’s predictions overfit to a single behavior mode. Case (2) is exemplified by SocialGAN in both datasets. While the simulations are relatively diverse, they have high failure rates, entailing poor action sample quality. The prediction-and-planning module has negligible effects on the simulation in both cases.

Quantifying behavioral realism. As discussed above, evaluating simulation realism remains a challenging open problem because there is no single correct answer for traffic simulation. Here we consider both dataset statistics and learned metrics as a proxy for quantifying behavioral realism. For Lyft, we see that all methods achieve comparable speed and jerk statistical distances relative to the recorded trajectories. As expected, SimNet has the lowest sADE due to its behavior cloning objective. In nuScenes, BITS achieves comparable performance to SimNet in dataset metrics, showing that our method does not have to sacrifice behavioral realism for diversity and stability.

Finally, we consider the learned metric described in Sec. 3.4. To show that this occupancy likelihood-based metric indeed captures meaningful data likelihoods, we roll out ground truth trajectories with different levels of Ornstein-Uhlenbeck noise [65] and measure the predicted likelihood score. As shown in Fig. 3, the likelihood score decreases smoothly as the noise intensity grows, indicating that the learned metric captures the effect of disturbances well. We report the learned likelihood scores for representative baselines in Table. 3. Sensibly, we see that in both datasets the ground truth dataset trajectories have the highest likelihood scores. BITS yields comparable or higher likelihood scores than other baselines, with scores on par with ground truth dataset trajectories on nuScenes data.

As shown in Fig. 4, our work includes three learned component: spatial goal network, goal-conditional policy, and the future state predictor. We will describe their shared input, network backbones, and the output heads below. Table 4 includes key hyperparameter selections.

Input. All three networks take rasterized decision context as input. A decision context $c_{t}$ includes (1) a semantic map including information such as lane marks, crosswalks, and lane boundaries and (2) kinematic history states of all agents (vehicles) in the view. For the semantic map, we follow the original rasterization scheme included in Lyft [16] and nuScenes [17] development kit, respectively. Specifically, Lyft semantic map is a 3-channel tensor that includes lane boundary, lane areas, traffic light faces, and crosswalk information. nuScenes’s semantic maps are 7-channel tensors including lane, road segment, drivable area, road divider, lane divider, pedestrian crossing, and walkaway. Following prior works [22, 61], we use a binary occupancy map same size as the semantic map to represent the kinematic states (i.e., position, heading, and extent) of agents at a given timestep. Each agent in the view is rasterized as filled bounding boxes on the occupancy map. State histories are represented as multi-channel occupancy maps. We concatenate the semantic map and the rasterized agent history map channel-wise and use the tensor as input for all models.

Architecture details. All three networks use an identical ResNet-18 ConvNet backbone to encode an input decision context tensor into a compact feature vector. Here we describe the output heads of each network.

Goal-conditional policy. Given the feature and a goal pose, i.e., target position and heading in the agent coordinate frame, the goal-conditional policy uses a MLP-based trajetory decoder to generate a length-$h$ trajectory as actions. As mentioned in the main text, inspired by prior works [34, 23], instead of directly regressing each state in an agent’s trajectory, the decoder predicts control inputs (velocity, change of heading) at each future time step and forward integrates them through an agent’s dynamics model (e.g., extended unicycle dynamics [62] for vehicles) to obtain the trajectory predictions. The model is trained end-to-end with an L2 regression loss against the recorded trajectories in the dataset, similar to a regular behavior cloning objective.

Future state predictor. The network is part of our prediction-and-planning module that selects action based on the predicted future states of the environment. Since we assume an analytical vehicle dynamics model and known static map, the main task of the network is to predict the future motion trajectories of nearby agents. As described in the main text, we follow a typical trajectory prediction pipeline and featurize each agent by its local and global scene context. Specifically, we use RoIAlign [63] to crop the features extracted at an intermediate layer of the ResNet-18 encoder. Specifically, we use a cropping window of $7\times 7$ at the end of the second block of the network. The per-agent features are then concatenated with a global scene context feature (extracted by the final fully-connected layer of the ResNet-18) to make the final trajectory prediction $s^{(i)}_{t:t+H}$ for each neighboring agent $i$. We only make predictions for agents that are visible in the rasterized map. The model is trained with an L2 regression loss against the recorded neighboring agent trajectories in the dataset. The goal-conditional policy and the future state prediction network share encoder network weights and are trained jointly.

Spatial goal network. The spatial goal predictor extends the ConvNet encoder backbone with a U-Net-style decoder network (mirroring the encoder size) to generate the spatial goal prediction. Specifically, the model generates a 2D grid of goal likelihood as well as residual components to refine the predicted goal location. The output takes the shape of a 4-channel tensor with the same spatial size as the input rasterized map. Channel $0$ is the likelihood of the coarse goal location 2D probability map. We use a grid size of 0.5 meter per pixel. Each pixel in channel $1$ and $2$ is the $(x,y)$ scalar residual (in meters) relative to the grid location. Channel $3$ is the heading prediction at each grid location in radian. Once a location is selected based on the probability map in channel $0$, the location is corrected by the residual and transformed into a goal pose $\tilde{s}_{t+H}$ in the agent local coordinate frame. Note that it is possible to further discretize the heading prediction into discrete bins [66], but we empirically found that the heading prediction has negligible impact on the goal-conditional policy. We treat the 2D location map as a joint distribution and train via cross-entropy loss across locations. The other channels are trained with regression losses (e.g., squared error).

Training setup. We train all models on shared cloud computing nodes equipped with NVIDIA Tesla V100 GPUs (32gb GPU memory). We train all models for 100k iterations (gradient steps) and choose the checkpoint based on their respective offline validation metrics (e.g., prediction error). On average, training takes between 20-30 hours to complete with single-GPU jobs.

Cost functions for prediction and planning. As mentioned in the main text, we consider two rule-based costs: collision and road departure. We follow prior work [34] and compute the collision cost based on distances from the corners of two bounding box rectangles using distances between the four corners. The minimum distance $d_{\min}(\cdot)$ is approximated as:

$$d_{\min}(\Delta X_{1:4},\Delta Y_{1:4},L,W)=\max\left\{\begin{aligned} |&\Delta X_{1}|-\frac{L}{2},...,|\Delta X_{4}|-\frac{L}{2},\\ &|\Delta Y_{1}|-\frac{W}{2},...,|\Delta Y_{4}|-\frac{L}{2}\end{aligned}\right\},$$

which is further illustrated in Fig. 5. We then add the following collision loss

$$\text{Col}=\text{Sigmoid}(-\alpha d_{\min}-\beta),$$

where $\alpha$ and $\beta$ are parameters used to shape the sigmoid loss. In our experiments, $\alpha=1.0,~{}\beta=4.0$.

To calculate the road departure cost, we first generate a distance map that records the Manhattan distance to the drivable area in pixel space. To be specific, we first pick a maximum distance constant $D$ and set all pixels inside the drivable area to 0, and all pixels outside the drivable area to $D$. Then we perform the following convolution step $D$ times:

$$x_{i,j}=\min\{x_{i,j},x_{i-1,j}+1,x_{i+1,j}+1,x_{i,j-1}+1,x_{i,j+1}+1\},$$

where $x_{ij}$ is the value at the $i,j$ coordinate. The resulting distance map assigns zero to points within the drivable area, with values increasing outside the drivable area until saturating at $D$. We directly use this Manhattan distance value as a penalty. To account for the size of the vehicle, we perform RoIAlign on the distance map with vehicle patches that take the vehicles’ size and orientation into account. Note that both cost terms are close to zero (the collision loss may be nonzero due to the sigmoid function) for nominal trajectories, i.e., trajectories that do not result in collision and road departure, minimizing their effect on selecting among rule-following action samples.

Learned metrics. To compute likelihood of a simulated trajectory, we use an occupancy-based prediction model that predicts where an agent will be in future time steps. The model uses a similar structure to our spatial goal network and discretizes the position space into bins. The model is trained to minimize a cross-entropy loss function and its predictions are then used to compute the trajectory likelihoods of simulated trajectories. During evaluation, the likelihood of an agent’s trajectory is calculated in a receding horizon manner. For each time step, we compare its rollout trajectory for the next $T=20$ steps to the predicted occupancy and calculate the likelihood and aggregate the whole rollout duration by taking the mean. The likelihood values of multiple agents in the scene are also aggregated by taking the mean.

SimNet. SimNet [22] is a deterministic behavior-cloning model for traffic simulation. Prior work also show that its rasterized encoding scheme yields strong performance in trajectory prediction tasks [61]. We use the same rasterization input and CNN-based encoding backbone for all baselines and our BITS model.

SocialGAN. SocialGAN is adapted from a trajectory prediction model [30]. The model is trained using conditional generative-adversarial learning, where the condition is the decision context and the generation target comes from the trajectory dataset. To facilitate a fair comparison, the model uses the same encoding backbone and MLP-based trajectory decoder as all other methods.

TPP (Trajectron++). TPP is adapted from the Trajectron++ model [34], which is a CVAE model with a discrete latent space and generates a Gaussian Mixure Model (GMM) as trajectory prediction. We modified the encoder to take rasterized features as input, which shares the same map encoder as the proposed BITS model. The TPP model generates predictions of the control input, which are propagated through a unicycle model to obtain the trajectory prediction. For simplicity, we directly let the decoder predict the variance of predicted trajectory instead of propagating the variance through the dynamics as in [34]. The latent dimension is chosen to be 10 in our experiments.

TrafficSim. TrafficSim is an agent-centric adaptation of the original TrafficSim work [23], meaning that agents in a scene are controlled by independent replica of the same model instead of by a single model that explicitly coordinates their actions. The TrafficSim baseline uses an isotropic Gaussian CVAE to model action distributions and facilitate diverse simulation. We adopt a similar differentiable dynamics trajectory decoder from the original paper but remove the scene consistency loss in training since we do not assume control over all agents.

Here we provide the exhaustive list of (non-learned) evaluation metrics we considered in this work and report performances in Table 5 for Lyft and Table 6 for nuScenes.

• •

  FR: The average fraction of failed agent in each scene. The failure can be caused by either collision failure or offroad failure (defined below).

• •

  collFR: The average fraction of failed agent in each scene, where the failure is caused by agent colliding with any other agent in the scene at any timestep throughout an episode.

• •

  offroadFR: The average fraction of failed agent in each scene, where the failure is caused by agent departing drivable region for more than 1 second (10 timesteps for a simulation frequency of 0.1 second).

• •

  coll: The average fraction of agent colliding with another agent. We in addition list the type of collision occurred (rea, front, side). Note that each collision type is independently calculated, i.e., each agent can have multiple type of collision per episode.

• •

  offroad: The fraction of timesteps that an agent spend outside of drivable area, averaged across agents per scene.

• •

  coverage and diversity: As described in detail in the main text, both metrics measure the diversity of rollouts across multiple simulation trials with the same initial condition. The coverage metric calculates the (discretized) map area covered by multiple simulation trials with the same initial condition. The diversity metric in addition computes the Wasserstein distances among the density profiles of the map coverage attained by different simulation trials. In practice, we accumulate the metrics across 5 trials and compute the coverage density by discretizing the map at a 2m by 2m resolution.

• •

  Dataset metrics. Dataset metrics compare the driving profile of simulation trajectories with trajectories recorded in the dataset. We consider speed, lon acc (longitudinal acceleration magnitude), lat acc (latitudinal acceleration magnitude), and jerk. To calculate the difference between the driving profile of the simulation and the dataset, we first collect histograms of these driving profile measurements for both the simulated and the dataset trajectories across all evaluation scenes. We then calculate the distances between the histograms of the simulation and the dataset using Wasserstein distance normalized by a constant. We use 20 bins for all histograms, with a speed range of $[0,30](m/s)$, an acceleration magnitude range of $[0,10](m/s^{2})$, and a jerk magnitude range of $[0,10](m/s^{3})$.

  We also report scene Average Distance Error (sADE) and scene Final Distance Error (sFDE), which measures the average and the final position differences between simulated and recorded trajectories, respectively. Note that sADE and sFDE are not suitable for measuring simulation realism and is included only as a reference since it heavily penalizes alternative simulations (e.g., turning left vs. going straight).

As described above and in the main text, we use two cost terms: collision and road departure in our prediction-and-planning module. We conduct an ablation study on different planning cost weight setting using the Lyft dataset and report the result in Fig. 6. Because there are only two cost terms, we mainly consider the ratio of the cost weights. We set offroad cost weight to either $0.0$ and $1.0$ and continuously vary the collision cost weight at a log scale. We observe that both terms are effective at reducing rates of their corresponding cause of failure, the reduction saturates when the weight is above a certain value, and the overall failure reduction performance is not sensitive to the cost weight ratio.

Here we investigate the impact of the planning horizon $H$ in BITS’s planning and control module. We train all three components (i.e., goal-conditional policy, future state predictor, spatial goal network) with the same set of prediction horizons: $\{10,20,50,80\}$ on the Lyft dataset. We report performances using essential metrics in Table 7. We observe that $H=\{10,20\}$ yield similar performances, but the failure rates increase significantly as the model expands its prediction horizon to $H=\{50,80\}$. This is because predicting long-term future is extremely challenging, especially when all agents are controlled by stochastic policies. The model’s decisions can be easily misled by wrong future predictions, leading to critical mistakes. At the same time, we also observe that while short-term predictions result in more successful simulation, they lead to less smooth driving behavior caused by more frequent braking and acceleration, as indicated by the dataset metrics (e.g., higher acc and jerk). Hence an important future direction is to dynamically adjust the prediction and planning horizon by taking into account prediction uncertainties.

We include additional qualitative results for BITS. Rollouts using the nuScenes dataset is shown in Fig. 7 and that of the Lyft dataset is shown in Fig. 8. Each row shows different rollouts from the same initial scene configuration. The recorded trajectories are shown in the rightmost column for reference.