GenAD: Generative End-to-End Autonomous Driving

The goal of end-to-end autonomous driving can be formulated as obtaining a planned $f$-frame future trajectory $\mathbf{T}(T,f)=\{\mathbf{w}^{T+1},\mathbf{w}^{T+2},\cdots,\mathbf{w}^{T+f}\}$ for the ego vehicle given the current and past $p$-frame sensor inputs $\mathbf{S}=\{\mathbf{s}^{T},\mathbf{s}^{T-1},\cdots,\mathbf{s}^{T-p}\}$ and trajectory $\mathbf{T}(T-p,p+1)=\{\mathbf{w}^{T},\mathbf{w}^{T-1},\cdots,\mathbf{w}^{T-p}\}$.

$\displaystyle\mathbf{T}(T-p,p+1),\mathbf{S}\rightarrow\mathbf{T}(T,f),$

(1)

where $\mathbf{T}(T,f)$ denotes a $f$-frame trajectory starting from the $T$-th frame, $\mathbf{w}^{t}$ denotes the waypoint at the $t$-th frame, and $\mathbf{s}^{t}$ denotes the sensor input at $t$-th frame.

The first step of end-to-end autonomous driving is to perceive the sensor inputs to obtain high-level descriptions of the surrounding scene. These descriptions usually include semantic map [30, 34] and instance bounding box [27, 26]. To achieve this, we follow a conventional vision-centric perception pipeline to extract bird’s eye view (BEV) $\mathbf{B}\in\mathbb{R}^{H\times W\times C}$ features first and then build on them to refine map and bounding box features.

Image to BEV. We basically follow BEVFormer [27] to obtain the BEV features. Specifically, we use a convolutional neural network [11] and feature pyramid network [31] to obtain multi-scale image features $\mathbf{F}$ from camera inputs $\mathbf{s}$. We then initialize $H\times W$ BEV tokens $\mathbf{B}_{0}$ as queries and use deformable cross-attention [56] to transfer information from the multi-scale image feature $\mathbf{F}$:

$\displaystyle\mathbf{B}=\text{DA}(\mathbf{B}_{0},\mathbf{F},\mathbf{F}),$

(2)

where $\textbf{DA}(\mathbf{Q},\mathbf{K},\mathbf{V})$ denotes the deformable attention block consisting of interleaved self-attention and deformable cross-attention layers using $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ as queries, keys, and values, respectively. We then align the BEV features from the $p$ past frames into the current coordinate system and concatenate them as the final BEV features $\mathbf{B}$.

BEV to map. As semantic map elements are usually sparse in the BEV space, we follow a similar concept [20, 21] and use map tokens $\mathbf{M}\in\mathbb{R}^{N_{m}*C}$ to represent semantic maps. Each map token $\mathbf{m}\in\mathbf{M}$ can be decoded to a set of points in the BEV space by a map decoder $d_{m}$ representing a category of map elements and their corresponding positions. Following VAD [21], we consider three categories of map elements (i.e., lane divider, road boundary, and pedestrian crossing). We use the global cross-attention mechanism to update learnable initialized queries $\mathbf{M}_{0}$ from BEV tokens $\mathbf{B}$:

$\displaystyle\mathbf{M}=\text{CA}(\mathbf{M}_{0},\mathbf{B},\mathbf{B}),$

(3)

where $\textbf{CA}(\mathbf{Q},\mathbf{K},\mathbf{V})$ denotes the cross-attention block composed of interleaved self-attention and cross-attention layers using $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ as queries, keys, and values, respectively.

BEV to agent. Similar to the representation of semantic maps, we adopt a set of agent tokens $\mathbf{A}$ to represent the 3D position of each instance in the surroundings. We use deformable cross-attention to obtain the updated agent tokens $\mathbf{A}$ from the BEV tokens $\mathbf{B}$:

$\displaystyle\mathbf{A}=\text{DA}(\mathbf{A}_{0},\mathbf{B},\mathbf{B}),$

(4)

where $\mathbf{A}_{0}$ are learnable tokens as initialization.

Having obtained the agent tokens $\mathbf{A}$, we employ a 3D object detection head $d_{a}$ to decode the position, orientation, and category information from each agent token $\mathbf{a}$.

Instance-centric scene representation. As prediction and planning mainly focus on the instances of agents and ego vehicles, respectively, we propose an instance-centric scene representation to comprehensively and efficiently represent the autonomous driving scenario. We first add an ego token $\mathbf{e}$ to the learned agent tokens $\mathbf{A}$ to construct a set of instance tokens $\mathbf{I}=\text{concat}({\mathbf{e},\mathbf{A}})$.

Existing methods [16, 15, 21] usually perform motion prediction and planning in a serial manner, which ignores the effect of future ego movements on the agents. For example, the lane shift of the ego car would possibly affect the action of rear cars, rendering the motion prediction results inaccurate. Differently, we enable high-order interactions between the ego vehicle and other agents by performing self-attention on the instance tokens:

$\displaystyle\mathbf{I}\leftarrow\text{SA}(\mathbf{I},\mathbf{I},\mathbf{I}),$

(5)

where $\textbf{SA}(\mathbf{Q},\mathbf{K},\mathbf{V})$ denotes the self-attention block composed of self-attention layers using $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ as queries, keys, and values, respectively.

Furthermore, to perform accurate prediction and planning, both the agents and the ego vehicle need to be aware of the semantic map information. We thus employ cross-attention between the updated instance tokens and the learned map tokens to obtain map-aware instance-centric scene representations:

$\displaystyle\mathbf{I}\leftarrow\text{CA}(\mathbf{I},\mathbf{M},\mathbf{M}).$

(6)

The learned instance tokens $\mathbf{I}$ incorporate high-order agent-ego interactions and are aware of the learned semantic maps, which are compact yet contain all the necessary map and instance information to perform motion prediction and trajectory planning.

We find that the objectives of motion prediction of other agents and planning of the ego vehicle share the same output space and are essentially the same. They both aim to produce a high-quality realistic trajectory of the concerned instance, given semantic maps and interactions with other agents. The goal for the proposed GenAD can be then formulated as inferring the future trajectory $\mathbf{T}$ given the map-aware instance-centric scene representations $\mathbf{I}$.

Different from existing methods which directly output the trajectory using a simple decoder, we model it as a trajectory generation problem $\mathbf{T}\sim p(\mathbf{T}|\mathbf{I})$ considering its uncertain nature.

The trajectories of both the ego vehicle and other agents are highly structured (e.g., continuous) and follow certain patterns. For example, most of the trajectories are straight lines as the vehicle driving at a constant speed, and some of them are curved lines with a near-constant curvature when the vehicle turning right or left. Only in very rare cases will the trajectories be zig-zagging. Considering this, we adopt a variational autoencoder (VAE) [24] architecture to learn a latent space $\mathbb{Z}$ to model this trajectory prior. Specifically, we employ a ground-truth trajectory encoder $e_{f}$ to model $p(\mathbf{z}|\mathbf{T}(T,f))$, which maps the future trajectory $\mathbf{T}(T,f)$ to a diagonal Gaussian distribution on the latent space $\mathbb{Z}$. The encoder $e_{f}$ outputs two vectors $\mathbf{\mu}_{f}$ and $\mathbf{\sigma}_{f}$ representing the mean and variance of the Gaussian distribution:

$\displaystyle p(\mathbf{z}|\mathbf{T}(T,f))\sim N(\mathbf{\mu}_{f},\mathbf{\sigma}_{f}),$

(7)

where $N(\mathbf{\mu},\mathbf{\sigma}^{2})$ denotes a Gaussian distribution with a mean of $\mathbf{\mu}$ and standard deviation of $\mathbf{\sigma}$.

The learned distribution $p(\mathbf{z}|\mathbf{T}(T,f))$ contains the prior of the ground-truth trajectories, which can be leveraged to improve the authenticity of motion prediction and planning for traffic agents and ego vehicles.

Having obtained the latent distribution of the future trajectories as prior, we need to explicitly decode them from the latent trajectory space $\mathbb{Z}$.

While a straightforward way is to directly use an MLP-based decoder to output trajectory points in the BEV space to model $p(\mathbf{T}(T,f)|\mathbf{z})$, it fails to model the temporal evolution of the traffic agents and ego vehicle. To consider the temporal relations of instances at different time stamps, we factorized the joint distribution $p(\mathbf{T}(T,f)|\mathbf{z})$ as follows:

	$\displaystyle p(\mathbf{T}(T,f)|\mathbf{z}^{T})=p(\mathbf{w}^{T+1}|\mathbf{z}^{T})\cdot p(\mathbf{w}^{T+2}|\mathbf{w}^{T+1},\mathbf{z}^{T})$		(8)
	$\displaystyle\cdots p(\mathbf{w}^{T+f}|\mathbf{w}^{T+1},\cdots,\mathbf{w}^{T+f-1},\mathbf{z}^{T}).$

We sample a vector from the distribution $N(\mathbf{\mu}_{f},\mathbf{\sigma}_{f})$ as the latent state at the current time stamp $\mathbf{z}^{T}$. Instead of decoding the whole trajectory at once, we adopt a simple MLP-based decoder $d_{w}$ to decode a waypoint $\mathbf{w}=d_{w}(\mathbf{z})$ from the latent space $\mathbf{Z}$, i.e., we instantiate $p(\mathbf{w}^{T+1}|\mathbf{z})$ with $\mathbf{w}=d_{w}(\mathbf{z})$.

We then adopt a gated recurrent unit (GRU) [7] as the future trajectory generator to model the temporal evolutions of instances. Specifically, the GRU model $g$ takes as inputs the current latent representation $\mathbf{z}_{t}$ and transforms it into the next state $g(\mathbf{z}_{t})=\mathbf{z}_{t+1}$. We can then decode the waypoint $\mathbf{w}^{t+1}$ at the $(t+1)$-th time stamp using the waypoint decoder $\mathbf{w}^{t+1}=d_{w}(\mathbf{z}_{t+1})$, i.e., we model $p(\mathbf{w}^{t+1}|\mathbf{w}^{T+1},\cdots,\mathbf{w}^{t},\mathbf{z})$ with $d_{w}(g(\mathbf{z}_{t}))$.

Compared with a single decoder that directly outputs the whole trajectory, the waypoint decoder performs a simpler task of only decoding a position in the BEV space and the GRU module models the movement of agents in the latent space $\mathbf{Z}$. The produced trajectory is thus more realistic and authentic considering the prior knowledge in this learned structured latent space. We illustrate the proposed trajectory prior modeling and latent future trajectory generation in Figure 3.

In this subsection, we present the overall architecture of the proposed GenAD framework for vision-centric end-to-end autonomous driving. Given surrounding camera signals $\mathbf{s}$ as inputs, we first employ an image backbone to extract multi-scale image features $F$ and then use deformable attention to transform them into the BEV space. We align the BEV features from the past $p$ frames to the current ego-coordinate to obtain the final BEV feature $\mathbf{B}$. We perform global cross-attention and deformable attention to refine a set of map tokens $\mathbf{M}$ and agent tokens $\mathbf{A}$, respectively. To model the high-order interactions between traffic agents and the ego vehicle, we combine agent tokens with an ego token and perform self-attention among them to construct a set of instance tokens $\mathbf{I}$. We also use cross-attention to inject semantic map information into the instance tokens $\mathbf{I}$ to facilitate further prediction and planning.

As realistic trajectories are highly structured, we learn a VAE module to model the trajectory prior and adopt a generative framework for both motion prediction and planning. We learn an encoder $e_{t}$ to map ground-truth trajectories to the structural space $\mathbf{Z}$ as Gaussian distributions. We then employ a GRU-based future trajectory generator $g$ to model the temporal evolutions of instances in the latent space $\mathbf{Z}$ and use a simple MLP-based decoder $d_{w}$ to decode waypoints from latent representations. We can finally reconstruct the trajectories $\hat{\mathbf{T}}_{a}$ and $\hat{\mathbf{T}}_{e}$ for traffic agents and the ego vehicle by integrating the decoded waypoint of each time stamp. For training, we additionally use a class decoder $d_{c}$ to predict the category for each agent $\hat{\mathbf{c}}_{a}$. To learn the future trajectory encoder $e_{f}$, the future trajectory generator $g$, the waypoint decoder $d_{w}$, and the class decoder $d_{c}$, we follow VAD [21] to impose trajectory losses on the reconstructed and ground-truth trajectories for both traffic agents and the ego vehicle:

	$\displaystyle J_{prior}=L_{tra}(\hat{\mathbf{T}}_{e},\mathbf{T}_{e})+\frac{1}{N_{a}}L_{tra}(\hat{\mathbf{T}}_{a},\mathbf{T}_{a})$		(9)
	$\displaystyle+\lambda_{c}L_{focal}(\hat{\mathbf{C}}_{a},\mathbf{C}_{a}),$

where $||\cdot||_{1}$ denotes the L1 norm, $N_{a}$ is the number of agents, and $L_{focal}$ represents the focal loss to constrain the predicted agent class. $L_{tra}$ denotes the trajectory losses [21] including the L1 discrepancy, ego-agent collision constraint, ego-boundary overstepping constraint, and the ego-Lane directional constraint. $\hat{\mathbf{C}}_{a}$ and $\mathbf{C}_{a}$ represent the predicted and ground-truth classes for all the agents, respectively.

To infer the future trajectory for traffic agents and the ego vehicle from the instance tokens $\mathbf{I}$, we use an instance encoder $e_{i}$ to map each instance token to the latent space $\mathbb{Z}$. The encoder $e_{i}$ similarly outputs a mean vector $\mu_{i}$ and variance vector $\sigma_{i}$ to parameterize a diagonal Gaussian distribution:

$\displaystyle p(\mathbf{z}|\mathbf{I})\sim N(\mathbf{\mu}_{i},\mathbf{\sigma}_{i}).$

(10)

With the learned latent trajectory space to model realistic trajectory prior, the motion prediction and planning can be unified and formulated as a distribution matching problem between the instance distribution $p(\mathbf{z}|\mathbf{I})$ and the ground-truth distribution $p(\mathbf{z}|\mathbf{T}(T,f))$. We impose the Kullback-Leibler divergence loss to enforce distribution matching:

$\displaystyle J_{plan}=D_{KL}(p(\mathbf{z}|\mathbf{I}),p(\mathbf{z}|\mathbf{T}(T,f))),$

(11)

where $D_{KL}$ denotes the Kullback-Leibler divergence.

Additionally, we use two auxiliary tasks to train the proposed GenAD model: map segmentation and 3D object detection. We use a map decoder $d_{m}$ on the map tokens $\mathbf{M}$ and an object decoder $d_{o}$ on the agent tokens $\mathbf{A}$ to obtain the predicted maps and 3D object detection results. We follow the task decoder design of VAD [21] and employ bipartite matching for ground truth matching. We then impose semantic map loss [53] $J_{map}$ and 3D object detection loss [27] $J_{det}$ on them to train the network.

The overall training objective of our GenAD framework can be formulated as:

$\displaystyle J_{GenAD}=J_{prior}+\lambda_{plan}J_{plan}+\lambda_{map}J_{map}+\lambda_{det}J_{det},$

(12)

where $\lambda_{plan}$, $\lambda_{map}$, and $\lambda_{det}$ are balance factors. The proposed GenAD can be trained efficiently in an end-to-end manner. For inference, we discard the future trajectory encoder $e_{f}$ and sample a latent state following the instance distribution $p(\mathbf{z}|\mathbf{I})$ as input for the trajectory generator $g$ and waypoint decoder $d_{w}$. Our GenAD models end-to-end autonomous driving as a generative problem and performs future prediction and planning in a structured latent space, which considers the prior of realistic trajectories to produce high-quality trajectory prediction and planning.

We conducted extensive experiments on the widely adopted nuScenes [2] dataset to evaluate the proposed GenAD framework for autonomous driving. The nuScenes dataset is composed of 1000 driving scenes, where each scene provides RGB and LiDAR video of 20 seconds. The ego vehicle is equipped with 6 surrounding cameras with $360^{\circ}$ horizontal FOV and a 32-beam LiDAR sensor. nuScenes provides semantic map and 3D object detection annotations for keyframes at 2Hz. It includes 1.4M 3D bounding boxes of objects from 23 categories. We partitioned the dataset into 700, 150, and 150 scenes for training, validation, and testing, respectively, following the official instructions [2].

Following existing end-to-end autonomous driving methods [15, 16], we use the L2 displacement error and collision rate to measure the quality of planning results. The L2 displacement error measures the L2 distance between the planned trajectory and the ground-truth trajectory. The collision rate measures how often the self-driving vehicle collapses with other traffic participants following the planned trajectory. By default, we take as inputs 2s history (i.e., 5 frames) and evaluate the planning performance at the 1s, 2s, and 3s future.

We adopted ResNet50 [12] as the backbone network to extract image features. We take as input images with a resolution of $640\times 360$ and use a $200\times 200$ BEV representation to perceive the surrounding scene. For fair comparisons, we basically use the same hyperparameters as VAD-tiny[21]. We fixed the number of BEV tokens, map tokens, and agent tokens to $100\times 100$, 100, and 300, respectively. Each map token contains 20 point tokens to represent a map point in the BEV space. We set the hidden dimension of each BEV, point, agent, ego, and instance token to 256. We used a latent space with a dimension of 512 to model trajectory prior and also set the hidden dimension of the GRU to 512. We employed 3 layers for each attention block.

For training, we set the loss balance factors to 1 and use the AdamW [37] optimizer with a cosine learning rate scheduler [36]. We set the initial learning rate to $2\times{10}^{-4}$ and a weight decay of 0.01. By default, we trained our GenAD for 60 epochs with 8 NVIDIA RTX 3090 GPUs and adopted a total batch size of 8.

Main results. We compared GenAD with state-of-the-art end-to-end autonomous driving methods in Table 1. We use bold and underlined numbers to represent the best and second-best results, respectively. We see that our GenAD achieves the best L2 errors among all the methods with an efficient inference speed. Though UniAD [16] outperforms our method with respect to the collision rates, it employs additional supervision signals during training such as tracking and occupancy information, which has been verified to be vital to avoid collision [16]. However, these labels in the 3D space are difficult to annotate, rendering it not trivial to use less labels to achieve competitive performance. Our GenAD is also more efficient than UniAD, demonstrating a strong performance/speed trade-off.

Perception and prediction performance. We further evaluated the perception and prediction performance of the proposed GenAD model and compared it with VAD-tiny [21] with a similar model size, as shown in Table 2. We use mean average precision (mAP) to measure the 3D object detection performance, and [email protected], [email protected], and [email protected] to evaluate the quality of predicted maps. For motion prediction, we report the end-to-end prediction accuracy (EPA) for both cars and pedestrians, which is a more fair metric for end-to-end methods to avoid the influence of falsely detected agents. For motion planning, we report the average L2 error and collision rate (CR) over 1s, 2s, and 3s.

We observe that GenAD outperforms VAD on all the tasks with a similar inference speed. Specifically, GenAD achieves better motion prediction performances by considering the influence of the ego vehicle on the other agents. GenAD also demonstrates superior performance in 3D detection and map segmentation, showing a better consistency between perception, prediction, and planning.

Effect of the instance-centric scene representation. We conducted an ablation study to analyze the effectiveness of the instance-centric scene representation, as shown in Table 3. We first added the ego-to-agent interaction with the proposed method to VAD-tiny [21], and observe a large improvement in both the L2 error and the collision rate. We also removed the ego-to-agent interaction in our GenAD model by masking the self-attention matrix to dissect its effect. We see that the collision rate performance drops greatly. We think this is because without considering the high-order interactions between the ego car and the other agents, it becomes very difficult to learn the true underlying distributions of trajectories.

Effect of the generative framework of autonomous driving. We also analyzed the designs of the proposed future trajectory generation model, which is composed of two modules: trajectory prior modeling (TPM) and latent future trajectory generation (LFTG). When using TPM alone, we directly decode the entire trajectory from the latent space. With only the LFTG module, we use a gated recurrent unit to gradually produce waypoints given the instance-centric scene representation. We see that both modules are effective and improve the planning performance. Combining the two modules further improves the performance by a large margin. This verifies the importance of factorizing the joint distribution as in (8) to release the full potential of latent trajectory prior modeling.

Visualizations. We provide a visualization of our GenAD model with comparisons with VAD-tiny [21] with a similar model size, as shown in Figure 4. We visualize the map segmentation, detection, motion prediction, and planning results on a single image and provide the surrounding camera inputs as references. We see that GenAD produces better and safer trajectories than VAD in various scenarios including going straight, overtaking, and turning. For the challenging scenarios when multiple agents are involved in complex traffic scenes, our GenAD still demonstrates good results while VAD cannot safely move through.