Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting

The VAE framework has been used to explicitly learn data distributions. The models based on the VAE framework learn mappings from samples in a dataset to points in a latent space and generate plausible samples from variables drawn from the latent space. The VAE-based generative models are known to suffer from two problems: 1) posterior collapse (that the models ignore the latent variable when generating samples) and 2) blurry sample generation. To mitigate the problems, many approaches have been proposed in the literature, primarily for image reconstruction or synthesis tasks [29, 15, 11, 18, 28, 14, 36, 33]. In trajectory forecasting, some researchers [5, 31] have employed the techniques for the mitigation of the posterior collapse. To mitigate the blurry sample generation, [2] proposed a “best-of-many” sample objective that leads to accurate and diverse trajectory generation.

Because the movement of vehicles on the road is greatly restricted by the lane geometry, many works have been proposed to utilize the lane information provided by High-Definition (HD) maps [9, 5, 31, 24, 10, 27, 12, 23, 16, 26]. There are two types of approaches to the representation of the lane information: 1) rasterizing the components of the HD maps on a 2D canvas to obtain the top-view images of the HD maps, 2) representing each component of the HD maps as a series of coordinates of points. In general, Convolutional Neural Network (CNN) is utilized for the former case while Long Short-Term Memory (LSTM) or 1D-CNN is utilized for the latter case to encode the lane information. In this paper, we adopt the second approach. The centerline of each lane in the HD maps is first represented as a series of equally-spaced 2D coordinates and then encoded by an LSTM network. The ability to handle individual lanes in the HD maps allows us to calculate lane-level scene contexts.

Since instance-level lanes are closely related to the modes of the trajectory distribution, recent works [24, 16, 26, 10] proposed calculating lane-level scene contexts and using them for generating trajectories. Our work shares the idea with the previous works. However, ours differs from them in the way it calculates the lane-level scene contexts, which leads to significant gains in the prediction performance. Instead of considering only a single lane for a lane-level scene context, we also take into account surrounding lanes along with their relative importance. The relative importance is calculated based on the past motion of the target vehicle, thus reflecting the vehicle-lane interaction. In addition, for the interaction between the target vehicle and surrounding vehicles, we consider only the surrounding vehicles within a certain distance from the reference lane as illustrated in Figure 1c. This approach shows improved prediction performance compared to the existing approaches that consider either all neighbors [26] or only the most relevant neighbor [16]. This result is consistent with the observation that only a subset of surrounding vehicles is indeed relevant when predicting the future trajectory of the target vehicle [22].

Assume that there are $N$ vehicles in the traffic scene. We aim to generate plausible trajectory distributions $p(\mathbf{Y}_{i}|\mathbf{X}_{i},\mathcal{C}_{i})$ for the vehicles $\{V_{i}\}_{i=1}^{N}$. Here, $\mathbf{X}_{i}=\mathbf{p}_{i}^{(t-H:t)}$ denotes the positional history of $V_{i}$ for the previous $H$ timesteps at time $t$, $\mathbf{Y}_{i}=\mathbf{p}_{i}^{(t+1:t+T)}$ denotes the future positions of $V_{i}$ for the next $T$ timesteps, and $\mathcal{C}_{i}$ denotes additional scene information available to $V_{i}$. For $\mathcal{C}_{i}$, we use the positional histories of the surrounding vehicles $\{\mathbf{X}_{j}\}_{j=1,j\neq i}^{N}$ and the lane candidates $\mathbf{L}^{(1:M)}$ available for $V_{i}$ at time $t$, where $\mathbf{L}^{m}=\mathbf{l}_{1,...,F}^{m}$ denotes the $F$ equally spaced coordinate points on the centerline of the $m$-th lane. Finally, we note that every positional information is expressed in the coordinate frame defined by $V_{i}$’s current position and heading. According to [16], $p(\mathbf{Y}_{i}|\mathbf{X}_{i},\mathcal{C}_{i})$ can be re-written as

$$p(\mathbf{Y}_{i}|\mathbf{X}_{i},\mathcal{C}_{i})=\sum_{m=1}^{M}\underbrace{p(\mathbf{Y}_{i}|E_{m},\mathbf{X}_{i},\mathcal{C}_{i})}_{\text{mode}}\underbrace{p(E_{m}|\mathbf{X}_{i},\mathcal{C}_{i})}_{\text{weight}},$$

(1)

where $E_{m}$ denotes the event that $\mathbf{L}^{m}$ becomes the reference lane for $V_{i}$. Equation 1 shows that the trajectory distribution can be expressed as a weighted sum of the distributions which we call modes. The fact that the modes are usually much simpler than the overall distribution inspired us to model each mode through a latent variable, and sample trajectories from the modes in proportion to their weights as illustrated in Figure 1b.

We introduce two latent variables $\mathbf{z}_{l}\in\mathbb{R}^{D}$ and $\mathbf{z}_{h}\in\mathbb{R}^{M}$ to model the modes and the weights for the modes in Eq. 1. With the low-level latent variable $\mathbf{z}_{l}$, our forecasting model defines $p(\mathbf{Y}_{i}|E_{m},\mathbf{X}_{i},\mathcal{C}_{i})$ by using the decoder network $p_{\theta}(\mathbf{Y}_{i}|\mathbf{z}_{l},\mathbf{X}_{i},\mathcal{C}_{i}^{m})$ and the prior network $p_{\gamma}(\mathbf{z}_{l}|\mathbf{X}_{i},\mathcal{C}_{i}^{m})$ based on

$$p(\mathbf{Y}_{i}|E_{m},\mathbf{X}_{i},\mathcal{C}_{i})=\int_{\mathbf{z}_{l}}p(\mathbf{Y}_{i}|\mathbf{z}_{l},\mathbf{X}_{i},\mathcal{C}_{i}^{m})p(\mathbf{z}_{l}|\mathbf{X}_{i},\mathcal{C}_{i}^{m})d\mathbf{z}_{l},$$

(2)

where $\mathcal{C}_{i}^{m}\subset\mathcal{C}_{i}$ denotes the scene information relevant to $\mathbf{L}^{m}$. To train our forecasting model, we employ the conditional VAE framework [32] and optimize the following modified ELBO objective [14]:

$$\mathcal{L}_{ELBO}=-\mathbb{E}_{\mathbf{z}_{l}\sim q_{\phi}}[\log p_{\theta}(\mathbf{Y}_{i}|\mathbf{z}_{l},\mathbf{X}_{i},\mathcal{C}_{i}^{m})]\\ +\beta KL(q_{\phi}(\mathbf{z}_{l}|\mathbf{Y}_{i},\mathbf{X}_{i},\mathcal{C}_{i}^{m})||p_{\gamma}(\mathbf{z}_{l}|\mathbf{X}_{i},\mathcal{C}_{i}^{m})),$$

(3)

where $\beta$ is a constant and $q_{\phi}(\mathbf{z}_{l}|\mathbf{Y}_{i},\mathbf{X}_{i},\mathcal{C}_{i}^{m})$ is the approximated posterior network. The weights for the modes $p(E_{m}|\mathbf{X}_{i},\mathcal{C}_{i})$ are modeled by the high-level latent variable $\mathbf{z}_{h}$, which is output of the proposed mode selection network $\mathbf{z}_{h}=f_{\varphi}(\mathbf{X}_{i},\mathcal{C}_{i}^{(1:M)})$.

As shown in Eq. 3 and the definition of the mode selection network, the performance of our forecasting model is dependent on how the lane-level scene information $\mathcal{C}_{i}^{m}$ is utilized along with $\mathbf{X}_{i}$ for defining the lane-level scene context. One can consider two interactions for the lane-level scene context: the VLI and V2I. This is because the future motion of the vehicle is highly restricted not only by the vehicle’s motion history but also by the motion histories of the surrounding vehicles and the lane geometry of the road. For the VLI, the existing works [10, 16, 24, 26] considered only the reference lane. For the V2I, [16] considered only one vehicle most relevant to the reference lane, while the others considered all vehicles. In this paper, we present novel ways of defining the two interactions. For the VLI, instead of considering only the reference lane, we also take into account surrounding lanes along with their relative importance, which is calculated based on the target vehicle’s motion history. The V2I is encoded through a GNN by considering only surrounding vehicles within a certain distance from the reference lane. Our approach is based on the fact that human drivers often pay attention to surrounding lanes and vehicles occupying the surrounding lanes when driving along the reference lane. Driving behaviors such as lane changes and overtaking are examples.

We show in Fig. 2 the overall architecture of our forecasting model. In the following sections, we describe the details of our model.

To generate more clear image samples, [20] proposed a method that combines VAE and GAN. Based on the observation that the discriminator network implicitly learns a rich similarity metric for images, the typical element-wise reconstruction metric (e.g., $L_{2}$-distance) in the ELBO objective is replaced with a feature-wise metric expressed in the discriminator. In this paper, we also propose training our forecasting model with a discriminator network simultaneously. However, we don’t replace the element-wise reconstruction metric with the feature-wise metric since the characteristic of trajectory data is quite different from that of images. We instead use the discriminator to regularize our forecasting model during the training so that the trajectories generated by our model well match the shape of the reference lane.

The proposed discriminator network is defined as follows:

$$s=D(\mathbf{Y}_{i},\mathbf{L}^{m})=\text{MLP}_{dis}([\tilde{\mathbf{Y}}_{i};\tilde{\mathbf{L}}^{m}])\in\mathbb{R}^{1}.$$

(15)

We explored different choices for the encoding of the inputs to the discriminator network and observed that the following approaches improve the prediction performance: 1) $\tilde{\mathbf{Y}}_{i}$ is the result of encoding $[\mathbf{Y}_{i};\Delta\mathbf{Y}_{i}]$ through an LSTM network where $\Delta\mathbf{Y}_{i}=\Delta\mathbf{p}_{i}^{(t+1:t+T)}$, $\Delta\mathbf{p}_{i}^{t}=\mathbf{p}_{i}^{t}-\mathbf{l}_{f}^{m}$, and $\mathbf{l}_{f}^{m}$ is the coordinate point of $\mathbf{L}^{m}$ closest to $\mathbf{p}_{i}^{t}$, 2) $\tilde{\mathbf{L}}^{m}$ is from the feature extraction module. We also observed that generating trajectories for the GAN objective ($\mathcal{L}_{GAN}$ defined in Eq. 18) from both the encoder and prior yields better prediction performance, which is consistent with the observations in [20]. However, not back-propagating the error signal from the GAN objective to the encoder and prior does not lead to the performance improvement, which is not consistent with the observations in [20].

The proposed model is trained by optimizing the following objective:

$$\mathcal{L}=\mathcal{L}_{ELBO}+\alpha\mathcal{L}_{BCE}+\kappa\mathcal{L}_{GAN}.$$

(16)

Here, $\mathcal{L}_{BCE}$ is the binary cross entropy loss for the mode selection network and is defined as follows:

$$\mathcal{L}_{BCE}=\texttt{BCE}(\mathbf{g}^{m},\texttt{softmax}(\mathbf{z}_{h})),$$

(17)

where $\mathbf{g}^{m}$ is the one-hot vector indicating the index of the lane, in which the target vehicle traveled in the future timesteps, among the $M$ candidate lanes. $\mathcal{L}_{GAN}$ is the typical adversarial loss defined as follows:

$$\mathcal{L}_{GAN}=\mathbb{E}_{\mathbf{Y}\sim p_{data}}[\mathtt{log}D(\mathbf{Y},\mathbf{L})]+\mathbb{E}_{\mathbf{z}\sim p_{z}}[\mathtt{log}(1-D(G(\mathbf{z}),\mathbf{L}))],$$

(18)

where $G$ denotes our forecasting model. The hyper-parameters ($\alpha$, $\kappa$) in Eq. 16 and $\beta$ in Eq. 3 are set to $1$, $0.01$, and $0.5$, respectively. More details can be found in supplementary materials.

Future trajectories for the target vehicle are generated from the modes based on their weights. Assume that $K$ trajectories need to be generated for $V_{i}$. $\lfloor K\times w_{m}\rfloor$ out of $K$ future trajectories are generated by the decoder network using $\mathbf{c}_{i}^{m}$ and $\mathbf{z}_{l}$. In the end, a total of $K$ trajectories can be generated from $\{\mathbf{c}_{i}^{m}\}_{m=1}^{M}$ since $\sum_{m=1}^{M}w_{m}=1$.

Two large-scale real-world datasets, Argoverse Forecasting [7] and nuScenes [4], are used to evaluate the prediction performance of our model. Both provide 2D or 3D annotations of road agents, track IDs of agents, and HD map data. nuScenes includes 1000 scenes, each 20 seconds in length. A 6-second future trajectory is predicted from a 2-second past trajectory for each target vehicle. Argoverse Forecasting is the dataset for the trajectory prediction task. It provides more than 300K scenarios, each 5 seconds in length. A 3-second future trajectory is predicted from a 2-second past trajectory for each target vehicle. Argoverse Forecasting and nuScenes publicly release only training and validation sets. Following the existing works [16, 31], we use the validation set for the test. For the training, we use the training set only.

For the quantitative evaluation of our forecasting model, we employ two popular metrics, average displacement error (ADE) and final displacement error (FDE), defined as follows:

$$ADE(\hat{\mathbf{Y}},\mathbf{Y})=\frac{1}{T}\sum_{t=1}^{T}||\hat{\mathbf{p}}^{t}-\mathbf{p}^{t}||_{2},$$

(19)

$$FDE(\hat{\mathbf{Y}},\mathbf{Y})=||\hat{\mathbf{p}}^{T}-\mathbf{p}^{T}||_{2},$$

(20)

where $\mathbf{Y}$ and $\hat{\mathbf{Y}}$ respectively denote the ground-truth trajectory and its prediction. In the rest of this paper, we denote $ADE_{K}$ and $FDE_{K}$ as the minimum of ADE and FDE among the $K$ generated trajectories, respectively. It is worth noting that $ADE_{1}$ and $FDE_{1}$ metrics shown in the tables presented in the later sections represent the average quality of the trajectories generated for $\mathbf{Y}$. Our derivation can be found in the supplementary materials. On the other hand, $ADE_{K}$ and $FDE_{K}$ represent the quality of the trajectory closest to the ground-truth among the $K$ generated trajectories. We will call $ADE_{K\geq 12}$ and $FDE_{K\geq 12}$ metrics in the tables the best quality in the rest of this paper. According to [5], the average quality and the best quality are complementary and evaluate the precision and coverage of the predicted trajectory distributions, respectively.