Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking

Extensive research has been conducted on trajectory prediction of humans and autonomous agents (e.g. on-road vehicles, mobile robots, etc). Early literature mainly introduced physics-based or rule-based approaches, such as state estimation techniques based on kinematic models. These methods do not consider mutual influence between intelligent agents and they are only able to perform well in short-term prediction tasks with limited model flexibility [13, 14]. As machine learning techniques are studied more extensively, people began to employ learning-based models for prediction purpose, such as hidden Markov models [15], Gaussian mixture models [16], dynamic Bayesian network [17], and inverse reinforcement learning [18]. In recent years, researchers have proposed various learning-based prediction models, which enables more flexibility and capacity to capture underlying interactive behavior patterns [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 4, 30, 31, 32, 33, 34, 35, 36, 37]. The prediction hypotheses can be directly obtained from the model outputs. However, with an end-to-end fashion, physical feasibility constraints are usually ignored in these methods, which may result in implausible forecast. In this paper, we address multi-agent interaction modeling and introduce a probabilistic prediction system based on a deep generative framework. Our approach also explicitly considers the feasibility constraints of vehicles.

Although many research have been conducted on end-to-end tracking through real-time detection in the computer vision community, here we only focus on the approaches based on recursive Bayesian state estimation, which consists of prior (prediction) update and posterior (measurement) update. One of the challenges in the tracking problem is noisy or missing observations (occlusions). Kinematics based models tend to be sensitive to measurement noise and lose tracking when the targets are occluded for a period. Moreover, they are not able to consider interactive behaviors between tracking targets since the target states are propagated independently. Some approaches to handle these issues were proposed in [11]. In their work, a learning-based behavior model is employed to improve the tracking accuracy. However, the behavior model requires specific assumptions on agent numbers, agent roles and scenarios, which significantly restricts the applicability. The prediction model proposed in this paper can serve as a strong alternative to the original behavior model, which brings more versatility and larger model capacity.

In general, the objective of relational reasoning is to reason about different entities and their relations from observed structured or unstructured data, such as image pixels [38], words or sentences [39], human skeletons [40] and interactive navigating agents [41, 42, 43]. Popular techniques for relational reasoning and interaction modeling in earlier literature include, but are not limited to, social pooling mechanism [5], convolutional pooling mechanism [7], soft attention mechanism [10], etc. Recently, graph networks have proved to be effective for relational reasoning on graph-structured data, where there is no restriction on the message passing rules. In traffic scenarios, a typical representation of the whole scene is to formulate a graph, where nodes are agents and edges are their relationships. Most existing works focused on the approximation function parameterized by deep neural network due to its high flexibility, which leads to graph neural networks (GNN). In this paper, we present a graph neural network with both topological and temporal attention mechanisms to capture underlying interaction patterns and jointly predict future behaviors.

Our approach is also related to deep generative models, which have been widely applied to representation learning and distribution approximation tasks [44, 45]. One of the advantages of generative modeling lies in the data distribution learning without supervision. Coupled with highly flexible deep networks, deep generative models have achieved satisfying performance in image generation, style transfer, sequence synthesis tasks, etc. Despite the variational auto-encoder, a highly flexible latent variable model with encoder-decoder architecture, tries its best to make the posterior of the latent variable and its prior (usually a normal distribution) as similar as possible, the two distributions do not match well in many tasks. Hence, it breaks the consistency of the model. Also, although generative adversarial networks have achieved satisfying performance on image generation tasks, it usually suffers from mode collapse problems, especially when applied to sequential data under the conditional setting. In order to mitigate these drawbacks, The Wasserstein auto-encoder (WAE) [46], proposed from the optimal transport point of view and combined with information theory, encourages the consistency between the encoded latent distribution and the prior distribution. A variant of variational auto-encoder was proposed in [47] and a modified approach was proposed in our previous work [12]. In this paper, we adopt the similar Wasserstein generative method in [12] as the basis of model training, and significantly extends pair-wise prediction to multi-agent prediction.

The graph neural network is a type of deep learning models which are directly applied to graph structures. It naturally incorporates relational inductive bias into the model design. In the context of graph neural network, most graphs are attributed (with node attributes and/or edge attributes and/or global attributes). Generally, there are three basic operations in graph representation learning with GNN: edge update, node update and global update [48]. Note that the global update is optional, which is applied only if the graphs have global attributes. More formally, denote the graph with $n$ nodes as $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}=\{v_{i},i\in\{1,...,n\}\}$ is a set of node attributes and $\mathcal{E}=\{e_{ij},i,j\in\{1,...,n\}\}$ is a set of edge attributes. Denote $u$ as the global attribute. Then, the three update operation can be written as

where $E^{\prime}_{i}=\{e^{\prime}_{ij},j\in N(i)\}$, $E^{\prime}=\bigcup E^{\prime}_{i}$, $V^{\prime}=\{v^{\prime}_{i},i=1,...,n\}$, and $N(i)$ is the neighbors of node $i$. $\phi^{e}(\cdot)$,$\phi^{v}(\cdot)$ and $\phi^{u}(\cdot)$ are neural networks. $\rho^{e\rightarrow v}(\cdot)$, $\rho^{e\rightarrow u}(\cdot)$ and $\rho^{v\rightarrow u}(\cdot)$ are aggregation functions with the property of permutation invariance.

In this work, it is natural to represent intelligent agents as nodes, and their relation as edges. We only apply edge update and node update, since there is no global attribute in our setup.

The Wasserstein distance is defined in a metric space $(\chi,\rho)$:

where $||f||_{\text{Lip}}$ is the Lipschitz constant of the function $f$. By the Kantorovich-Rubinstein duality, we can formulate Eq. (2) as an optimal transport problem, which is given by

where $\mathbf{M}(x,x^{\prime})$ is the coupling distribution of $x\sim P$, $x^{\prime}\sim Q$, which is a probability measure for $\chi\times\chi$.

Define

Following the WAE [46] with the assumption that $p_{G}(x|z)$ is deterministic, we have

where $X\sim P_{X}$, $Z\sim Q(Z|X)$ and $Q_{z}$ is the marginal distribution of $Z$. We relax the optimization problem as:

where $\epsilon_{1}$ and $\epsilon_{2}$ are pre-defined constants. Then, the dual optimal problem is formulated as

where $\alpha$ and $\beta$ are chosen from a proper range.

After some algebra derivations, we obtain the following equivalent optimization problem

where $I_{\phi}(x,z)$ is the mutual information between $x$ and $z$.

Please refer to our prior work [12] for more details on the derivation. The goal of both WAE and vanilla variational auto-encoder (VAE) is to learn the data distribution. According to the numerical experiment results in [12], the vanilla VAE tends to learn a distribution with smaller variance and have mode collapse issue; while WAE is better at capturing the true data distribution. Also, WAE is able to learn a better latent representation than VAE due to the regularization terms.

The detailed architecture of STG-DAT is shown in Fig. 2, where a standard encoder-decoder architecture is employed. There are three key components: a deep feature extractor, an encoder with spatio-temporal graph generation and dual-attention network, and a decoder with a kinematic constraint layer. First, the feature extractor takes in both history and future information and outputs state, relation, and context feature embeddings. The information contains the trajectories of the involved interactive agents, and a sequence of context density maps and mean velocity fields. The scene images or semantic maps can also be included, if they are available in the dataset. Since the neural networks have a the capability of extracting highly flexible features, we choose multi-layer perceptron (MLP) to generate state and relation embeddings and convolutional neural network (CNN) to generate context embedding. The extracted features are utilized to generate a spatio-temporal graph for both the history and the future, respectively. The node attributes are updated by a spatio-temporal graph attention mechanism. Then, the updated node attributes are transformed from the feature space into a latent space by an encoding function according to the derivation of the conditional generative modeling. Finally, the decoder based on the recurrent neural network generates feasible and human-like future trajectories for all the involved agents. The number of agents can be flexible in different cases due to the weight sharing and permutation invariance of the graph representation. All the components are implemented with deep neural networks, thus they can be trained end-to-end efficiently and consistently.

The feature extractor consists of three parts: State MLP, Relation MLP, and Context CNN. The operations below are applied at each time step, and a sequence of state, relation, and context feature embeddings can be obtained.

• •

  State MLP: It embeds the position, velocity, and heading information into a state feature vector for each agent. Different types of agents use distinct state embedding functions and the same type of agents share the same one. In this paper, we consider three types: vehicles, cyclists and pedestrians. The state embedding (SE) of agent $i$ at time $k$ is obtained by

  $$SE_{i}^{k}=\text{MLP}_{s}(\bm{\tau}_{i}^{k}).$$

  (11)

• •

  Relation MLP: It embeds the relative information between each pair of agents into a relation feature vector. We differentiate the edges with opposite directions between the same pair of nodes. The relative information can be either the distance and relative angle (in a 2D polar coordinate), or the differences between the positions of the two agents along two perpendicular axes (in a 2D Cartesian coordinate). We use the latter in this work, since it is simpler to compute and the performance is comparable to the former one. More specifically, consider a pair of agents $i$ and $j$. When calculating the relation embedding associated with edge $e_{ij}$ in a Cartesian coordinate, we set agent $i$ as the origin and its heading as the positive direction. The relative position, velocity and heading angle of agent $j$ with respect to agent $i$ can be calculated and denoted as $\bm{\phi}_{ij}^{k}$. The relation embedding (RE) is obtained by

  $$RE_{ij}^{k}=\text{MLP}_{r}(\bm{\phi}_{ij}^{k}).$$

  (12)

• •

  Context CNN: It extracts spatial features for each agent from a local occupancy density map ($H\times W\times 1$) as well as heuristic features from a local velocity field ($H\times W\times 2$) centered on the corresponding agent. The reason of using occupancy density maps instead of real scene images is to remove redundant information and efficiently represent data-driven drivable regions. This information provides a prior knowledge of common driving behaviors at specific areas of the scene. The context embedding (CE) of agent $i$ at time $k$ is obtained by

  $$CE_{i}^{k}=\text{CNN}(\bm{c}_{i}^{k}).$$

  (13)

After obtaining the extracted features, a history spatio-temporal graph (HG) and a future spatio-temporal graph (FG) are generated to represent the information related to the involved agents. Here, the state features and context features are concatenated to serve as the (agent) node attributes, whereas the relation features serve as edge attributes. The HG and FG contain different time steps, and they are processed in a similar fashion with the graph dual-attention network. In a specific case, the number of nodes (agents) in both HG and FG is assumed to be fixed, which implies the same agents appear in the whole horizon. The edges are eliminated at a certain time step if the Euclidean distance between two agents is larger than a threshold $d$. Therefore, the graph connectivity and topology at different time steps may vary.

The proposed graph dual-attention network consists of two consecutive layers: a topological attention layer which updates node attributes from the spatial or topological perspective, and a temporal attention layer which outputs a high-level feature embedding for each node. The temporal attention layer summaries both the topological and temporal information and figure out relative significance of the information at each time step. Following the notation in Section III, assume there are totally $n$ nodes (agents) in a graph, we denote a graph as $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}=\{v_{i}\in\mathbb{R}^{D_{n}},i\in\{1,...,n\}\}$ and $\mathcal{E}=\{e_{ij}\in\mathbb{R}^{D_{e}},i,j\in\{1,...,n\}\}$. $D_{n}$ and $D_{e}$ are the dimensions of node attributes and edge attributes.

We impose a kinematic constraint layer after the decoder gated recurrent unit (GRU) to enforce feasible trajectory prediction. The same types of agents share the same GRU unit, while different types use distinct ones. This is reasonable since the behavior patterns, speed ranges and traversable areas may be diverse among different types.

The bicycle model is a widely used nonlinear model to approximate the kinematics of vehicles, which is shown in Fig. 3. Here we adopt the discretized form, which is given by

where $(x,y)$ are the coordinates of the center of mass, $\psi$ is the inertial heading and $v$ is the speed of the vehicle. $\beta$ is the angle of the current velocity of the center of mass with respect to the longitudinal axis of the car. $l_{f}$ and $l_{r}$ denote the distance from the center of mass of the vehicle to the front and rear axles, respectively. The state of vehicles at time step $k$ is denoted as $\mathbf{s}(k)=[x(k),y(k),\phi(k),v(k),\beta(k)]^{\top}$, and the control input at time step $k$ as $\mathbf{u}(k)=[a(k),\dot{\beta}(k)]^{\top}$. Eq. (20) can be expressed as $\mathbf{s}(k+1)=\mathbf{f}(\mathbf{s}(k),\mathbf{u}(k))$.

The prediction system is expected to provide the position distribution of each agent at each time step. The distribution of the control input $\mathbf{u}(k)$ is assumed to be a multi-variate Gaussian distribution at each time step, which is parameterized by the output of the GRU cell. We provide an illustrative example for the agent $i$. The inputs of the gated recurrent unit (GRU) are the node attribute $\widetilde{v}_{i}$ at the first step and zero paddings for the following steps. The outputs are the raw $\mathbf{u}(k)$ at each step which are truncated by a saturation function in order to restrict the control actions in the feasible range. Then the kinematic cell takes in the current state and action, and outputs the state at the next step. This procedure is iterated until the prediction horizon is reached. If $l_{r}$ is not a prior knowledge or cannot be observed during testing, then we can approximate it with a constant based on the statistics of training data.

In order to propagate the uncertainty to future time steps, two options are available with tradeoffs. The details are introduced in the following.

We can approximate the distribution of position by a set of Monte Carlo samples, which is highly flexible without any restrictions on the nonlinear dynamic system. The particle samples are propagated by the nonlinear bicycle model. The prediction accuracy tends to be improved as the number of particles increases. However, the number of samples may need to be adjusted according to real-time requirements. If the explicit probability density function of the state variable is required, the kernel density estimation technique can be employed in a non-parametric way.

Since $\mathbf{u}(k)$ follows Gaussian distribution, we can obtain an analytic distribution of position by linearizing the bicycle model at the current state. It is easy to show that the position distribution is also a Gaussian distribution. This is simple to implement and computationally efficient, while the flexibility is very limited. This restriction leads to lower prediction accuracy in general, especially when the real distribution is multi-modal.

For a nonlinear system with Gaussian assumption, the expectation and covariance of the state variable can be propagated in a fashion similar to the prior (prediction) update of extended Kalman filter [51]. We can linearize the system around the current state $\mathbf{s}(k)$,

where $\mathbf{Df}_{s}(k)$ and $\mathbf{Df}_{u}(k)$ are the Jacobian matrices defined as below,

where $\gamma(k)=\phi(k)+\beta(k)$, $v_{x}(k)=v(k)\cos\gamma(k)$, and $v_{y}(k)=v(k)\sin\gamma(k)$. Then we have the distribution of $\mathbf{s}(k+1)$, which is expressed as

where

with the initial condition $\mathbf{s}(0)\sim\mathcal{N}(\bm{\mu}_{s}(0),\bm{\Sigma}_{ss}(0))$.

In this part, we demonstrate the loss function of our model, which is based on the optimization formulation of Wasserstein generative modeling aforementioned in Section III. In order to keep consistent with Section III, we use the same notations: $x$ denotes the predicted trajectories, $z$ denotes the latent variable, and $y$ denotes the condition variable.

The optimization problem can be formulated in the same way as in [12], which is written as

The detailed derivation can be found in [12]. Since the whole system is fully differentiable, we can train the network in an end-to-end fashion by the Adam optimizer [52]. The loss function is given by

where

FE is the deep feature extractor and GDAT is the proposed graph dual-attention network. $\gamma$ is a weight parameter to adjust the relative importance of the reconstruction loss, $N_{b}$ is the total number of training agents, $D_{\text{KL}}$ is Kullback-Leibler divergence, and MMD is maximum mean discrepancy. If $\gamma\gg\alpha,\beta$, then the loss function degenerates to the mean squared error loss. The whole model is trained in an end-to-end fashion.

Here we briefly introduce the datasets below. Please refer to the appendix for data processing details.

$\bullet$ ETH [53] and UCY [54]: These two datasets are usually used together in literature, which include bird-eye-view videos and annotations of pedestrians in both indoor and outdoor scenarios. The trajectories were extracted in the world space (unit: meters).

$\bullet$ Stanford Drone Dataset (SDD)[55]: The dataset also contains a set of bird-eye-view images and the corresponding trajectories of involved entities. It was collected in multiple scenarios in a university campus full of interactive pedestrians, cyclists and vehicles. The trajectories were extracted in the image pixel space.

$\bullet$ INTERACTION Dataset (ID)[2]: The dataset contains naturalistic motions of various traffic participants in a variety of highly interactive driving scenarios. Trajectory data was collected using drones and traffic cameras. The high-definition maps of scenarios and agents’ trajectories are provided. We consider three types of scenarios: roundabout (RA), unsignalized intersection (UI) and highway ramp (HR). The trajectories were extracted in the world space (unit: meters).

We evaluate the model performance in terms of average displacement error (ADE) and final displacement error (FDE), which are exactly the same as [5, 6, 56]. ADE is defined as the average distance between the predicted trajectories and the groundtruth over all the involved entities within the prediction horizon. FDE is defined as the deviated distance at the last predicted time step. For the ETH, UCY, and SDD dataset, we predicted the future 12 time steps (4.8s) based on the historical 8 time steps (3.2s). For the ID dataset, we predicted the future 10 time steps (5.0s) based on the historical 4 time steps (2.0s).

• •

  Probabilistic LSTM (P-LSTM) [57]: The backbone of the model is the same as an encoder-decoder architecture with vanilla LSTM. In order to incorporate uncertainty in the model, a noise term sampled from the normal distribution is added in the input, which results in a probabilistic model.

• •

  Social LSTM (S-LSTM) [5]: The trajectories are encoded with an LSTM layer, whose hidden states serve as the input of the proposed social pooling layer, which handles interaction modeling implicitly.

• •

  Social GAN (S-GAN) [6]: The model introduces a generative adversarial learning scheme into S-LSTM to further improve performance.

• •

  Social Attention (S-ATT) [10]: The model is based on the architecture of Structural-RNN [58], which deals with spatio-temporal graphs with recurrent neural networks.

• •

  DESIRE [20]: The model is a deep stochastic inverse optimal control framework based on conditional variational auto-encoder with RNN encoders and decoders. A ranking module for sampled trajectories was introduced to indicate their likelihood.

• •

  Social-BiGAT [56]: The model is a graph-based generative adversarial network, which is based on a graph attention network. A recurrent encoder-decoder architecture is trained via an adversarial scheme.

• •

  Trajectron [29]: The model combines tools from recurrent sequence modeling and variational deep generative modeling to produce a distribution of future trajectories.

A batch size of 64 was used and the models were trained for 100 epochs with early stopping using the Adam optimizer with an initial learning rate of 0.001. The models were trained on a single NVIDIA TITAN X GPU. We used a split of 70%, 10%, 20% as training, validation and testing data, respectively. During the testing phase, the data processing time for a single time step is around 8ms in average. According to our setting, we predict the future 10 time steps which needs around 80ms (equivalent to 12.5Hz).

The details of our model architecture are introduced in the following.

• •

  Deep Feature Extractor (FE): The State MLP and Relation MLP both have three hidden layers with 128 hidden units. The Context CNN adopts the same backbone structure of ResNet18 [59], which is trained from scratch.

• •

  Graph Dual-Attention Network (GDAT): The dimensions of node attributes and edge attributes are 64 and 16, respectively. These dimensions are fixed in different rounds of message passing. The activation functions in the attention mechanism are LeakyReLU.

• •

  Encoding Function: The encoding function is a three-layer MLP with 128 hidden units. The dimension of latent variable is 32.

• •

  Decoding Function: The decoding function is a GRU recurrent layer with 128 hidden units.

$\bullet$ ETH and UCY Datasets: The comparison of the proposed STG-DAT and baseline methods in terms of ADE and FDE is shown in Table I. Some of the reported statistics are adopted from the original papers. All the baseline methods deal with interaction modeling in a specific way. The S-LSTM employs a social pooling mechanism to model the interactions between entities. The S-GAN improve the performance by introducing deep generative modeling. The CGNS combines conditional latent space learning and variational divergence minimization to further enhance the generation capability. The Trajectron adopts a graph-structured model with sequence modeling. Both Social-BiGAT and our method leverage the trajectory and context information, but in different ways. Our model can achieve better performance owing to the explicit interaction modeling with graph neural networks and more compact distribution learning with conditional Wasserstein generative modeling. In general, our approach achieves the smallest ADE and FDE across different scenes. The average ADE / FDE are reduced by 34.8% / 37.2% compared to the best baseline (Trajectron).

$\bullet$ Stanford Drone Dataset: The comparison of results is provided in Table II, where the ADE and FDE are reported in the pixel distance. Note that we also included cyclists and vehicles in the dataset besides pedestrians. In general, the relative performance of baseline methods is consistent with the observation on the ETH / UCY datasets. Our approach achieves the best performance in terms of prediction error, which implies the superiority of explicit interaction modeling and necessity of leveraging both trajectory and context information. In order to show the effectiveness of distinct node embedding functions for different types, we provide an ablative result by treating all the agents as the same type, which leads to an increase on the prediction error. For the full model, the ADE / FDE are reduced by 15.1% / 22.4% with respect to the best baseline (CGNS).

$\bullet$ INTERACTION Dataset: We finally compare the model performance on the real-world driving dataset in Table III. For fair comparison, we only involved the baseline approaches whose codes are publicly available and can be adapted to the same setup as our approach. Although we trained a unified prediction model on different scenarios simultaneously, we analyzed the results for each type of scenario separately. In the HR scenarios, the results of baseline methods are comparable while our model achieves the best performance. The behavior patterns of vehicles in HR scenarios are relatively easy to forecast, since most vehicles are doing car following without highly interactive behaviors. In the RA and UI scenarios, however, the superiority of the proposed system is more distinguishable due to frequent interactions. The P-LSTM performs the worst since it predicts future trajectories for each agent individually without considering their relations. Although all the other baseline approaches incorporate interaction modeling by different strategies which further reduce the prediction error, our model still performs the best. This implies the advantages of graph dual-attention network for interaction modeling, as well as kinematic layer for feasibility constraints. By using our full model $\mathbf{T}+\mathbf{C}+\mathbf{K}$, the 5.0s ADE / FDE are reduced by 28.4% / 23.5%, 33.7% / 31.5% and 17.3% / 17.5% in RA, UI and HR with respect to the best baseline (Trajectron), respectively.

We also tested the tracking performance using our approach and baseline methods, which is shown in Table IV. The constant velocity model (CVM) and constant acceleration model (CAM) are widely used linear vehicle kinematics models with a constant velocity / acceleration assumption. These models are employed frequently in multi-target tracking literature. A Gaussian noise term is injected at each time step to model uncertainty. It shows that tracking with learning-based models performs consistently better than CVM and CAM due to the ability of interaction-aware prediction. Our approach STG-DAT achieves a significantly higher accuracy.

We qualitatively evaluated on prediction hypotheses of typical testing cases on the SDD dataset and ID dataset in Fig. 5 and Fig. 6, respectively. Although we jointly predict all the agents in a scene, we show predictions for a subset for clearness. It shows that our approach can handle different challenging scenarios (e.g. intersection, roundabout) and diverse behaviors (e.g. going straight, turning, waiting, stopping) of vehicles and pedestrians. Generally, the groundtruth trajectories are close to the mean of predicted distribution and the model also allows for uncertainty.

We also conducted comprehensive ablative analysis on the ID dataset to demonstrate relative significance of context information, dual-attention mechanism and the kinematic constraint layer for vehicle trajectory prediction. The descriptions of compared model settings are provided below:

• •

  $\mathbf{T}$: This is the model without the kinematic layer, which only uses trajectory information.

• •

  $\mathbf{T}+\mathbf{C}-\mathbf{ATT}$: This is the model without the dual-attention mechanism or the kinematic constraint layer. We used equal attention in this model setting instead.

• •

  $\mathbf{T}+\mathbf{C}$: This is the model without kinematic constraint layer.

• •

  $\mathbf{T}+\mathbf{C}+\mathbf{K}$: This is the whole proposed model including all the components.

The ADE / FDE of each model setting are shown in the lower part of Table III.

$\bullet$ $\mathbf{T}$ versus $\mathbf{T}+\mathbf{C}$: We show the effectiveness of employing scene context information. $\mathbf{T}$ is the model without the kinematic layer, which only uses trajectory information, while $\mathbf{T}+\mathbf{C}$ further employs context information. The models directly output the position displacements $(\Delta x^{k},\Delta y^{k})$ at each step, which are aggregated to get complete trajectories. We can see little difference on prediction errors over short horizons, while the gap becomes larger as the horizon extends. The reason is that the vehicle trajectories within a short period can usually be well approximated by a constant velocity model, which are not heavily restricted or affected by the static environmental context. However, as the forecasting horizon increases, the effects of context constraints cannot be ignored anymore, which leads to larger performance gain of leveraging context information. Compared with $\mathbf{T}$, the 5.0s ADE / FDE of $\mathbf{T}+\mathbf{C}$ are reduced by 5.6% / 6.4%, 9.3% / 7.7% and 2.1% / 2.4% in RA, UI and HR scenarios, respectively. This implies that the context information has larger effects on the prediction in RA and UI scenarios, where the influence of road geometries cannot be ignored. The context information does little help to HR scenarios, since most vehicles go straight on highways. In Fig. 6, the predicted distribution of $\mathbf{T}+\mathbf{C}$ is more compliant to roadways to avoid collisions and the vehicles near the “yield” or “stop” signs tend to yield or stop. However, $\mathbf{T}$ generates samples which are outside of feasible areas or violating traffic rules.

$\bullet$ $\mathbf{T}+\mathbf{C}-\mathbf{ATT}$ versus $\mathbf{T}+\mathbf{C}$: We show the effectiveness of the proposed dual-attention mechanism. $\mathbf{T}+\mathbf{C}-\mathbf{ATT}$ uses equal attention coefficients in both topological and temporal layers. According to the statistics reported in Table III, compared with equal attention, employing the dual-attention mechanism to figure out relative importance within the topological structure and along different time steps can reduce the 5.0s ADE / FDE by 21.3% / 25.0%, 27.9% / 21.5% and 7.8% / 8.0% in RA, UI and HR scenarios, respectively. The improvement in RA and UI are more significant due to frequent interactions.

$\bullet$ $\mathbf{T}+\mathbf{C}$ versus $\mathbf{T}+\mathbf{C}+\mathbf{K}$: We show the effectiveness of the kinematic constraint layer. Different from $\mathbf{T}$ which directly output position displacement, the outputs of the GRU unit in $\mathbf{T}+\mathbf{C}+\mathbf{K}$ are control actions, which are aggregated by the bicycle model to obtain complete trajectories. According to Table III, employing the kinematic constraint layer to regularize the learning-based prediction hypotheses can further reduce the 5.0s ADE / FDE by 20.0% / 13.7%, 12.5% / 16.0% and 9.5% / 9.3% in RA, UI and HR scenarios, respectively. Due to the restriction from the kinematic model, unfeasible movements can be filtered out and the model is unlikely to overfit noisy data or outliers. Moreover, the improvement in RA and UI is more significant than in HR. The reason is that most vehicles go straight along the road in HR, whose behaviors can be well approximated by linear models. However, there are frequent turning behaviors in RA and UI which need constraints by more sophisticated models. We also visualize the predicted trajectories in Fig. 6, where the ones in $\mathbf{T}+\mathbf{C}+\mathbf{K}$ are smoother and more plausible.