Aware of the History: Trajectory Forecasting with the Local Behavior Data

Historical behaviors are very helpful to trajectory forecasting since they reveal the motion pattern of agents. Several previous works have made significant progress in this direction by adopting memory-based designs. MANTRA [28] uses the observed past trajectories and the map as the key to query the possible hidden features of future trajectories. Similarly, Zhao et al. [45] build an expert repository to retrieve the possible destinations by matching the current trajectory with the ones in the repository. MemoNet [40] also considers memorizing the destinations of each agent and first applies the memory mechanism to multi-agent trajectory forecasting. Compared to those memory-based methods, our work: i) regards the historical behaviors as system inputs to benefit the task from the perspective of enriching scene information; ii) directly uses geometric coordinates to query related historical information which emphasizes the data locality and is more interpretable and robust.

To use a new type of scene information in systems requires fusing it with existing scene information sources. After reviewing how scene information is encoded in previous methods, we see two main types of scene representations: 1) rasterized-image-based representations [5, 20], 2) graph-based representations [11, 12, 23, 39]. Rasterized-image-based representations render static HD maps and motion data into a birds’ eye view (BEV) image, using various colors to represent different objects. Systems with this scene representation tend to use standard convolutional neural networks (CNNs) backbones to extract scene features [5, 9, 13, 20, 31]. These methods transform scenes into image coordinates systems.

Graph-based scene representations become popular with the recent development of graph learning [34] and attention mechanism [37]. These methods build a graph that can be either directional or non-directional, and use techniques such as graph neural network (GNN) [34] or attention-based operations [37] to enable the interaction among map objects and agents. The nodes are the semantic map objects, e.g., lane segments and agents. The edges are defined by heuristics, which can be the spatial distances between the two nodes or the semantic labels of the two nodes. Systems using graph-based scene representations [11, 12, 15, 23, 44] independently encode each map object to make the graph more informative. Graph-based scene representations have an outstanding information fusion capability and are substantially explored recently.

Other methods that do not strictly fall into the above two categories can share some properties with one or both of them. For example, TPCN [41] uses point cloud representations for the scene and does not manually specify the interaction structures. Yet, by using PointNet++ [32] to aggregate each point’s neighborhood information, TPCN still technically defines local complete subgraphs for the scene where each point is a node connected with its neighbors.

To show the generalizability of the proposed frameworks, in this work we introduce an implementation methodology for upgrading forecasting systems that use either rasterized-image-based or graph-based scene representations to the LBA and LBF frameworks in Sec 4.2 and 5.2 respectively.

Our LBF framework is inspired by the recent study of knowledge distillation (KD). Knowledge distillation is a technique that compresses a larger teacher network to a smaller student network by urging the student to mimic the teacher at the intermediate feature or output level [19]. KD is widely used for various tasks, including object detection [7, 16], semantic segmentation [18, 25] and tracking [26]. Its usage is still being explored. For example, researchers start to use KD for collaborative perception tasks [22]. In our work, the proposed LBF framework uses KD in the trajectory forecasting task. Compared to previous works that pay more attention to model compression, we seek to compress the volume of input data. We use the offline framework as the teacher network, which takes each agent’s local behavior data as the input along with HD maps and the agents’ observed trajectories, while the student network (i.e., the online framework) only uses the later two data modalities as the input without requiring local behavior data. Our experiments show that KD-based information compression significantly boosts the performance of trajectory forecasting.

Like a typical prediction framework pipeline adopted by previous works [11, 20, 31] (see Figure 2 (a)), our LBA framework follows an encoder-decoder structure (see Figure 2 (b)). The encoder $\Theta$ extracts features from multiple scene information sources. Then, the subsequent decoder $\Phi$ generates the predicted future trajectories $\widehat{\mathcal{Y}}$ based on the scene features.

The main difference between LBA and previous frameworks is the design of the encoder $\Theta$. Our scene encoder involves two steps: 1) scene modeling, which assembles data from all three sources to generate a comprehensive representation of the current scene, and 2) scene feature extraction, which extracts the high-dimensional features from the generated scene representation. The way to represent a scene would largely determine the inner structure of a scene encoder.

In the current literature, the two most frequently used scene representations are graph-based and rasterized-image-based representations. To show that most of the existing forecasting systems can be upgraded to fit the proposed LBA prediction framework, we will describe the implementation strategies for the graph-based and rasterized-image-based systems.

The LBF framework includes a teacher network and a student network. See Fig. 2 (b), (c). The teacher network follows the LBA framework. For the student network, we remove the input stream of local behavior data in the LBA framework and add a behavior estimator $\Omega_{\mathcal{B}}$ to the pipeline. $\Omega_{\mathcal{B}}$ takes the intermediate scene features $f^{\prime}_{\mathcal{S}}$ from the scene encoder $\Theta$ as the input, and outputs the estimated local behavior features $\widehat{f}_{\mathcal{B}}$. Note that $f^{\prime}_{\mathcal{S}}$ is not involved with local behavioral information. Next, we let $\widehat{f}_{\mathcal{B}}$ join the scene encoder along with $f^{\prime}_{\mathcal{S}}$ for the final feature generation. The scene encoder outputs the updated $f_{\mathcal{S}}$, which contains the estimated local behavioral information, to the decoder $\Phi$. The decoder then processes $f_{\mathcal{S}}$ and generates the predicted future trajectories $\widehat{\mathcal{Y}}$. The core step of the LBF student network is to link the behavior estimator and the scene encoder.

Like Sec 4.2, this section introduces the implementation of the student networks in the LBF prediction framework based on two scene representations.

To train such a teacher-student framework, we pre-train an LBA prediction network as the teacher network, and then use the intermediate features from the teacher network as the supervision signals to guide the training of the student network. We consider features from the teacher network that are leveraged to guide the student network training as $\mathbf{F}_{t}$ and the corresponding features from the student network as $\mathbf{F}_{s}$. Note that $\mathbf{F}_{s}$ may include but are not limited to the reconstructed local behavior features and the the scene encoder’s outputs .

The training loss thereafter contains the trajectory forecasting loss $\mathcal{L}_{pred}$, which is defined by the original system and identical with the training loss of the teacher network as well as a KD loss. In this work, we use $\ell_{2}$ loss as the KD loss and set $\lambda_{kd}$ as the adjustable weight of the KD loss. The overall loss function aggregates the loss over all the samples; that is,

Datasets. We consider two widely used trajectory forecasting benchmarks, nuScenes [3] and Argoverse [6]. nuScenes collected 1000 scenes. Each scene is 20s long with a sampling rate of 2 Hz. The instances for trajectory forecasting tasks were split into train, validation and test splits, which respectively entail 32186, 8560 and 9041 instances. Each agent in the instances had 2s’ observed trajectories and the ground truth of 6s’ future trajectories. Argoverse collected over 30K scenarios. Each scenario is sampled at 10 Hz. The train/ val/ test splits had 205942/ 39472/ 78143 sequences respectively. Each agent in the scene had 2s’ observed trajectories and the ground-truth of 3s’ future trajectories.

Database Construction. To evaluate local-behavior-aware (LBA) framework without ground-truth data leakage, we only use the 2s’ observed trajectories of all agents in each data split to build the behavior database $\mathcal{D_{B}}$ for each corresponding phase (e.g. testing phase only uses the test split). For the local-behavior-free (LBF) framework, we only use the observed trajectories from the training split during the training phase. Please see appendix for detailed information of behavior database construction.

Metrics. We adopt three widely used metrics for quantitative evaluation: minimum average displacement error among $K$ predictions (minADE_K), minimum final displacement error among $K$ predictions (minFDE_K) and the missing rate (MR). minADE evaluates the minimum average prediction error of all timestamps among the predicted $K$ trajectories; minFDE is the minimum error of the final position among the $K$ predictions; MR_K is the ratio of agents whose minFDE_K is larger than 2.0 meters.

We pick three SOTA trajectory forecasting methods: LaneGCN [23], DenseTNT [15] and P2T [9], and adapt them to our behavior-aware framework. We use their official code packages as the implementation start code. The baseline performances in Table 1 and 2 are reproduced by ourselves for variables controlling.

LaneGCN and DenseTNT are graph-based methods. We use stacked linear residual blocks mentioned in [23] as our behavior encoder $E_{\mathcal{B}}$ for both methods. In the LBF prediction, we use attention based architecture to implement the behavior estimator $\Omega_{\mathcal{B}}$ as well as the auxiliary fusion module $\Psi$.

P2T is a rasterization-based method. We use behavioral probability map to represent the local behavior data, and ResNet [17] as the encoder backbone to extract features from the behavioral probability map. For the LBF prediction, we use three 1D Convolutional Layers to implement the behavior estimator $\Omega_{\mathcal{B}}$.

To train the network, we adopt the hyper-parameter configuration from each method’s official instructions. More implementation details are in the appendix.

We evaluate LaneGCN and P2T on nuScenes (see the quantitative results in Table 1). For Argoverse, we evaluate LaneGCN and DenseTNT (see Table 2). We also show the qualitative evaluation in Fig. 7.

In all experiments, when upgraded to either the LBA or LBF framework, the baseline methods significantly improve in performance. Unconventionally, our proposed frameworks bring consistent improvements to various models (P2T, LaneGCN, DenseTNT) across all metrics. This is a substantive progress compared to previous methods, because SOTA methods [12, 14, 41, 43] usually only show improvements in one or some metrics but not all metrics. Furthermore, on both datasets, the gains brought by local behavior data are consistently larger on $K=1$ metrics than on other metrics. This may result from the raise of average performance of the worst prediction among K predictions, as local behavior data efficiently narrows down the solution search space (explained in Sec 1).

Interestingly, the prediction performance of the LBF framework occasionally surpasses that of the LBA framework, even though the LBF framework lacks local behavior data input. Table 2 shows an example of this case, with the LaneGCN results on Argoverse. Our educated conjecture is that sometimes the LBF framework enjoys more data representativeness thanks to the reconstructed behavioral features; whereas, in the meantime, the LBA framework may be using pre-gathered local behavior data of a small size in its testing phase.

Besides the comparison with the baselines, we also show the comparison with the other published works on the benchmarks; see Tables 4 and 4.

We conduct the ablation study on the hyper-parameters regarding local behavior data generation and our prediction frameworks.

Local Behavior Data Size. Regarding the relationship between prediction performance and the local behavior data size of each agent, Table 6.3 shows: when there is few local behavior data, prediction is not as accurate as the baseline; but, even a small size of local behavior data can improve the prediction.

Local Range $\epsilon$. The adjustable parameter $\epsilon$ defines the local radius (see Sec 3). See Table 7 for the impact of $\epsilon$ on LaneGCN performance with Argoverse test split. When $\epsilon$ increases, the performance of the LBA framework first go up but later go down. It matches our intuition that a large $\epsilon$ will introduce distractions. The LBF framework, however, shows less confusion brought by the pre-trained LBA network, demonstrating stronger robustness to $\epsilon$.

Knowledge Distillation Parameters. We study the impact of knowledge-distillation-related parameters, i.e., the loss weight $\lambda_{kd}$ and the number of times that intermediate features are involved in the KD loss (denoted as KD times). See the results in Table 7. Detailed information about the choices of intermediate features is in the appendix. We see that firstly, the knowledge distillation structure does help the framework infer the impact of local behavior data. The framework, across all parameter settings in Table 7, outperforms the baseline method (in Table 2, row 1) and the model without KD loss (in Table 7, row 1). Secondly, the comparatively similar results across all the settings show that the LBF framework is relatively robust to the KD hyper-parameters.