EDA: Evolving and Distinct Anchors for Multimodal Motion Prediction

In recent times, there has been a significant increase in the study of motion prediction owing to the rising interest in autonomous driving. Motion prediction involves using the past agent states and road maps to forecast the future paths of traffic participants. Early studies (Chai et al. 2019; Casas et al. 2020; Park et al. 2020; Gilles et al. 2021; Casas, Sadat, and Urtasun 2021) commonly rasterize the inputs into images and capture the contextual information through CNNs. LaneGCN (Liang et al. 2020) and LaneRCNN (Zeng et al. 2021) construct lane graphs to efficiently represent the topology of road maps. Recent works (Gu, Sun, and Zhao 2021; Varadarajan et al. 2022; Shi et al. 2022a) have widely adopted the VectorNet (Gao et al. 2020) representation scheme, which regards the road maps as polylines. As Transformers (Vaswani et al. 2017) have gained popularity, an increasing number of studies (Liu et al. 2021; Ngiam et al. 2021; Jia et al. 2023) have utilized the attention mechanism to encode scene context. Encouraged by the successful application of DETR (Carion et al. 2020), many Transformer-based models (Girgis et al. 2021; Varadarajan et al. 2022; Nayakanti et al. 2023) have adopted learnable queries in decoder to generate multiple potential future trajectories. In our study, we utilize the architecture presented in MTR (Shi et al. 2022a), which is an advanced transformer framework incorporating a local attention based encoder and a decoder with intention queries.

Previous studies have investigated different approaches for modeling multimodal future behaviors. Earlier generative models (Lee et al. 2017; Gupta et al. 2018; Rhinehart, Kitani, and Vernaza 2018; Rhinehart et al. 2019) generate a collection of samples to represent the distribution of future. Many other works (Chai et al. 2019; Mercat et al. 2020; Ngiam et al. 2021) have utilized mixture models to parameterize multi-modal predictions, which mainly fall into two lines: prediction-based and anchor-based matching, as elaborated in introduction. In prediction-based matching methods (Ngiam et al. 2021; Varadarajan et al. 2022; Nayakanti et al. 2023), the positive mixture component is chosen by directly comparing predicted trajectories to the ground truth. Some models (Tang and Salakhutdinov 2019; Girgis et al. 2021) using the loss based on EM algorithm can also be viewed as prediction-based matching when its KL term converges. Due to the challenge of selecting representative future trajectories, these methods have opted to use well-designed aggregation techniques (Varadarajan et al. 2022; Nayakanti et al. 2023), or to directly utilize an end-to-end version (Ngiam et al. 2021; Girgis et al. 2021). However, their scoring performance still lags behind that of anchor-based matching methods. The anchor-based matching (Chai et al. 2019; Zhao et al. 2021) regards as positive the component matching the closest predefined anchor to ground truth. The HOME series (Gilles et al. 2021, 2022) and DenseTNT (Gu, Sun, and Zhao 2021) can be considered as variations of anchor-based matching, where the anchors are the grids in heatmaps or target candidates placed on roads, but they require an additional sampling process to obtain the final predictions. The MTR (Shi et al. 2022a) achieves remarkable scoring performance using predefined anchors, while its end-to-end prediction-based matching version demonstrates significantly better performance in terms of minimum error and miss rate. Motivated by the findings, we propose a novel matching paradigm to exploit the regression potential hidden by the state-of-the-art anchor-based matching strategy.

According to Zhang et al., considering one-to-one label assignment in object detection, sparse queries cannot ensure a high recall, while dense queries inevitably bring more similar queries and face optimization challenges in classification. Therefore, they propose Dense Distinct Queries (DDQ), in which dense queries are first laid and then distinct queries are selected for one-to-one assignments. Inspired by DDQ (Zhang et al. 2023), we adopt distinct anchors to mitigate the ambiguity in trajectory scoring induced by prediction clustering phenomena.

We have implemented our ideas on a cutting-edge encoder-decoder structure, as the one presented in MTR (Shi et al. 2022a). This transformer framework employs an encoder with local self-attention for scene context modeling, in addition to a multi-layer decoder that incorporates learnable intention queries to predict multimodal trajectories.

It is important to note that our approach presented in this paper is centered on the design of loss. Consequently, the proposed Evolving and Distinct Anchors (EDA) can be readily applied to any network structure that includes a multi-layer decoder.

Although the spatial priors significantly alleviate the challenge in classification optimization, the vanilla anchor-based matching encounters a limitation in its regression capability, which will be demonstrated later. Regarding the above issue and encouraged by the successful adoption of multi-layer decoders in motion prediction (2021; 2022a), we naturally consider enabling anchors to evolve through multiple decoder layers for an enlarged regression capacity.

Take a 6-layer decoder for instance, as illustrated in Fig. 3(a), we can implement twice-evolving anchors by updating the anchors with outputs from the \nth2 and \nth4 layers, in which the evolving anchors for the $n$-th layer are:

where we have omitted the index subscripts for simplicity.

In a word, the evolving anchors are initially predefined and later adjusted by the intermediate outputs from decoder layers, which means the anchors are allowed to redistribute themselves under specific scenes.

Although predicting trajectories that cluster around the most probable regions contributes to better coverage of future behaviors with high uncertainty in prediction-based matching, this preference also introduces a serious issue of ambiguity in the scoring task. With multiple gathering outcomes, it becomes difficult for the model to distinguish the actual one closest to the ground truth. Hence the model tends to output similar scores for such predictions, making it hard to pick representative trajectories for downstream tasks.

In our proposed evolving anchors, as stated in the above analysis on effects of evolving anchors, the more frequently we update anchors, the greater the opportunity for substantial adjustments to unreal components. However, this also increases the potential for the phenomenon of prediction clustering. Such patterns can be observed intuitively in Fig. 5. As a result, this issue continues to pose a challenge for optimization in classification, particularly when updating the anchors multiple times.

Taking inspiration from DDQ (Zhang et al. 2023) in the object detection domain, we attempt to adopt distinct anchors to improve scoring performance. Specifically, we apply non-maximum suppression (NMS) to the anchors for each decoder layer prior to matching them with the ground truth during training, as illustrated in Fig. 3(b). Mixture components that correspond to the excluded anchors will neither serve as positive nor negative samples. Through the aforementioned operations:

• •

  We prevent the labeling of similar anchors as opposite, which significantly reduces the optimization difficulty for the classification task.

• •

  Moreover, the model is encouraged to prioritize the most probable trajectory among the similar ones, making it easier to select the representative predictions using simple post-processing techniques such as NMS.

We train the model with a combination of winner-takes-all regression loss and classification term, which is commonly used in mixture-model based methods (Chai et al. 2019; Nayakanti et al. 2023). Same as MTR (Shi et al. 2022a), we employ a Gaussian regression loss. Instead of Cross Entropy (CE) in MTR, we use Binary Cross Entropy (BCE) for classification loss, which is suitable for arbitrary numbers of mixture components filtered by distinct anchors. Please refer to the Appendix for more implementation details.

We first investigate the impacts of Evolving Anchors, and then assess the effectiveness of Distinct Anchors. All models are evaluated on the validation set of WOMD (Ettinger et al. 2021). In terms of mAP metrics, the results based on the original, scaled, and ranking-oriented top 6 scores are all presented, as referred in implementation details.

We evaluate the model that performs the best in our ablation study, namely twice-evolving and distinct anchors with ranking-oriented top 6 scores, on the test set of WOMD (Ettinger et al. 2021). We need to point out that the model for testing is trained solely on the WOMD training set without any ensemble techniques applied, consistent with our baseline MTR (Shi et al. 2022a).

As shown in Table 2, our single model outperforms previous ensemble-free approaches on the WOMD. The proposed EDA has demonstrated significant improvements in all performance metrics compared to the baseline MTR on both the validation and test sets. Specifically, there is a relative improvement of 13.5% on Miss Rate, 5.5% on minADE, and 4.1% on minFDE, as well as a +2.94% absolute growth in SoftmAP on the test set. Furthermroe, the performance of our EDA surpasses that of MTR++ (Shi et al. 2023), the latest improved version of MTR, on both the validation and test sets of WOMD. It is worth noting that MTR++ primarily enhances the network structure of MTR, while our approach centers on the design of loss, which means that combining the two complementary refinements has the potential to yield even more remarkable performance. Please refer to the Appendix for more experimental results.

We have implemented our methods based on the Motion TRansformer (Shi et al. 2022a). To predict the motion of a target agent, we utilize the agent-centric strategy in which all inputs are normalized to the coordinate system centered around the agent in question. The MTR adopts the vectorized representation (Gao et al. 2020) to arrange the input agent states and road maps as polylines. The encoder extracts the scene context information including agent and map features with local self-attention on the input polylines. A dense future prediction generates the future states for all surrounding agents to boost their features. Taking the context features as keys and values, a multi-layer decoder incorporates learnable intention queries to produce multimodal trajectories. The learnable queries are initialized with intention endpoints generated by a k-means clustering algorithm on the training set. The endpoints also serve as the predefined anchors for the vanilla anchor-based matching and our EDA, as shown in Fig. 7. In cross attention for each decoder layer, a dynamic map collection is applied to collect the closest map features to the latest predicted trajectories for querying, and the query position embedding is continuously updated using the layer outputs. We adopt the default parameters of MTR in our experiments, which are shown in Table 3.

Given the context $X$ and the ground truth $Y$, the aim of optimization is to maximum the log-likelihood:

where $\theta$ denotes the learnable weights of neural network, and $Z$ is the discrete random variable representing the probability of each component in mixture models. If we take:

where $z^{*}$ corresponds to the positive component whose anchor or prediction is closest to the ground truth, namely we utilize a winner-takes-all strategy.

Correspondingly, the term $E_{Z\sim q(Z)}[log\ p_{\theta}(Y|Z,X)]$ is the regression loss between the prediction of positive component $\hat{Y}^{*}$ and ground truth $Y$, where we model the probability using a Gaussian distribution:

Since $H[q]$ is irrelevant to the learnable weights $\theta$, the other term $-KL[q(Z)\|p_{\theta}(Z|X)]$ is equivalent to the Cross Entropy loss on predicted scores $\hat{Z}$:

To accommodate varying numbers of the mixture components selected for distinct anchors, we employ Binary Cross Entropy for classification, which our experiments have shown to be a viable alternative to the CE loss:

where the subscript $d$ indicates the components corresponding to the distinct anchors, and the positive prediction $\hat{Y}^{*}$ in regression should also be replaced by $\hat{Y}_{d}^{*}$, which means that only the selected anchors have the chance to be positive.

In conclusion, the mixture model loss can be expressed as a combination of regression loss and a classification term:

The proposed Evolving and Distinct Anchors (EDA) is centered on the design of loss, where the anchors for identification of positive components are updated by the layer outputs and selected to be distinct before matching. Therefore, the paradigm can be readily applied to any network structure that includes a multi-layer decoder. When selecting distinct anchors through NMS, we utilize the scores from each layer to sort the corresponding anchors, since these scores actually represent the probability of the relevant anchors. Below is the pseudo-code in PyTorch (Paszke et al. 2019) demonstrating the EDA paradigm:

To align with the baseline, in addition to the mixture model loss, we also incorporate a loss for dense future prediction and an L1 loss on the agent velocity. As shown in Table 4, we adopt the same training configuration settings as in MTR (Shi et al. 2022a), where we train a single model for all three categories without any data augmentation.