BOTT: Box Only Transformer Tracker for 3D Object Tracking

Under the tracking-by-detection framework, AB3DMOT [20] serves as a baseline, using a simple KF tracking framework. Many methods have been proposed to improve the tracking performance based on the same KF-based tracking framework, such as ProbTrack [5] and SimpleTrack [13]. Their major difference lies in the association metrics: AB3DMOT used 3D Intersection of Union (IoU); ProbTrack used Mahalabnobis distance; and SimpleTrack used 3D generalized 3D (GIoU) while a second association was applied for better handling lower-score objects.

Lately, more learning-based tracking algorithms have been reported. Some of them proposed to use GNNs [3, 7, 21, 24] as graphs are a natural representation of the MOT problem, where the detected objects are encoded as nodes and their spatial-temporal relations are represented by edges. GNN3DMOT [21] used GNNs to estimate an affinity matrix and solved the association using the Hungarian algorithm. OGR3MOT [24] solved 3D MOT with GNNs in an end-to-end manner, focusing on the data association and the classification of active tracklets. Batch3dmot [3] presented a novel multi-modal GNN framework for offline 3D MOT on multi-class tracking graphs that introduces $k$ nearest neighborhood attention across graph components to allow information exchange across disconnected graph components. Closest to our approach, PolarMOT [7] explored the impact of geometric relationships between objects based on GNNs with solely 3D boxes as well. The difference is that in the PolarMOT, mutual object interactions within local regions are learned implicitly through iterative message-passing steps, while we used self-attention to get the global context information in one shot.

Over the past few years, transformer networks have gained great momentum for their ability to effectively process sequence data. Encouraging tracking performance on 2D MOT has been demonstrated from transformer trackers with image appearance features [6, 12, 15, 17, 25, 26]. This is mainly because transformers could handle long-term dependencies and is robust to occlusions and complex scenarios. Among these trackers, SODA [6] and GTR [25] are more related to our work. SODA proposed an attention measurement encoding to compute track embeddings from objects’ appearance features to reason about the spatial-temporal dependencies between all objects. GTR fed the objects’ appearance features into the encoder of the transformer, additionally took trajectory queries as the decoder input, and produced association scores between each query and object. Different from their work, our work learns the spatial-temporal context information from 3D bounding boxes only without appearance using simple self-attention encoders.

Recently, offline auto-labeling methods [22, 14] have drawn great attention in the autonomous driving industry, as they could scale up the data annotation drastically. In the auto-labeling pipeline, future information could be introduced to improve tracking performance. It is not straightforward for KF-based trackers [5, 13, 20] to leverage on the future information, as they are designed to operate recursively and rely solely on the current state and observations. In contrast, BOTT provides a convenient solution for both online and offline multi-class tracking.

In a scene with $T$ frames $\{I^{1},I^{2},\cdots,I^{T}\}$, there are $n_{t}$ detected 3D boxes for frame $I^{t}$: $B^{t}\mathrm{=}\{b^{t}_{1},b^{t}_{2},\cdots,b^{t}_{n_{t}}\}$. Each box $b^{t}_{i}$ contains raw features including $(x^{t}_{i},y^{t}_{i},z^{t}_{i},w^{t}_{i},l^{t}_{i},h^{t}_{i},\theta^{t}_{i},t,c^{t}_{1i},c^{t}_{2i},\cdots,c^{t}_{Ci})$ ($xyz$: center, $wlh$: size, $\theta$: yaw, $t$: time, $c_{1}{\cdots}c_{C}$: classification scores for $C$ categories). A sliding window $S^{t}_{K}$ is defined as a set of all the boxes from $K$ consecutive frames: $S^{t}_{K}\mathrm{=}\{B^{t-K+1},\cdots,B^{t-1},B^{t}\}$. For simplicity, $S^{t}$ denotes a sliding window with latest frame $I^{t}$ without explicit statement of $K$, and $N_{t}\mathrm{=}\sum_{j=0:K-1}n_{t-j}$ denotes the total box number within $S^{t}$.

As shown in Figure. 1, BOTT accepts sliding windows as input and estimates pairwise box linking scores using a BOTT network. Then a box tracking module connects the boxes to produce tracks using the linking scores. The BOTT network contains three components: 1) single-box feature encoding; 2) inter-box encoding with self-attention; and 3) linking scores estimation.

This module aims to learn high-level per-box features from raw geometric features. The raw center values $xyz$ may vary greatly in different sliding windows because they are in the shared global coordinates. To reduce variance, center positions of all boxes in $S^{t}$ are normalized by subtracting the minimal center values $(x^{t}_{min},y^{t}_{min},z^{t}_{min})$ of all boxes. Take $x^{t}_{min}$ for instance, $x^{t}_{min}=\min_{\{(t,i)|b^{t}_{i}\in S^{t}\}}{x^{t}_{i}}$. The time feature $\Delta_{i}^{t}$ of box $b_{i}^{t}$ is encoded as the delta between the box frame and center frame in $S_{t}$. Box heading angle is encoded as $[\sin(\theta^{t}_{i}),\cos(\theta^{t}_{i})]$. In summary, the input feature of $b^{t}_{i}$ includes $(x^{t}_{i}-x^{t}_{min},y^{t}_{i}-y^{t}_{min},z^{t}_{i}-z^{t}_{min},w^{t}_{i},l^{t}_{i},h^{t}_{i},\sin(\theta^{t}_{i}),\cos(\theta^{t}_{i}),\Delta_{i}^{t},c^{t}_{1i},c^{t}_{2i},\cdots,c^{t}_{Ci})$. Each box feature vector is fed into a shared Multi-Layer Perception (MLP), which has $N_{MLP}$ latent layers with $d_{l}$ dimensions and a final output layer with $d$ dimensions. $N_{t}\times d$ feature embeddings are learned for all boxes $S^{t}$.

After per-box feature encoding, the $N_{t}\times d$ output embeddings are fed to a self-attention module to encode inter-box relationship. Specifically, $N_{enc}$ identical transformer encoder blocks [19] are applied sequentially to exchange information between all input box embeddings. Each encoder block consists of a multi-head attention network with $n_{h}$ heads and a following feed-forward network with $d_{f}$ hidden nodes. After self-attention, the size of output box embeddings remains $N_{t}\times d$ for $S^{t}$.

Notably, the self-attention in BOTT is class-agnostic, i.e. each box learns to get information from all the other boxes in the sliding window. As shown in Figure. 5.1, the moving car in sub-figure (a) and (b) also gets attention from nearby pedestrians, and the pedestrian in sub-figure (c) and (d) also gets attention from nearby cars. Self-attention is expected to learn robust box representation by encoding global spatial-temporal box distributions into each box. Another advantage of class-agnostic self-attention is that it enables BOTT to handle multi-class objects tracking with a single model. It greatly reduces the deployment complexity in practice.

Tracked boxes from the same object share a similar spatial-temporal context within a sliding window. With the learned $N_{t}\times d$ box embeddings, $L_{2}$ normalization is performed to normalize each box embedding to unit length. Then, simple dot-product is used to compute inter-box linking scores, resulting in a $N_{t}\times N_{t}$ linking score matrix $LS^{t}$ (illustrated in Figure. 1). For convention, the linking score is normalized to range $[0,1]$ by $LS^{t}\mathrm{=}\frac{LS^{t}+1}{2}$. A linking score of $1$ indicates that two boxes belong to the same object, and $0$ means the opposite. In this way, the tracking is converted to a binary classification problem of box links.

Assume $y^{t}\in\mathcal{R}^{N_{t}{\times}N_{t}}$ denote the corresponding ground truth linking score matrix for $LS^{t}$, binary cross entropy between them is used as the loss. During training, to prevent the easy cases from overwhelming the loss, a binary mask $M^{t}\in\mathcal{R}^{N_{t}{\times}N_{t}}$ is constructed to ignore losses from: 1) inter-class box links; 2) box links from the same frame; 3) box links between two false positive boxes (detected boxes without associated ground truth boxes); and 4) box links whose center distances exceed the expected maximum displacement for the object over the given time duration.

Given a batch data of $b$ sliding windows $\{S^{t},t\in\{t_{1},t_{2},\cdots,t_{b}\}\}$, the linking loss $L_{link}$ is computed as:

	$\displaystyle L^{t}_{ij}$	$\displaystyle=(1-\beta)(1-y^{t}_{ij})\log(1-LS_{ij}^{t})+\beta y^{t}_{ij}\log LS_{ij}^{t}$
	$\displaystyle L_{link}$	$\displaystyle=\frac{\sum_{t}\sum_{i,j}(M^{t}_{ij}\cdot L^{t}_{ij})}{\sum_{t}\sum_{i,j}M^{t}_{ij}}$		(1)

where $\beta$ is the positive sample weight.

Linking scores are utilized to create tracks in box tracking module. Depending on whether future data can be used, BOTT can perform both online and offline tracking.

We evaluate BOTT on the two largest benchmarks for 3D MOT: nuScenes [4] and Waymo Open Dataset (WOD) [18].

CenterPoint [23] is deployed on the training, validation and test sets on both nuScenes and WOD to get the detections to generate a track database. With the detections, a Non-Maximum Suppression (NMS) is applied to remove overlapped boxes, and boxes with detection scores below a threshold are also filtered. To generate the database, an association between detections and GT boxes provided by nuScenes or WOD is performed. The track IDs in the GT boxes are assigned to the associated detection boxes, and unmatched detection boxes are considered as false positives. For both nuScenes and WOD, the track database are generated at 10Hz. The 2Hz GT provided by nuScenes is interpolated to get 10Hz GT. Each scene in the track database is divided into overlapping $K-$frame sliding windows with a stride of one. This work uses $K\mathrm{=}{16}$.

Tracking performance heavily relies on the detection quality. For fair comparison, we compare BOTT with these published trackers that are also based on the commonly used CenterPoint detections [23]. Among them, AB3DMOT [20] and SimpleTrack [13] trackers belongs to classic motion trackers, while PolarMOT [7], OGR3MOT [24], CenterPoint [23] and Batch3DMOT [3] are learning based trackers. Table. 1 shows the 3D MOT results on nuScenes validation and test sets. In the validation set, we compare the performance for both online and offline tracking. Our online BOTT achieves a 2.53 AMOTA improvement over learning based tracker PolarMOT, and achieves slightly better performance that the advanced classic motion trackers, i.e. SimpleTrack [13]. Our offline BOTT achieves state-of-the-art performance of the Lidar box-based offline tracking algorithms, with 71.38 AMOTA. One should note that the tracking performance for Batch3DMOT [3] is presented with box feature only. For the test set, only online tracking results are compared between different trackers. Similarly, our online BOTT achieves better performance than learning based box tracker, and comparable results with classic motion trackers.

Table. 2 shows the 3D MOT results on WOD. Our online BOTT could achieve better results than learning based trackers, i.e. CenterPoint [23], and achieves comparable performance to advanced classic motion trackers, i.e. SimpleTrack [13].

Figure. 3 illustrates the qualitative results of BOTT which accumulate over 40 frames in a scene on the nuScenes validation set. Figure. 3 (a) and (b) shows GT and raw detections from CenterPoint [23], respectively, and (c) and (d) shows the tracking results of CenterPoint [23] and our BOTT.

Figure. 5.1 showcases a few examples of the attentive boxes from the same frame that relate to the circled reference boxes. We could see that the boxes which are closer to the reference boxes normally have stronger attention impact than the boxes far away. Nevertheless, faraway boxes also contributes to the reference box due to the global self-attention. Notably, the figure only shows attentive boxes from the same frame for illustration convenience, in fact all boxes in the sliding window contribute to the reference box for a robust box embedding.