ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking

Most recent 3D multi-object tracking (3D MOT) works follow the tracking-by-detection paradigm [1, 16, 3, 8, 10, 5], making the quality of 3D detectors a key component to tracking performance [3, 17, 18]. LiDAR-based 3D detectors extend the image-based 2D detection algorithms [19, 20] to 3D. The typical detector starts with an encoder [21, 22, 23] applied to unstructured point cloud data to extract an intermediate 3D feature grid [13] or bird’s-eye-view (BEV) feature map [24, 7]. Then, a decoder is applied to extract objectness [25] and detailed object attributes [26, 7]. Our approach proposes to leverage the intermediate BEV feature map for robust multi-object tracking, which can be generalized to most modern 3D detectors. CenterPoint detections have become a standardized 3D detection in the 3D MOT community, since the authors of the work released the official detection files for all data splits. As of this writing, CenterPoint has the highest mAP of all publicly-released 3D detections. Thus, in this work, we apply CenterPoint [7] for fair comparison against existing 3D MOT techniques.

With the advances in 3D detection, the 3D MOT community has been focusing on four directions to establish robust tracking within the tracking-by-detection paradigm: motion prediction, data association, track lifecycle management, and confidence refinement.

In 3D MOT, abstracted detection predictions are often used for motion prediction and data association, which is completed using pair-wise feature distance between tracks and detections [1, 2, 7]. AB3DMOT [1] provides a baseline to combine Kalman Filters and Intersection Over Union (IoU) association for 3D MOT. Subsequent works in this line explore variations of data association metrics, such as the Mahalanobis distance to leverage uncertainty measurements in the Kalman Filter [2]. As an alternative, CenterPoint [7] uses high-fidelity instance velocity predictions as a motion model to complete data association. Unlike these works that only use low-dimensional bounding boxes and explicit motion models for tracking, we propose to leverage intermediate representations from a detection network for richer information about each object, while also modeling the global relationship of all the objects in a scene to improve data association.

Additionally, track lifecycle management has been reasonably successful with heuristics that are tuned to each dataset [1, 2, 7], but these hand-tuned techniques have shortcomings in more complex situations. Recently, OGR3MOT [8] has proposed combining tracking and predictive modeling through a unified graph representation that learns data associations and track lifecycle management, FP elimination, and FN interpolation. Additionally, [5] proposes a statistical approach for data association under a Bayesian estimation framework. This work was later extended with Neural-Enhanced Belief Propagation (NEBP) [6] to include a learning-based aspect to the framework for FP suppression. Though these techniques yield competitive results compared to heuristics, their main shortfall is their inability to capture object shape information. Unlike these techniques, our approach leverages raw LiDAR data to capture shape information about objects in our scene, which we can use to further improve tracking performance.

Finally, track confidence is a less-explored but also important aspect of the tracking problem. Track confidence indicates the quality of a track relative to tracks for the same object class within a given frame. Just as it is the case for detection confidences, the most important characteristic of a track’s confidence is its value relative to other tracks’ confidences. In most 3D MOT works, algorithms assign the track confidence to be the same as that of the confidence score that comes with off-the-shelf detections matched to the track [1, 2, 7, 8]. In a recent effort to improve on this aspect of the tracking problem, [4] used an existing detection score refinement technique from [27] to obtain new tracking scores and improve the overall tracking performance. Though this method demonstrates improvements in ablative studies, its main weakness is that it directly applies a detection score refinement technique to the track score, overlooking the fact that unlike detections, tracks are ever-changing, time-dependent quantities that cannot be treated in isolation for the current time step. In this paper, we argue that track score refinement should be a sequential process that reflects changing environmental factors. Thus, we propose a first-of-its-kind approach that represents track score refinement as a sequential problem and leverages shape and spatio-temporal information encoded in our learned affinity matrix to obtain significant improvement in overall tracking accuracy.

Another line of 3D MOT algorithms focuses on using camera-LiDAR fusion to learn data association, track lifecycle management, and FP suppression [11, 3, 9, 10]. However, in this work we focus on LiDAR-only 3D MOT due to the practical advantages it offers. First, it is more cost-efficient to use fewer sensors. Additionally, LiDAR-only tracking reduces the risk of calibration and synchronization issues between various sensor modalities in a more complex system configuration.

Allowing for up to $N_{max}$ detections per frame, ShaSTA estimates the affinity matrix $A\in\mathbb{R}^{(N_{max}+2)\times(N_{max}+2)}$. As shown in Figure LABEL:fig:teaser, the rows of our affinity matrix represent the previous frame tracks and the columns represent our current frame detections. The two augmented columns are for the DT and FN anchors, so previous frame tracks can match with DT or FN. Similarly, because current frame detections can match with NB or FP, the affinity matrix has two augmented rows for the NB and FP anchors. Though there can be no more than one match for each bounding box in the current and previous frames, the same is not true for our learned anchors, i.e. more than one current frame bounding box can match with the NB or FP anchors, respectively. To handle this situation, we create a forward matching affinity matrix $A_{fm}\in\mathbb{R}^{N_{max}\times(N_{max}+2)}$ by removing the two row augmentations from $A$ and applying a row-wise softmax so that more than one previous frame track can have sufficient probability to match with the DT and FN anchors, respectively:

Using similar logic, we create a backward matching affinity matrix $A_{bm}\in\mathbb{R}^{(N_{max}+2)\times N_{max}}$ by removing the two column augmentations from $A$ and applying a column-wise softmax so that more than one current frame detection can have sufficient probability to match with the NB and FP anchors, respectively:

Here, forward matching indicates that we want to find the best current frame match for each previous frame track, while backward matching refers to finding the best previous frame match for each current frame detection.

To form tracks, we combine the affinity matrix outputs and the matching algorithm from [7] as follows. We take $A_{fm}$ and label any previous frame track that has a probability above the threshold $\tau_{dt}$ in the DT column as a DT, and we forward propagate any previous frame track into the current frame if it has a probability above the threshold $\tau_{fn}$ in the FN column. Therefore, in addition to provided detections, forward propagation creates a new detection for the current frame to handle occluded objects that off-the-shelf detections missed. We create a box in the current frame $b^{\prime}_{t}:=(x^{\prime},y^{\prime},z,w,l,h,r_{y})$, by moving the 2D center coordinate of the existing track $b_{t-1}:=(x,y,z,w,l,h,r_{y})$ to the next frame based on the estimated velocity provided by our off-the-shelf 3D detector and $\Delta t$ between the two frames, i.e. $x^{\prime}=x+v_{x}\Delta t$ and $y^{\prime}=y+v_{y}\Delta t$. All previous frame tracks that do not qualify as FN or DT are kept as is. Analogously, we take $A_{bm}$ and remove any current frame detection that has a probability above the threshold $\tau_{fp}$ in the FP row and we label any current frame detection that has a value above the threshold $\tau_{nb}$ in the NB row as an NB. All current frame detections that do not qualify as FP or NB are kept as is. See Section V-B for details on the threshold values we choose.

With these detections, we then run the greedy algorithm from [7]. In its original form, the greedy algorithm takes all unmatched detections and initializes NBs with them. However, we check if an unmatched detection is labeled as an NB in the affinity matrix based on the threshold $\tau_{nb}$ and whether it is outside the maximum distance threshold with respect to other tracks to initialize it as an NB. Otherwise, it is discarded. Additionally, we check if an unmatched track is labeled as a DT based on the threshold $\tau_{dt}$ and whether it is outside the maximum distance threshold with respect to other detections to terminate it as a DT.

ShaSTA uses off-the-shelf 3D detections, as well as the pre-trained LiDAR backbone used to generate the detections. ShaSTA first learns bounding box and shape representations for the anchors so we can later use this information to learn a residual that encodes spatio-temporal and shape similarities not only between current frame detections and previous frame tracks, but also between current frame detections and FP and NB anchors or between previous frame tracks and FN and DT anchors. The residual is then used to predict affinity matrices that assign probabilities for matching detections to tracks, eliminating FPs, propagating FNs, initializing NBs, and terminating DTs.

Inspired by [28, 29, 30], we train our model with the log affinity loss $\mathcal{L}_{la}$. We define the ground-truth affinity matrix $A_{gt}$, the estimated affinity matrix $A$, and the Hadamard product $\odot$ to get:

The affinity matrix prediction has corresponding ground-truth matrices $A_{gt,fm}$ and $A_{gt,bm}$ for forward matching and backward matching, respectively. In total, our overall loss function $\mathcal{L}$ is defined as follows:

We propose a sequential track confidence refinement technique that is used to assess the quality of tracks in real-time. We first offer an intuition behind our sequential track confidence refinement technique before formalizing our approach in Equation 24.

As stated earlier, track confidence should accurately reflect a track’s quality relative to the quality of tracks for objects of the same class in a given time step. Thus, to obtain more accurate confidence scores to characterize our tracks, we want to leverage our affinity matrix estimation as follows: (1) If the affinity matrix indicates that the detection matched to our track has a high probability of being true-positive, we want to take a weighted average between the confidence of our matched detection and the existing confidence of our track. (2) If the affinity matrix indicates that the detection matched to our track almost got eliminated as an FP – only marginally missing the elimination threshold $\tau_{FP}$ – then we want to downscale the existing confidence for our track, because it likely should not be kept alive with this matched detection. This approach is captured in Equation 24.

According to Equation 24, for track ID $i$ at time step $t$, we have a confidence of $c_{trk,i}^{(t)}$. The affinity matrix gets leveraged in the indicator function with $P_{FP,i}^{(t)}$, which is the probability that the detection got matched with the false-positive anchor at time step $t$. We fix $\beta_{1}$ to be a value slightly less than $\tau_{FP}$ to account for detections that are very close to getting eliminated as FP. In other words, we view detections with $P_{FP,i}^{(t)}\in[\beta_{1},\tau_{FP}]$ to be very uncertain detections, and we take this into account by reducing the overall track confidence for tracks matching to such detections. However, if the matched detection has a very high probability of being true-positive, then the track confidence $c_{trk,i}^{(t)}$ becomes a weighted average between $c_{det,i}^{(t)}$ and $c_{trk,i}^{(t-1)}$. Based on cross validation, we found that this refinement technique is not very sensitive to values for $\beta_{1}$ and $\beta_{2}$. In fact, we set $\beta_{1}=0.5$ for all object classes, and $\beta_{2}=0.5$ for all object classes except for bicycle ($\beta_{2}=0.4$), bus ($\beta_{2}=0.7$), and trailer ($\beta_{2}=0.4$). In the case of bicycle and trailer, we choose $\beta$ to be slightly lower, because we want to give more weight to the fact that it has been tracked since 3D detectors tend not to be very good on these object types and thus skew towards lower detection confidence scores. For the converse reason, we give bus a higher $\beta_{2}$.

In the special case of a newborn track, the track confidence is only the first term of Equation 24, i.e.

This works well, because the detection confidences are also created in relative terms. Thus, if all newly initialized track confidences are the detection confidences scaled with $\beta_{2}$, their relative confidence rankings will be preserved.

Since CenterPoint [7] provides detections at 2Hz, but the LiDAR has a sampling rate of 20Hz, we use the nuScenes [15] feature for accumulating multiple LiDAR sweeps with motion compensation. This provides us with a denser 4D point cloud with an added temporal dimension, and makes the LiDAR input match the detection sampling rate. We use 10 LiDAR sweeps to generate the point cloud.

We find the true-positive, false-positive, and false-negative detections using the matching algorithm from the nuScenes [15] dev-kit. For each frame, we define a ground-truth affinity matrix $A_{t}\in\mathbb{R}^{(N_{max}+2)\times(N_{max}+2)}$ between the current frame and previous frame off-the-shelf detections. For all true-positive previous and current frame detections, $B_{t-1}^{tp}$ and $B_{t}^{tp}$, we say they are matched if the ground-truth boxes they each correspond to have the same ground-truth tracking IDs. If there is a match between true-positive boxes $b_{t-1,i}^{tp}\in B_{t-1}^{tp}$ and $b_{t,j}^{tp}\in B_{t}^{tp}$, then $A_{t}(i,j)=1$. For all false-positive previous frame detections and true-positive previous frame detections whose tracking IDs do not appear in any of the current frame ground-truth tracking IDs, we call them dead tracks $B_{t-1}^{dt}$. For all $b_{t-1,i}^{dt}\in B_{t-1}^{dt}$, we set $A_{t}(i,N_{max}+1)=1$. For all true-positive previous frame detections that do not have a match in the current frame but whose tracking IDs do appear in the current frame ground-truth tracking IDs, we call them false-negatives $B_{t-1}^{fn}$. For all $b_{t-1,i}^{fn}\in B_{t-1}^{fn}$, we set $A_{t}(i,N_{max}+2)=1$. Additionally, for all true-positive current frame detections whose tracking IDs do not appear in the previous frame ground-truth tracking IDs, we call them newborn tracks $B_{t}^{nb}$, and we set $A_{t}(N_{max}+1,j)=1$ for all $b_{t,j}^{nb}\in B_{t}^{nb}$. Finally, we define the set of false-positive current frame detections as $B_{t}^{fp}$, and we set $A_{t}(N_{max}+2,j)=1$ for all $b_{t,j}^{fp}\in B_{t}^{fp}$. Unmatched $(i,j)$ entries are set to $0$.

The primary metrics for evaluating nuScenes are Average Multi-Object Tracking Accuracy (AMOTA) and Average Multi-Object Tracking Precision (AMOTP) [31]. AMOTA measures the ability of a tracker to track objects correctly, while AMOTP measures the quality of the estimated tracks.

AMOTA averages the recall-weighted MOTA, known as MOTAR, at $n$ evenly spaced recall thresholds. Note that MOTA is a metric that penalizes FPs, FNs, and ID switches (IDS). GT indicates the number of ground-truth tracklets.

Additionally AMOTP averages MOTP over $n$ evenly spaced recall thresholds. Note that MOTP measures the misalignment between ground-truth and predicted bounding boxes. For the definition of AMOTP below, $d_{i,t}$ indicates the distance error for track $i$ at time step $t$ and $TP_{t}$ indicates the number of true-positive (TP) matches at time step $t$.

The nuScenes benchmark [15] defines TP tracks as estimated tracks that are within an L2 distance of 2m from ground-truth tracks. This allows for a margin of error that can become unsafe in certain driving scenarios where small distances can make a dramatic difference in safety. Thus, even though the official nuScenes leaderboard uses AMOTA solely to rank trackers, we emphasize AMOTP as well due to its safety implications.

We also present secondary metrics, including the total number of TP, FP, and FN tracks to better analyze the effectiveness of our approach for downstream decision-making tasks.

We train a different network for each object category following [3, 4]. The value of $N_{max}$ depends on the object type since some classes are more common than others. We choose the value for $N_{max}$ based on the per-object detection frequency we gather in the training set. The smallest is $N_{max}=20$ and the largest is $N_{max}=90$. We use the Adam optimizer with a constant learning rate of $10^{-4}$ and L2 weight regularization of $10^{-2}$. We use the pre-trained VoxelNet weights from CenterPoint [7] and freeze them for our LiDAR backbone so that the shape information correlates with our detection bounding boxes from [7]. Additionally, there is a significant class imbalance between FPs and TPs since state-of-the-art detectors function in low precision-high recall regimes. Thus, we downsample the number of FP detections during training. Finally, we fix the thresholds $\tau_{fp}$ for FP elimination, $\tau_{fn}$ for FN propagation, $\tau_{nb}$ for NB initialization, and $\tau_{dt}$ for DT termination with cross-validation. We found that $\tau_{fp}=0.7$, $\tau_{fn}=0.5$, $\tau_{nb}=0.5$, and $\tau_{dt}=0.5$ works best for all classes.

Our nuScenes test results can be seen in Tables I and II. Because previous works have shown that the standard AMOTA tracking metric heavily depends on the upstream 3D detections [3, 17, 18], all tracking methods reported use CenterPoint detections to ensure a fair comparison¹¹1At the time of this writing, CenterPoint detections achieve the highest mAP of all publicly-available 3D detections. Higher-scoring detections appear on the nuScenes leaderboard, but are proprietary.. For more information on this evaluation protocol and the AMOTA metric, please refer to Section V-A.

The CenterPoint tracking algorithm [7] is our baseline, and we include recent, peer-reviewed works cited in Section II that tackle data association, track lifecycle management, FPs, and FNs in the LiDAR-only tracking domain. Most notably, before ShaSTA was developed, NEBP [6] was the #1 LiDAR-only tracker using CenterPoint detections.

In Table I, we can see that ShaSTA balances correct tracking (AMOTA) with precise, high-quality tracking (AMOTP). Most notably ShaSTA not only achieves the highest overall AMOTA, but also does so for 6 out of 7 object classes. The only class we are not first in AMOTA is the bicycle class, with NEBP taking the lead. However, it is interesting to note that our precision is much better for bicycles than NEBP is. NEBP averages over 1 meter in distance from the ground-truth tracks on bicycles, which can be very dangerous in real-world traffic scenes. Compared to NEBP, ShaSTA has a 34% improvement in AMOTP while only having an 8% reduction in AMOTA. Thus, the trade-off for the bicycles is clearly in ShaSTA’s favor. For the overall AMOTP, we achieve the second best value only trailing CenterPoint by 0.005 meters, which is a favorable trade-off when compared to our significant boost in overall AMOTA.

Examining further metric breakdowns in Table II, it becomes clear that ShaSTA is effective in de-cluttering environments suffering from FPs and compensating for missed detections that lead to missing TPs and rising FNs. The two most observed classes that constitute over 80% of the test set are cars and pedestrians, and we get highly competitive results in both of those categories, which tend to suffer the most from crowded scenes. Thus, our FPs reflect our effectiveness at dealing with these sorts of scenes. Another interesting interplay between these metrics is that often times decreasing FPs comes at the risk of increasing FNs and decreasing TPs. Thus, it is quite challenging to tackle the FP, FN, and TP problem simultaneously, yet our approach does so most effectively out of all existing tracking methods.

We complete four ablative experiments on the nuScenes validation set to measure the impacts of (1) the sequential track confidence refinement, (2) the number of shape descriptors from the BEV map, (3) each of the residuals used to make the overall residual prediction, and (4) including FP elimination, FN propagation, NB initialization, and DT termination in the learned affinity matrix. In total, our ablative studies demonstrate that all of the components in ShaSTA contribute to the overall success of this tracking framework.

First, we analyze the effect of the sequential tracking confidence refinement as seen in Table III. When we do not use our refinement technique, we apply the CenterPoint detection confidence scores to our tracks as in past works [1, 6, 2, 3, 7, 8]. We note a significant boost in the overall AMOTA for the validation set when using the proposed refinement method. All object classes benefited, but the ones that improved the least are large objects like trucks, cars, and buses. This makes sense since these types of objects tend to provide the best LiDAR point clouds and are less likely to suffer from full occlusion, allowing the detection confidence to be a good indicator for tracking. Conversely, smaller objects like bicycles and motorcycles that tend to suffer the most from LiDAR point cloud sparsity due to increased distance from the ego vehicle and occlusion in traffic scenes benefit the most from this technique. In these cases, the pure detection confidences can drop dramatically when there are scenarios such as full occlusion of these small objects, but with our sequential refinement we take into account the existing confidence that has accumulated for a track before such scenarios occur.

Additionally, in Table IV, we can see that as we add more bounding box face centers for extracting shape descriptors, the accuracy of our model monotonically increases. This result supports our argument that leveraging raw LiDAR point clouds to get shape information is integral to the success of our tracking framework. Most notably, when we have less shape information, more objects within the same class are more likely to appear similar, leading to more ambiguity in distinguishing between TPs and FPs. Such ambiguity makes the track confidence refinement less effective, specifically in the indicator function in Equation 24.

Furthermore, our ablation on the individual residuals in Table V demonstrates that even though we can achieve competitive tracking accuracy solely with spatio-temporal (VoxelNet or Bounding Box only) or shape (Shape only) information, we achieve the most optimal results when leveraging both types of information.

Lastly, in Table VI, we see that the most impactful augmentations are FP elimination and NB initialization. Eliminating FPs is expected to boost our accuracy since off-the-shelf detections suffer from such high rates of FP detections. NB initialization can be seen as a second reinforcement on eliminating FP detections because unmatched detections that do not have a probability of at least $\tau_{nb}$ for being NBs do not get initialized as tracks and are discarded. Finally, even though FNs and DT termination have not been as detrimental to tracking algorithms in the past, these augmentations alone give us competitive accuracies, showing that every augmentation contributes to effective tracking.

As indicated in Table I, there are significant increases in test accuracy over the CenterPoint [7] baseline, particularly for objects like cars and pedestrians that are often found in crowded scenes and thus, have increased susceptibility to experiencing high rates of FP tracks.

At high recall thresholds, FP tracks tend to appear more frequently, but the AMOTA metric does not strongly penalize FP tracks at higher recall thresholds, diminishing their significant effect on real-world deployment for downstream decision-making tasks. To showcase ShaSTA’s ability to eliminate FPs effectively, we present tracking results in Figure 3. We only show the tracking results on the car object class from validation scene 270. (a) shows the CenterPoint [7] baseline which has FP tracks in all three frames. On the other hand, (b) demonstrates ShaSTA’s ability to declutter the scene and accurately track the car, while ignoring background information that should not be tracked. Typically, past works [4] use non-maximal suppression heuristics such as intersection over union (IoU) to remove FP tracks, but in more complex settings like scene 270, such heuristics would fail because none of the FPs overlap with other track predictions. As a result, ShaSTA proves to be especially effective at using shape and spatio-temporal information from LiDAR point clouds to localize and track actual objects of interest in the scene.