Spb3DTracker: A Robust LiDAR-Based Person Tracker for Noisy Environment

Many recent person trackers are designed based on end-to-end deep learning networks. A common approach involves integrating recurrent layers with detector modules. For instance, ROLO combines the convolutional layers of YOLO with LSTM recurrent units [16], while TrackR-CNN extends multi-object tracking to include instance segmentation [20]. Tractor++ offers an efficient tracking solution by utilizing bounding box regression to predict object positions in subsequent frames without requiring training on tracking data [2]. Other approaches focus on enhancing tracking performance through various techniques. Deep SORT integrates appearance information with the Simple Online and Realtime Tracking (SORT) technique, employing a recursive Kalman filter and frame-by-frame data association [25] [28]. This method incorporates an offline pre-training stage to learn a deep association metric using large-scale person re-identification datasets [12]. Similarly, some trackers adopt a single-stage approach, learning target detection and appearance embedding in a shared manner, often complemented by Kalman filters for location prediction [22]. While 2D trackers can leverage rich RGB information for appearance models, a resource not typically available in LiDAR-based 3D tracking, they often struggle with accurately representing 3D scene dynamics [3]. Conversely, 3D tracking methods can better handle scale variations and provide more accurate spatial information [9]. Ultimately, the design of person tracking methods should align with the specific characteristics of each modality to achieve optimal results, whether utilizing 2D image data, 3D LiDAR point clouds, or a fusion of multiple sensor inputs [24].

Recent developments in Lidar-Based Tracking (LBT) have explored diverse strategies to enhance tracking performance in dynamic environments [32]. Some approaches focus solely on 3D bounding boxes detected from point clouds, offering rapid processing and comprehensive scene dynamics capture, albeit potentially underutilizing point cloud appearance features [32]. Alternative methods have incorporated custom point cloud features or employed deep networks to estimate 3D bounding boxes from single-view images [32][17]. More recent approaches have shown promising results by combining features extracted from both RGB images and point clouds [34]. Many LBT techniques rely on rule-based components. For instance, AB3DMOT, a widely-used baseline, employs Intersection over Union (IoU) for association and a Kalman filter for motion modeling [32]. Subsequent enhancements have focused on improving the association step, such as substituting IoU with Mahalanobis or L2 distance metrics [34]. Researchers have also recognized the significance of lifecycle management in tracking systems [34]. Additionally, recent studies have investigated the application of graph neural networks (GNN) to address LBT in an end-to-end manner, with a focus on data association and active tracklet classification [27][11]. As the field progresses, there is an increasing need for comprehensive studies on 3D Multi-Object Tracking (MOT) methods to identify areas for improvement and guide future research directions [34]. This systematic approach will help in addressing the challenges unique to LBT and advancing the state-of-the-art in this domain [32][34].

Kalman filters (KF) are widely used in tracking-by-detection methods for object tracking in autonomous driving due to their effectiveness in filtering noise from state estimations while handling noisy measurements [33]. However, standard KFs have limitations in real-world scenarios. They assume linear system models and Gaussian noise distribution, which may not hold in complex environments [33][29]. They struggle with non-linear motion and poor observation conditions due to fixed system covariance, leading to error accumulation in state predictions, especially during periodic occlusions of objects [29]. Additionally, spatial distortions in 3D sensor detections can result in imprecise motion parameter estimations [33]. To address these issues, advanced filtering techniques such as the Unscented Kalman Filter (UKF) and Cubature Kalman Filter (CKF) have been proposed [29][30]. These filters are designed to better handle non-linearities and approximate state distributions more accurately than standard KFs [30]. Furthermore, adaptive filters can adjust system covariance dynamically to better respond to changing environments, improving tracking performance in complex, real-world autonomous driving scenarios [30][31][13].

Our approach diverges from conventional algorithms that typically retain only high-confidence 3D detection boxes. Instead, we implement a more inclusive strategy that avoids strict confidence thresholds [33]. This method aims to preserve potential tracklets and enhance recall performance. In our pipeline, we employ the DSVT (Dynamic Sparse Voxel Transformer) detection model[21] and apply Non-Maximum Suppression (NMS) to refine the detection results. Unlike image-based approaches, our LiDAR-based method focuses on 3D space, simplifying the post-processing steps. By eschewing a fixed confidence threshold, our approach seeks to strike a balance between retaining potentially valid detections and mitigating false positives. This strategy is particularly effective in challenging scenarios where low-confidence detections may still represent genuine objects. As illustrated in Fig. 3, our experiments reveal an inverse relationship between the confidence threshold and tracking performance metrics (AMOTP and AMOTA). Lower thresholds correspond to improved metric scores, underscoring the importance of managing "ghost" objects (false positives) in enhancing recall.

Given the limited availability of person data in the KITTI dataset[8], we adopted a strategy to enhance our model’s performance through transfer learning and multi-dataset training. This approach is particularly effective for image-based detection tasks, where pre-trained models often yield superior results. To create a robust baseline model, we utilized a combination of another public datasets including person class. Rather than training sequentially on each dataset, we implemented a batch-wise shuffling technique. This method ensures that during each training iteration, the model is exposed to a diverse range of samples from all datasets simultaneously. Our training procedure can be summarized as follows:

• •

  We combined samples from KITTI, our custom dataset, and another public dataset.

• •

  During batch formation, we randomly shuffled samples from all datasets.

• •

  This shuffled batch was then used for training, allowing the model to learn from diverse data sources concurrently.

After creating this pre-trained model using the multi-dataset approach, we fine-tuned it exclusively on the KITTI dataset. This two-stage process allows our model to benefit from the rich, diverse data of multiple sources while still specializing in the specific characteristics of the KITTI dataset. This methodology aims to address the scarcity of person data in KITTI by leveraging additional datasets, potentially improving the model’s generalization capabilities and performance on the target dataset. Table 1 demonstrates that using a pretrained model with shuffling improves the Average Precision (AP).

Most previous methods apply simplistic motion models that fail to adequately account for the complexity of a person’s movement. persons can move freely along the X and Y axes and rotate around the Z-axis (yaw direction) according to the right-hand rule. Furthermore, analysis of person detection results reveals the difficulty in distinguishing between front and back views of individuals. Consequently, prediction models relying solely on heading angle may be inaccurate. To address these limitations, we propose formulating the state of an object trajectory as an 11-dimensional vector:

$(x,y,z)$ represents the 3D location of the object’s geometric center.$(\theta)$ represents the object’s orientation around the Z-axis.$(v_{x},v_{y})$ represents the object’s velocity components in the X and Y directions.$(a_{x},a_{y})$ represents the object’s acceleration components in the X and Y directions.$(w,l,h)$ represents the object’s 3D dimensions (width, length, height)

The standard Kalman Filter is widely used in Track-By-Detection (TBD) tracking modules for associating motion model systems with measurement systems (detection systems). Most previous methods assume linear object motion and Gaussian distribution. However, person motion is inherently non-linear, and the typical LiDAR sensor frame rate of 10Hz, coupled with TBD detection module latency of 0.3-0.7 seconds, often violates the Gaussian distribution assumption in the tracking module. To address these issues, we propose a Dynamic Unscented Kalman Filter (D-UKF). We leverage the D-UKF to estimate the trajectory state. The prediction process can be described by:

where $T$ denotes the prediction model, $D$ means the detection model, $P$ is the covariance matrix of the previous frame, $Q$ is the prediction model system’s noise, and $R$ is the detection model system’s noise. $f(\cdot)$ represents the motion model, and $h(\cdot)$ is the detection model. The motion model is defined as:

Traditional Kalman Filters use fixed measurement system covariances $R$. In an ideal scenario, this error would be zero. However, in real-world applications, this is not achievable. Our D-UKF approach, when generating sigma points, spreads these points according to the system covariance. Unlike traditional KF that use fixed system noise, our SpbTracker dynamically adjusts system noise based on confidence of detection. We propose an D-UKF that modifies the measurement system covariance as follows:

This adaptive mechanism allows the filter to adjust to varying levels of measurement uncertainty, improving tracking performance across diverse scenarios without manual parameter adjustments. By incorporating this approach, our method aims to enhance robustness and generalizability in PDT, particularly in challenging environments with complex motion patterns and varying sensor characteristics. Tables 2 and 3 demonstrate the effectiveness of our Dynamic UKF in noisy environments. We conducted an ablation study using the KITTI validation dataset. For the 5Hz dataset, we analyzed every other data point in the sequence to maintain consistency.

The Generalized 3D Generalized IoU (GIoU) is a widely adopted association metric in object tracking. This metric enhances the traditional IoU (Intersection over Union) by accounting for the spatial relationship between non-overlapping bounding boxes, thereby providing a more comprehensive measure of object overlap. However, GIoU solely reflects the proportion of overlap, neglecting crucial factors such as volume size and object viewpoint. To address these limitations, we incorporate the concept of Complete-IoU (CIoU) into our association framework. CIoU extends the capabilities of GIoU by considering additional geometric properties, thus offering a more nuanced approach to object association. We suggest new metric by modifying CIoU metric as Modified Complete-IoU(MCIoU). MCIoU is specifically designed to reflect the unique characteristics of person tracking, particularly taking into account the ratio of person height:

Nevertheless, IoU-based association methods inherently face challenges in Re-Identification (Re-ID) tasks, particularly when dealing with objects that have been absent from the scene for extended periods. This limitation arises from the reliance on threshold distances for matching, which can fail to associate objects that have undergone significant spatial displacement over time. To mitigate this issue, we augment our association strategy by incorporating feature similarity measures. This additional dimension allows for more robust matching, especially in scenarios where spatial proximity alone is insufficient. we utilize bird’s eye view(BEV) features sampled via region of interest(ROI) as feature similarity. Considering that the matched object ROI features of the two branches, we design the feature similarity(FS) as:

Our final association metric is derived from a weighted sum of the MCIoU score and FS measure. This composite approach leverages the strengths of both geometric and appearance-based cues, resulting in a more robust and versatile association framework capable of handling diverse tracking scenarios.

Fig. 4 presents a comparison of association metrics on the KITTI validation set. The optimal method is positioned closest to the bottom-left corner, indicating the lowest number of ID-Switches and the highest AMOTA score. Our proposed metric demonstrates superior performance among the compared methods, as evidenced by the plot.

While previous approaches often rely exclusively on high-confidence 3D bounding boxes, our method incorporates all predicted 3D boxes to enhance recall metrics. To address the challenges associated with this comprehensive approach, we adopt a holistic strategy as follows: Firstly, we utilize the F1 score as the classification criterion. We associate the track pool with all detection results using our previously described association method. This approach, however, can potentially lead to decreased precision due to the inclusion of low-confidence detections and increased computational complexity in the association step, which employs the Hungarian algorithm. In contrast to conventional algorithms that use distance-based thresholds for association, we employ a Feature Similarity (FS) metric. Among pairs exceeding a distance threshold, we compare their FS values. Pairs with FS values above a predetermined threshold are retained in the pair memory pool. Matched pairs undergo processing through a Dynamic Unscented Kalman Filter (UKF) and a Low-Pass Filter (LPF) for confidence score smoothing. The LPF mitigates rapid fluctuations in confidence scores, applying to both matched trajectory scores and detection result scores. If the LPF result exceeds the F1-score threshold, the corresponding tracklet is classified as active. The LPF is defined as:

where $T_{score}$ represents the matched trajectory confidence score, and $D_{score}$ denotes the matched detection result confidence score. For unmatched detection results, if the score exceeds the F1-score threshold, the detection is either initialized as a new tracklet or classified as a candidate trajectory. Unmatched trajectories undergo score decay based on their distance from the ego-vehicle(or lidar), reflecting the increased likelihood of object disappearance at greater distances. Trajectories with scores falling below the death threshold are removed from the manager memory. The Confidence Decay Distance (CDD) is calculated as:

Fig. 5 demonstrates the long-term tracking persistence capability of our life-cycle memory approach. In frame 2, SpbTracker successfully tracks a person labeled as ID-2. From frames 3 to 20, the object becomes occluded and temporarily disappears from view. Despite this extended period of occlusion, SpbTracker maintains the object’s identity in memory. When the object reappears in frame 21, SpbTracker correctly recognizes it as ID-2, avoiding an identity switch (IDsw). This description effectively conveys the key aspects of your tracking system’s performance, highlighting its ability to maintain object identity through extended periods of occlusion.

We implemented our proposed methodology using Python and C++, and conducted experiments on an Intel Xeon Gold 5218R CPU (2.10GHz) with 125GB RAM. All reported results reflect online performance.