3D Multiple Object Tracking on Autonomous Driving: A Literature Review

In the realm of 3D MOT, it’s crucial to acknowledge that the foundation of current research extensively relies on the outcomes of 3D object detection. In essence, many of the inputs for 3D MOT are derived from the outputs of 3D object detection [7, 8, 9, 10, 11, 12, 13]. Therefore, it becomes imperative to explore the landscape of 3D object detection in a comprehensive manner, as it serves as the cornerstone for advancements in 3D MOT.

A prevalent approach in contemporary 3D object detection is the utilization of point clouds or multi-modal data [14, 15, 16, 17, 18, 19]. Point clouds represent the real world as a set of discrete points, where each point encapsulates essential data pertaining to an object. In essence, point cloud data embodies an assemblage of vectors in a three-dimensional coordinate system.

It’s universally recognized that images inherently possess dense and structured information, encompassing vibrant color palettes and intricate texture details. Nonetheless, images do come with inherent limitations, particularly concerning scale variations induced by objects’ varying distances and proximities. In stark contrast, point clouds proffer an abundance of three-dimensional spatial insights, encompassing geometric structures and depth data. Consequently, Lidar point cloud-based methodologies have garnered acclaim for delivering heightened detection accuracy and efficiency when juxtaposed with their camera-based counterparts. A prominent and ongoing area of exploration within 3D object detection and tracking revolves around the integration of multi-modal data to augment detection and tracking precision.

The nexus between point clouds and image fusion pivots on the establishment of absolute coordinates. Essentially, determining the transformation matrix between Lidar and camera sensors suffices to derive the coordinate transformation between the two sensors’ respective coordinate systems. This juncture forms a focal point of investigation and discourse within this paper, where we delve into the intricacies of multi-modal fusion.

Presently, the most prominent methodologies for multi-modal fusion in the realm of 3D object detection can be categorized into three primary domains.

• •

  Early Fusion: These approaches involves feature fusion before extracting features from the raw sensor data. It has the advantage of maximizing the use of multimodal information. Notable papers in this area include PointPainting [20], PIR-CNN [21], and Pointaugmenting [22], etc.

• •

  Deep Fusion: These methods extract features from the original data and then fuse the two feature layers. There are two types of fusion methods: Point-based and Voxel-based. The Point-based method has the inherent advantage of having the same index as the image and no spatial variation, while the Voxel-based method can make more efficient use of the perceptual power of convolution. Representative papers include EPNet  [23], 3D-CVF [24], UVTR [18], LoGoNet [16] and BEVFusion [19], etc.

• •

  Late Fusion: This fusion method first detects the image and the point cloud separately, presents the Proposals, and only after this does the fusion take place. The two are only correlated at the detection level and the multi-modal data does not need to be synchronized or aligned with other modalities; only the final fusion step requires joint alignment and labeling of the data. It has the advantage of low complexity. Notable papers include CLOCs [25] and Fast-CLOCs [26], etc.

The primary distinction between object detection and object tracking is that object detection involves both localization and classification, while object tracking focuses on the trajectory of the target movement in addition to localization based on object detection. The classification referred to in this paper mainly pertains to target types such as cars (vehicle), pedestrians, and bicycles. Undoubtedly, the vehicle category is of utmost importance. Many comprehensive reviews on 3D object detection [27, 28, 29] present detailed experimental results specifically for this category.

Object tracking involves establishing the position relationship of the object being tracked in a continuous video sequence and obtaining its complete motion trajectory. During this tracking process, several challenges may arise, including changes in shape and lighting, variations in target scale, target occlusion, target deformation, motion blur, fast target motion, target rotation, target escape from parallax, background clutter, low resolution, and other phenomena.

Autonomous vehicles are equipped with various sensors, including cameras, radar, and Lidar. These sensors collect a large amount of raw data while the vehicle is in motion and send it to the detector module. In this module, the camera obtains the bounding box, the radar obtains the orientation in polar coordinates and Doppler measurements, and the Lidar obtains point cloud data. This data is then fed into the multi-object tracking module.

Most current research on 3D MOT are based on 3D object detection methods, using both point cloud data or multi-modal fusion approaches. The mainstream research methods have focused on kernel correlation filtering methods and deep learning methods. In recent years, there has been a shift towards deep learning-based approaches [30, 31, 32], with the main branches being LSTM, CNN, R-CNN, GNN, and Transformer.

Object tracking involves establishing the position of a target within a continuous video sequence to obtain its complete motion trajectory. This process presents several challenges, including changes in shape and lighting, variations in target scale, target occlusion, target deformation, motion blur, rapid target movement, target rotation, parallax escape of the target, background clutter, and low resolution phenomena. While MOT aims to identify the targets to be tracked in a sequence of images and correspond the targets in different frames one by one. This process also presents numerous challenges. For instance, there can be overlap and occlusion between targets, re-identification of old and new targets, and matching of targets in consecutive frames.

The effectiveness of 3D object tracking systems is largely determined by the accuracy of 3D object detection. The precision of 3D MOT models can be significantly affected by factors such as scale changes, frequent ID switching, rotation, and lighting changes. Despite extensive research on 3D object detection [33, 34, 35, 36, 37, 38], practical applications still face numerous challenges. Firstly, there is a need for robustness to object occlusion, truncation, and changes in the surrounding dynamic environment. Secondly, most existing approaches rely on object surface texture or structural features, which can easily lead to confusion. Furthermore, there are significant issues with algorithm efficiency while meeting accuracy requirements. Given that 3D object detection forms the necessary foundation for 3D multi-object tracking, achieving a balance between detection efficiency and accuracy is a critical issue that warrants in-depth exploration in this paper.

From the analysis, it’s clear that 3D MOT aims to address the challenges of 3D object detection and MOT [39]. The current technology in this field represents a new direction of development, but also faces a series of issues such as frequent target occlusion, background clutter, point cloud sparsity and density, unknown trajectory start and end times, small targets, apparent similarity, inter-target interaction, and low frame rates. On the other hand, current 3D MOT work tends to prioritize accuracy of development over computational cost and system complexity. This indicates that the 3D MOT problem is more complex and requires further study to understand its classification, principles, algorithms, models, experimental results, and trends. This understanding will aid in the investigation of potential solutions.

The traditional types of 2D object tracking include Single-Object Tracking (SOT), Video Object Segmentation (VOS), MOT, and Multi-Object Tracking and Segmentation (MOTS). In this paper, we introduce a structured framework for categorizing 3D MOT along two key dimensions. 3D MOT can be approached through various data modalities, including camera-based, point cloud-based, and multi-modality-based methods. Additionally, 3D MOT methods can be broadly classified into two categories: Kernel Correlation Filtering methods (KCF) and Deep Learning methods.

The evolution of object tracking methods has followed a trajectory from classical algorithms to KCF algorithms, and subsequently to deep Learning-based tracking algorithms. Classical tracking algorithms, employed in the early stages, primarily relied on target modeling or tracking target features. This category includes methods such as optical flow and particle filtering.

KCF methods, originally used in signal processing and communication, were adapted for object tracking. These methods found widespread use in real-time tracking systems, contributing to improved tracking efficiency.

With the advent and proliferation of deep learning techniques, entirely new tracking paradigms emerged, employing deep learning methods for object tracking. The choice of robust classifiers plays a pivotal role in achieving high-quality tracking results. Compared to traditional object tracking algorithms like optical flow, the class of algorithms involving correlation filtering tends to offer faster tracking, whereas deep learning-based methods typically deliver superior accuracy. Additionally, factors such as scale adaptation and model update mechanisms significantly influence tracking accuracy.

In our comprehensive analysis, we provide a chronological overview of 3D MOT within the context of autonomous driving (refer to Figure 2). It reveals that KCF approaches (above the timeline) and Deep Learning approaches (below the timeline) constitute a substantial portion of the research landscape. In recent years, a prominent trend has been the fusion of point cloud or multi-modal data with deep learning methods. Nevertheless, the KCF method for point cloud data maintains its unique advantages, particularly in terms of speed. It’s worth noting that 3D multi-object tracking using Camera-based methods, while having a smaller representation in current literature, predominantly relies on deep learning techniques for its implementation.

The basic idea of Correlation Filter tracking is to design a filter template and use it to correlate with a target candidate region. The position of the maximum output response is considered to be the target position of the current frame. Kernelized Correlation Filter [40] is an extension of this concept, where kernel functions are introduced in correlation filtering to map the feature space to a higher dimensional space.

In 3D multi-object tracking, the correlation filtering model is divided into four main modules: object detection, motion prediction, association matching, and trajectory management. There are two types of prediction methods commonly used in 3D MOT filtering: the Kalman filter (KF) and the Constant Velocity prediction model (CV). The KF provides smoother predictions when the quality of detection is low, while the CV prediction model is better at handling sudden and unpredictable movements. There are two main approaches commonly used in the association module: IOU-based association and distance-based association. Current matching algorithms take two main forms: one is to formulate the problem as a bipartite graph matching problem using the Hungarian algorithm, and the other is to use a greedy algorithm for matching.

The advantage of the correlation filtering method is that it is faster to perform and the method is relatively simple. However, it relies heavily on the sample information of the last few frames, which accounts for a large portion of the model. If there is inaccurate target localization, occlusion, background disturbance, etc., the fixed learning rate approach will treat these "problematic" samples equally, and the target model will be contaminated and lead to tracking failure.

In this paper, 3D MOT has more algorithms and models related to deep learning, including LSTM, MLP, CNN, RNN, GNN, and Transformer. Graph clustering methods and GNN focus on aggregating information across frames and objects, while LSTM and RNN focus on improving association performance with motion cues. LSTM and GNN have relatively more applications, and the more promising one at present is the research on moving from 2D and 3D SOT to 3D MOT using transformer.

The 3D MOT deep learning framework, similar to the filtering part, is divided into five main steps:

• •

  Object Detection stage: This stage involves analyzing the input frames and using a bounding box to locate the object in the series of frames.

• •

  Motion Prediction stage: This stage involves analyzing the detection to extract appearance, motion, or interaction features.

• •

  Affinity calculation phase: The extracted features are used for the similarity distance calculation between detection pairs.

• •

  Association phase: The similarity/distance metric is utilized in association by providing the same ID to detect corresponding to the same target.

• •

  Life Cycle phase: This stage manages a series of movement states such as the beginning and end of a goal and its creation and disappearance.

Compared with the traditional methods, the 3D MOT tracking algorithm based on deep learning can better adapt to environmental changes, and improve tracking accuracy and tracking speed. The future research direction is to further improve the robustness and accuracy of tracking and can cope with more complex scenes.

In the realm of 3D Multi-Object Tracking (MOT), extending beyond the methods introduced above, there are additional techniques that can be harnessed for tracking tasks. These include radar data processing, spatio-temporal scene integration, and dynamic vision systems, among others.

One intriguing approach in the field of 3D MOT involves the fusion of Camera-Radar or Lidar-Radar for target detection. This approach is particularly captivating because the fusion of a camera with low-cost radar can provide precise long-range measurements. Radar TrackNet [90] is an exemplary study in this domain and it introduced a deep neural network architecture, which employs radar point clouds from multiple time steps to detect road users and compute their tracking information. Nevertheless, it’s noteworthy that existing research frequently segregates object detection and object association for enhancement, potentially resulting in inconsistencies in multi-object tracking. An effective method for enhancing tracking accuracy is one grounded in spatio-temporal consistency. For instance, Spatio-Temporal Object Tracking (SpOT) [91] reformulates tracking as a spatio-temporal issue by representing tracked objects as sequences of time-stamped points and bounding boxes over an extended temporal history. This approach ameliorates the precision of location and motion estimates for tracked objects through learned refinement across the complete sequence of object history. By jointly considering time and space, this representation naturally incorporates fundamental physical principles such as object permanence and temporal consistency. This spatio-temporal tracking framework attains state-of-the-art performance on benchmarks like Waymo and nuScenes. While MOTSLAM [92] is a dynamic visual Simultaneous Localization and Mapping (SLAM) system configured for monocular tracking of both poses and bounding boxes of dynamic objects. It initially conducts multiple object tracking in association with both 2D and 3D bounding box detection to establish initial 3D object representations. Subsequently, neural network-based monocular depth estimation is applied to infer the depth of dynamic features. Finally, camera poses, object poses, and both static and dynamic map points are jointly optimized using a novel bundle adjustment. The experiments on the KITTI dataset demonstrate that this system has achieved the highest performance in terms of both camera ego-motion and object tracking within monocular dynamic SLAM.

In the current phase, detection-based tracking remains the prevailing approach in the field of 3D Multi-Object Tracking (MOT). Substantial enhancements in tracking outcomes have been achieved thanks to recent advancements in 3D object detection algorithms. Nonetheless, the precision of these methods remains limited by the characteristics of the sensors employed. LiDAR is a commonly utilized sensor for data acquisition in 3D MOT; however, the point cloud generated by LiDAR tends to exhibit significant sparsity at extended distances (>80m) due to inherent beam and distance constraints. This inherent sparsity limitation poses a substantial challenge when it comes to accurately detecting and localizing objects. In contrast, image-based detection can deliver high-quality results even over long distances, thanks to the camera’s high pixel resolution. Additionally, current 2D detectors can effectively handle occlusion problems. Therefore, it is crucial to explore 3D MOT methods based on sensor fusion (multi-modal).

Early 3D Multi-Object Tracking (MOT) methods relied on conventional clustering detection techniques, including down-sampling and clustering. However, these methods are susceptible to noise since they rely on the bounding box computation process to obtain detection results. The precision and estimation of bounding box dimensions involve real values with relatively significant errors. With the emergence and growing effectiveness of end-to-end detection methods, the detection component of 3D MOT has gradually transitioned to these techniques.

3D MOT presents various challenges such as target occlusion and target loss over long distances. Another challenge is correlating data between predicted and observed states, i.e., establishing a cost function between them to find a correspondence between observed and predicted state quantities. Most 3D MOT frameworks tend to use the Intersection over Union (IOU) between 3D frames for data correlation, which is indeed a simple and effective method. However, this approach can easily overlook connections between other features, and there may be cases where there are no overlapping regions. Therefore, some data association methods based on Euclidean distance or Mahalanobis distance combined with the Hungarian algorithm or greedy algorithm have been proposed.

From the comprehensive analysis and classification presented above, it’s clear that the primary considerations in current 3D MOT algorithms include:

• •

  Minimizing or eliminating additional training tasks.

• •

  Integration of appearance features, motion features, and geometric information.

• •

  Highlighting the addition and differentiation of background information.

• •

  Reducing or eliminating manually designed associations.

• •

  Ensuring the stability and periodicity of the tracking track.

• •

  Enhancing the security of trackers, such as research on Kalman filter security patches [93].

• •

  Addressing obscure problems [94].

Many studies in the field of 3D Multi-Object Tracking (MOT) predominantly employ network frameworks like CenterPoint [78], PointNet [68], and PointNet++ [55] for 3D object detection. While earlier approaches often treated object detection and tracking as separate tasks, recent research has shifted its focus towards jointly addressing these two tasks, underlining the significance of synchronization. Currently, deep learning technology is increasingly utilized, particularly in the context of end-to-end Graph Neural Networks (GNNs). However, there is an overarching trend towards the convergence of various methodological models and algorithms, including emerging approaches such as transformers. Within the realm of 3D MOT research, the primary objectives revolve around tracking accuracy and operational efficiency, with all research endeavors centered on these two fundamental aspects.

A multitude of driving datasets have been compiled to provide multi-modal sensory data and 3D annotations for 3D object detection and 3D MOT. The most prominent 3D MOT datasets related to autonomous driving currently include KITTI [95], nuScenes [96], Waymo Open Dataset [97], JRDB [98], Argoverse V2 [99], etc.

KITTI: This dataset offers a comprehensive collection of images and point cloud data in BIN format, along with various tools for calibration and annotation. The most significant feature is the txt-terminated annotation data stored in the label’s directory. Serving as a fundamental resource for detection and tracking tasks related to autonomous driving, its latest version, KITTI 360, provides an extensive dataset that includes abundant sensory information and complete annotations. It annotates both static and dynamic elements of 3D scenes using rough bounding primitives, effectively translating this information into the image domain. As a result, it provides dense semantic & instance annotations for both 3D point clouds and 2D images.

nuScenes: In addition to a substantial amount of image and point cloud data in PCD format, detailed annotation information is stored across numerous interlinked JSON files. nuScenes has gained widespread acceptance in conference papers and journal publications, establishing itself alongside KITTI as one of the standard mainstream datasets within object detection and tracking research domains. Figure 4 illustrates a typical application scenario utilizing the nuScenes dataset.

Waymo Open Dataset: Waymo is a relatively recent addition to the available datasets. Despite its novelty, it has already been extensively used for research and evaluation purposes in several new studies.

JRDB: The JRDB dataset is an innovative large-scale 2D and 3D dataset that has recently gained attention due to its utilization in the development, training, and evaluation processes by JRMOT researchers.

Argoverse2 (AV2): Primarily focused on the perception and prediction aspects within the context of autonomous driving systems, AV2 comprises 1000 annotated multi-modal data sequences. These sequences feature high-resolution images captured by seven surround-view cameras, along with imagery output from two stereo cameras. The dataset is the largest Lidar sensor dataset ever created and supports self-supervised learning and emerging point cloud prediction tasks.

In traditional 2D MOT, two key metrics are used to evaluate performance: the Multiple Object Tracking Accuracy (MOTA) and the Multiple Object Tracking Precision (MOTP). MOTA measures the accuracy of the tracking system, while MOTP assesses the precision, which is the closeness of the obtained value to the true value. High scores in both MOTA and MOTP indicate superior performance of the tracking system. Therefore, an ideal tracker would aim to achieve high values in both these metrics.

$$MOTA=1-\frac{\sum_{t}\left(\mathrm{FN}_{t}+\mathrm{FP}_{t}+\mathrm{IDSW}_{t}\right)}{\sum_{t}\mathrm{GT}_{t}}$$

(1)

The MOTA is a performance metric that evaluates the ability of a system to detect objects and maintain their trajectories. It takes into account three types of errors: False Negatives (FN), False Positives (FP), and Identity Switches (ID Switch). The denominator, GT, represents the number of ground truth objects. An interesting aspect of the MOTA formula is that it calculates a weighted average across all frames, rather than averaging the results of each frame separately. This means that the performance of the detector on each individual frame contributes to the overall MOTA score. MOTA is primarily concerned with the performance of the detector. If the detector has a low rate of False Negatives and Identity Switches, and a high rate of True Positives, then the MOTA score will be higher. This underscores the importance of accurate detection in achieving a high MOTA score.

$$MOTP=\frac{\sum_{t,i}d_{t,i}}{\sum_{t}c_{t}}$$

(2)

The MOTP is a metric that measures the localization accuracy, which is primarily a reflection of the detector’s performance rather than the tracker’s. The denominator in the MOTP calculation is the number of successful matches for the corresponding frame. The numerator measures the d-dimensional distance between the detected target and the corresponding ground truth. This distance could be a euclidean distance (where smaller is better) or an overlap rate (where larger is better).

In 3D multi-object tracking, starting from the baseline model AB3DMOT, three important integrated metrics are introduced: Averaged Multi-Object Tracking Accuracy (AMOTA), Averaged Multi-Object Tracking Precision (AMOTP), and sAMOTA. sAMOTA is a scaled accuracy metric, representing a scaled average MOTA. The purpose of sAMOTA is to scale the value of the integrated metric AMOTA to range from 0% to 100%. This allows for a more intuitive understanding of the tracker’s performance.

$$AMOTA=\frac{1}{n-1}\sum_{r\in\left\{\frac{1}{n-1},\frac{2}{n-1}\ldots 1\right\}}\operatorname{MOTA}^{\prime}$$

(3)

$$AMOTP=\frac{1}{n-1}\sum_{r\in\left\{\frac{1}{n-1},\frac{2}{n-1}\ldots 1\right\}}\frac{\sum_{t,i}d_{t,i}}{\sum_{t}c_{t}}$$

(4)

A comprehensive compilation of evaluation metrics related to 3D Multi-Object Tracking (MOT) is still in progress. After reviewing numerous articles and collecting metrics from various experimental contents, we have assembled the most recent set of 3D MOT evaluation metrics (refer to Table 1)

Among these evaluation metrics, two key measures are particularly noteworthy: False Positives (FP) and False Negatives (FN). FP refers to points that are incorrectly predicted, while FN, also known as Miss, refers to ground truth points that are not matched.

In addition to these measures, Frames per Second (FPS) and ID-Switches (IDS) are two other important metrics used for evaluating MOT. FPS indicates the rate at which video frames are tracked; a higher rate signifies greater efficiency. This is especially noticeable in point cloud approaches where baseline models using filtering can achieve rates of over 200. IDS is another crucial metric for evaluating MOT, reflecting the number of times that predicted IDs do not match real IDs; lower numbers indicate better performance with zero being optimal. Our analysis suggests that mismatches may occur due to incorrect association or early termination. These insights underline the complexity and challenges inherent in MOT.

After extensive screening and analysis, we have compiled experimental measurement tables for the categories of Camera, Point Clouds, and Multi-modality. These are presented in Table 2. Please pay special attention to the parts of the table highlighted in bold, as these are key areas that need to be summarized.

3D MOT is a relatively new research field, with the majority of results emerging after the second half of 2020, particularly from 2021 to 2023 (as depicted in Figure 5a) based on Table 2. In terms of publication venues, arXiv papers account for over two-fifths. The predominant datasets used are KITTI and nuScenes, with KITTI dataset being extensively classified (including lane lines, optical flow, scene flow, depth maps) and nuScenes dataset being more specialized. Waymo stands out due to its close relevance to autonomous driving along with newer and more specialized annotation information; whereas JRDB is the most recent dataset that offers exceptional annotations but not exclusively focused on autonomous driving.

From the data in Table 2, it can be concluded that:

• •

  It should be noted that formal publications on 3D MOT are still relatively scarce as it is a pioneering research topic developed only in the last three or four years; many recent articles can be found on arXiv.

• •

  The most commonly utilized datasets in the field of autonomous driving include KITTI and nuScenes, while more recent ones consist of JRDB and Waymo datasets.

• •

  The current research on 3D MOT primarily focuses on vehicles and pedestrians, with a particular emphasis on the vehicle category.

• •

  The existing challenges in 3D MOT research encompass occlusion, crowding, small targets, and multi-modality issues.

• •

  Regarding specific metrics, AB3DMOT, GNN3DMOT, EagerMOT, PTP, 3DMODT, and InterTrack exhibit higher sAMOTA values (as depicted in Figure 5b).

• •

  sAMOTA, AMOTA, and AMOTP are the latest evaluation metrics for 3D MOT that serve as better indicators of model framework performance.

• •

  Point clouds combined with filtering methods offer relatively efficient results while multi-modal approaches combined with deep learning methods provide greater accuracy. The latter technique is a novel development direction showing promising potential.

• •

  Some notable baseline models or methods for 3D MOT include AB3DMOT, GNN3DMOT, P3DMMMOT, EagerMOT, InterTrack, DeepFusionMOT, 3DMODT, and PTP.

From the above analysis, we can draw further insightful conclusions.

Camera-based 3D MOT approaches have garnered attention due to their affordability and rich contextual information. Popular methods include monocular, pseudo-Lidar, multi-camera, and multi-angle approaches. These primarily utilize online efficient joint detection and tracking models, multi-camera tracking datasets, Convolutional Neural Networks, and other deep learning techniques. They offer the benefits of low cost and a relatively simple model structure. However, since images lack the depth information provided by Lidar, ensuring accurate and reliable position and motion cues is challenging. Beyond basic target detection, 3D MOT research concentrates on similarity calculation and data association. This leads to issues such as low MOTA and AMOTA metrics, severe occlusion, and a relative inability to handle cluttered and crowded environments. Despite these challenges, given their low cost and complexity, camera-based 3D MOT approaches remain a significant research direction.

KCF approaches are notable for their low complexity and relatively simple model structure. Experimental results show that not only Lidar-based but also multi-modal-based approaches have high FPS, indicating faster tracking speed. They also have extremely low IDS, indicating a low number of mismatches. Some baseline models (e.g., AB3DMO, FlowMOT, EagerMO, P3DMMMO, DeepFusionMOT) perform exceptionally well. They typically employ effective filters to combine point-wise motion information with traditional matching algorithms, enhancing motion prediction robustness. On the other hand, multi-modality-based KCF approaches focus on more critical aspects such as interactive feature fusion between multi-scale features, effective association strategies, and efficient management of the tracking queue. The fundamental goal of these approaches is to leverage sensor fusion patterns to enhance autonomous driving’s accuracy and reliability.

Many traditional autonomy systems use Kalman Filters for multi-object tracking due to their computational efficiency. However, these methods often rely on hand-engineered association and fail to generalize to crowded scenes and multi-sensor modalities. This often results in poor state estimates leading to inaccurate predictions. Deep learning approaches can address these problems relatively well, although their computational efficiency is not as good as the KFC approach and the model structure is more complex. However, they offer advantages such as high prediction accuracy, stable tracking trajectory, and proficiency in handling occlusion and congestion scenarios. Experimental results reveal that the baseline model GNN3DMOT of multi-modal-based deep learning approaches performs excellently. Many 3D MOT research methods related to GNNs are derived from its model structure and methods. Other models and methods such as InterTrack, PTP, 3DMODT, YONTD-MOT also perform exceptionally well. They primarily focus on end-to-end graph neural network design, labeling of new datasets, heuristic matching, integrated detection and tracking, and tracking track stability improvement. These methods can mitigate the impact of false detections in tracking, reduce unnecessary extra training, enable multi-object tracking in multi-category scenes, and achieve robust tracking results. Recently deep learning methods for 3D MOT have increasingly focused on applications in the direction of GNNs, Transformers, and multi-modal applications.

While current research predominantly focuses on 2D MOT approaches encompassing image localization and tracking across multiple cameras, the progress in 3D MOT remains relatively limited, with many methods being derived from upgraded versions of their 2D counterparts. However, leveraging 3D MOT techniques can offer more accurate position and size estimation as well as effective occlusion handling for advanced computer vision tasks. Nevertheless, it necessitates stricter equipment requirements and scene layouts while also entailing higher costs and energy consumption compared to traditional 2D MOT systems.

Contemporary MOT systems typically adhere to the ’tracking by detection’ paradigm wherein a detector is followed by an association method for linking detection to tracks. Although combining motion and appearance features has been extensively explored in tracking to enhance robustness against occlusions and other challenges, this often leads to complex and computationally intensive implementations that require further improvements for effectively handling small targets or crowded scenes. Additionally, integrating both 2D and 3D modalities in the context of MOT poses several challenges related to feature alignment, fusion difficulties between different modalities, algorithmic efficiency limitations, as well as high hardware requirements. Research on multi-modal-based 3D MOT approaches involving camera-radar or radar-lidar combinations still remains relatively scarce.

Through the previous analysis and summary, we can conclude that 3D object detection plays a fundamental and central role in 3D multi-object tracking. Multi-modal fusion is crucial for achieving accurate 3D object detection and tracking. Additionally, filtering algorithms or deep learning methods are essential for optimizing models in this context. Therefore, it is important to explore potential future research directions.

Recently, the incorporation of deep learning and multi-modality into the field of 3D MOT has emerged as a novel research direction with promising prospects for future investigations, including the following aspects.

• •

  Multiple classes object tracking (MCOT): In practical applications, there is generally a need for simultaneous tracking of multiple categories of targets (e.g. sudden animals, balloons, special objects, etc.), especially in the case of dense scenes.

• •

  Multi-Camera and multi-object tracking (MMOT): The problem of cross-camera data association is also an important research direction.

• •

  Multi-object tracking in combination with other computer vision tasks: The combinations that can be considered include but are not limited to object segmentation, pedestrian re-identification, human pose estimation, and motion recognition, etc.

• •

  Framework based on tracking and detection joint: Future methods based on tracking and detection joint will be more applied to the field of 3D MOT.

Regarding other details, including the following aspects.From the perspective of methods, deep learning is considered the primary direction for future 3D MOT development, particularly through the integrated application of models and methods such as LSTM, GNN, R-CNN, and scene flow. Many researchers are also focused on enhancing current schemes by designing novel loss functions, network architectures, transformer attention mechanisms, etc. In terms of dataset application, object tracking data annotation is a time-consuming and labor-intensive task; however, a comprehensive annotated dataset can significantly contribute to 3D MOT research and has led to increased recognition of annotators as highly skilled professionals who command higher remuneration. Another crucial research direction involves striking a balance between FPS and metrics like MOTA, AMOTA, and sAMOTA while considering both efficiency and accuracy. Additionally, exploring more efficient multi-modal combined deep learning methods along with conducting comparative analyses across multiple datasets through feature ablation experiments and refining management strategies for 3D MOT tracking trajectories represent meaningful avenues for further investigation.