: Towards Generic 3D Single Object Tracking in the Wild

As one of the most crucial problems in 3D computer vision, 3D single object tracking (SOT) aims to localize the desired target with a sequence of 3D bounding boxes, given its state in the first frame. Due to its key roles in many applications, such as intelligent vehicles, mobile robotics, navigation, etc, 3D object tracking has gained extensive attention in the past decade with many models proposed (e.g., [2, 3, 12, 28, 39]).

Current research mainly focuses on the point cloud (PC)-based 3D tracking. Relying on popular autonomous driving benchmarks (e.g., KITTI [11] and NuScenes [5]), numerous deep 3D trackers have been proposed and demonstrated state-of-the-art results (e.g., [36, 37, 25, 34]). Despite such progress, further development of generic 3D SOT is heavily restricted by currently adopted benchmarks due to several reasons: (1) limited object classes. To achieve general tracking capacity, a 3D tracker is expected to learn with sequences from a large set of categories during training. However, existing datasets for 3D SOT (e.g., [11, 5]), specially designed for autonomous driving, comprise very few available categories (e.g., 8 in [11] and 23 in [5]) for tracking, making them inadequate for designing generic 3D trackers. (2) constrained scenarios. In applications, a general tracker should be able to localize the target object under various scenarios, which requires it to be trained and assessed with sequences collected from diverse environments. Yet current datasets, due to their own specific aims, only offer sequences from the traffic scenario and thus are unsuitable for general tracking. (3) restricted degrees of freedom (DoF). For generic 3D tracking, a tracker needs to handle objects with arbitrary pose and size, often described with 9DoF consisting of 6D pose and 3D size. Nonetheless, currently used datasets [11, 5] comprise only targets of 7DoF, including 4D pose and 3D size, and thus are undesirable for developing general trackers locating arbitrary-pose objects.

It is worth noting that, besides the PC-based 3D SOT, the above autonomous driving datasets (e.g., [11, 5]) can also be used for developing multi-modal, i.e., RGB-PC, tracking by integrating point clouds and RGB images. Nevertheless, the aforementioned issues still exist, and therefore, limit the further development of generic 3D object tracking.

In addition to PC-based single- or multi-modal solutions, another direction that is more affordable is to leverage RGB and depth information for 3D tracking. For such a goal, a recent dataset [38] has been introduced by collecting RGB-D sequences from diverse categories and annotating each one with 9DoF 3D boxes. However, it is limited by its relatively small scale. In order to effectively train and reliably assess deep 3D trackers, it is desirable to have plenty of sequences in a dataset. Nonetheless in [38], there is a total of only 300 sequences with 36K frames, which might be insufficient for large-scale learning and evaluation of deep 3D trackers.

Specifically, our GSOT3D consists of 620 sequences and provides more than 123K frames in total. In order to ensure the diversity of GSOT3D, these sequences are carefully collected from a wide selection of 54 object classes from various environments. For each sequence in GSOT3D, multiple modalities, including the point cloud (PC), RGB image, and depth, are offered using different sensors (see examples in Fig. 1). This allows GSOT3D to support different 3D tracking tasks, comprising the single-modal 3D SOT on PC and multi-modal 3D SOT on RGB-PC or RGB-D, and therefore broadens the research directions in 3D tracking. For precise dense annotations, all the sequences in GSOT3D are manually labeled using 9DoF 3D bounding boxes with multiple rounds of inspection and refinement. To our best knowledge, GSOT3D is, to date, the largest benchmark dedicated to generic 3D object tracking. Besides, it is the first benchmark, to date, that simultaneously supports different single- and multi-modal 3D SOT tasks.

Compared with existing benchmarks (e.g., [11, 5]) with a few object classes for 3D SOT on PC and RGB-PC in traffic scene, GSOT3D is more diverse by containing 54 categories and various scenarios, making it more favorable for generic 3D tracking. Moreover, compared to [38] consisting of 300 sequences with 36K frames for RGB-D 3D tracking, GSOT3D is larger by providing 620 sequences (2$\times$ larger) with 123K frames (3$\times$ larger), and hence more desirable for large-scale learning and evaluation of deep 3D tracking.

In order to understand how existing 3D trackers perform and to provide comparisons for future research, we assess 8 representative PC-based tracking methods. Please note that, compared to 2D generic object tracking, there are not many open-sourced 3D trackers and most methods are PC-based. For this reason, we finally include 8 PC-based trackers, that are representative and provide executable implementations, for evaluation. Our evaluation reveals that, not surprisingly, all current models degrade severely on the more challenging GSOT3D, which demonstrates the difficulty in achieving generic 3D tracking in the real-world, and more efforts are needed for future improvements.

Moreover, to facilitate research on GSOT3D, we present a simple but effective generic 3D tracker, dubbed PROT3D, for class-agnostic 3D tracking on point clouds. The core of PROT3D is a progressive spatial-temporal architecture containing multiple stages. In each stage, target localization is performed by spatial-temporal matching with Transformer, and the result is applied to refine search region feature. The refined search region feature from one stage is forwarded to next stage for further improvements, and tracking result is generated after the final stage. This way, PROT3D gradually learns more discriminative features via progressive feature refinement, making it capable of handling more complex scenarios for generic tracking. It is worth noticing, unlike current trackers predicting a 7DoF box, our PROT3D produces a 9DoF box for more precise tracking. Despite its simplicity, PROT3D outperforms all other methods, and expects to provide a reference for future research.

In summary, our contributions are as follows: ♠ We propose a new benchmark GSOT3D comprising 620 sequences with more than 123K frames to facilitate 3D object tracking; ♥ GSOT3D provides multiple modalities to each sequence, making it a versatile platform for various research directions in 3D tracking; ♣ We evaluate eight representative trackers to understand their performance and to offer comparisons to future research; ♠ We present a simple yet effective tracker, PROT3D, to encourage future research on GSOT3D.

Benchmarks for 3D Single Object Tracking. Datasets are crucial for 3D single object tracking by providing platforms for training and assessment. Currently, the popular datasets, particularly for 3D tracking on point cloud, are mainly borrowed from the autonomous driving benchmarks, including KITTI [11] and NuScenes [5]. Specifically, KITTI comprises 21 sequences with 15K frames, and each one is offered with point clouds and RGB images. Similar to KITTI but with a larger size, NuScenes comprises 1,000 sequences with 40K frames. Since KITTI and NuScenes are originally designed for autonomous driving, they usually need appropriate conversions before being used for 3D SOT. Besides KITTI and NuScenes for point cloud-related 3D SOT, the work of [38] recently proposes a new benchmark, named Track-it-in-3D, dedicated to RGB-D-based 3D object tracking. It contains 300 sequences with 36K frames, collected from 44 classes. Each sequence is annotated with 9DoF 3D boxes for more precise generic 3D object tracking.

Despite the above benchmarks, the further development of 3D SOT remains constrained by the limitations discussed earlier, which motivates our GSOT3D in this work, a versatile dataset dedicated to different generic 3D tracking tasks. Tab. 1 compares our GSOT3D with other datasets in detail.

Besides 3D tracking on point clouds, another direction is to leverage RGB and depth information for 3D SOT. The work of [3] introduces a part-based 3D tracker using sparse learning. In [38], a Siamese network is proposed to fuse the RGB and depth information for RGB-D 3D tracking.

Sharing a similar goal with current large-scale 2D tracking benchmarks, GSOT3D aims at providing sufficient sequences from rich classes for generic 3D tracking. It is worthy to note that, compared to current large-scale 2D tracking benchmarks (e.g., [8, 14, 31, 27, 24]) with over a thousand or tens of thousands videos, GSOT3D is relatively smaller due to the extreme difficulty in collecting sequences and annotating them using the 9DoF bounding boxes. That being said, GSOT3D to date is still the largest dataset that is dedicated to generic 3D single object tracking.

We present a simple yet effective tracker, PROT3D, for 3D-SOT${}_{\text{PC}}$, as there are more available trackers for SOT${}_{\text{PC}}$, and we will explore 3D-SOT${}_{\text{RGB-PC}}$ and 3D-SOT${}_{\text{RGB-D}}$ in the future. The key is to progressively refine search region feature with multiple cascaded stages, as in Fig. 5. Each stage performs spatial-temporal target localization, and the result is used to augment the search region feature in the next stage.

Similar to [28], PROT3D treats 3D tracking as a matching problem. Inspired by [37], we leverage target cues from historical frames for robust performance. More specifically, given point cloud $\textbf{p}_{t}$ at frame $t$, we apply information from previous $K$ frames $\{\textbf{p}_{j}\}_{j=t-K}^{t-1}$ for tracking. We first extract their features through a shared backbone $\Phi(\cdot)$ as follows,

$$\textbf{x}_{t}^{1}=\Phi(\textbf{p}_{t})\;\;\;\;\;\textbf{z}_{j}=\Phi(\textbf{p}_{j})\;\;j=t-K,\cdots,t-1$$

(1)

where $\textbf{x}_{t}^{1}$ represents the feature of $\textbf{p}_{t}$ and $\textbf{z}_{j}$ is the feature of $\textbf{p}_{j}$ ($j=t-K,\cdots,t-1$). Then, we concatenate all features from historical frames via $\textbf{H}_{t-1}=\text{concat}(\textbf{z}_{t-K},\cdots,\textbf{z}_{t-1})$ to obtain memory feature $\textbf{H}_{t-1}$ for frame $t$. After that, $\textbf{H}_{t-1}$ and $\textbf{x}_{t}^{1}$ are sent to the progressive spatial-temporal network with multiple stages, with each performing localization.

Specifically, for stage $i$, it receives $\textbf{H}_{t-1}$ and $\textbf{x}_{t}^{i}$ as inputs. Then, a spatial-temporal Transformer is utilized to fuse the memory $\textbf{H}_{t-1}$ into $\textbf{x}_{t}^{i}$, as follows

$$\textbf{F}_{t}^{i}=\text{SPT}(\textbf{x}_{t}^{i},\textbf{H}_{t-1})$$

(2)

where $\textbf{F}_{t}^{i}$ is the feature after fusion. $\text{SPT}(\cdot,\cdot)$ represents the spatial-temporal Transformer, and comprises $L$ ($L$ is set to 2) layers. Similar to [37], each layer consists of cross- and self-attention operations [30] and a feed-forward network, as displayed in Fig. 6. After that, $\textbf{F}_{t}^{i}$ is forwarded to a multi-layer perceptron (MLP) for localization, as follows

$$R_{t}^{i}=\text{MLP}(\textbf{F}_{t}^{i})$$

(3)

where $R_{t}^{i}=[C_{t}^{i},M_{t}^{i},S_{t}^{i}]$ is the localization result, with $C_{t}^{i}$ potential target center, $M_{t}^{i}$ targetness mask, and $S_{t}^{i}$ proposal scores. Then, we perform Farthest Point Sampling (FPS) on $C_{t}^{i}$ to refine point clouds, as follows

$$\bar{C}_{t}^{i}=\text{FPS}(C_{t}^{i})$$

(4)

where $\bar{C}_{t}^{i}$ is sampled points. After FPS, the $\bar{C}_{t}^{i}$ and $M_{t}^{i}$ are fed to a feature transformation block (FTB) and the resulted feature is combined with the score information to generate the refined search region feature $\textbf{x}_{t}^{i+1}$, mathematically described as follows,

$$\textbf{x}_{t}^{i+1}=\text{FTB}(\bar{C}_{t}^{i},M_{t}^{i})+\text{Conv1D}(S_{t}^{i})$$

(5)

where $\text{FTB}(\cdot,\cdot)$ is feature transformation block, borrowed from [37], and contains point-to-reference and a 3D convolution operation (see supplementary material for details). $\text{Conv1D}(\cdot)$ is 1D convolution to embed $S_{t}^{i}$ to score feature.

Please note, $\textbf{x}_{t}^{i+1}$ in Eq. (5) is generated by encoding target information $C_{t}^{i}$, $M_{t}^{i}$, and $S_{t}^{i}$, obtained via localization, and thus more discriminative for distinguishing target from background. For further refinement, $\textbf{x}_{t}^{i+1}$ is fed to the next stage ($i+1$), forming a progressive cascade architecture. This way, the search region feature can be gradually refined with more target cues, benefiting the final localization.

After the last $N^{\text{th}}$ stage, the generated $\textbf{x}_{t}^{N+1}$ is employed for final 9DoF target localization via MLP, as follows,

$$\mathcal{R}_{t}=\text{MLP}(\textbf{x}_{t}^{N+1})$$

(6)

where $\mathcal{R}_{t}=[\mathcal{B}_{t},\mathcal{S}_{t}]\in\mathbb{R}^{D\times 10}$, with $\mathcal{B}_{t}\in\mathbb{R}^{D\times 9}$ the 9DoF box parameters, $\mathcal{S}_{t}\in\mathbb{R}^{D\times 1}$ the targetness scores and $D$ the number of points in $\textbf{x}_{t}^{N+1}$. Finally, the tracking result $b_{t}$ is determined as follows,

$$b_{t}=\mathcal{B}_{t}(h)\;\;\;\text{where}\;\;h=\operatorname*{arg\,max}_{d=1,\cdots,D}\mathcal{S}(d)$$

(7)

where $b_{t}=(x_{t}^{*},y_{t}^{*},x_{t}^{*},\alpha_{t}^{*},\beta_{t}^{*},\gamma_{t}^{*},l_{t}^{*},h_{t}^{*},w_{t}^{*})$, predicting the translation offset $(x_{t}^{*},y_{t}^{*},x_{t}^{*})$ of the center point and angle offset $(\alpha_{t}^{*},\beta_{t}^{*},\gamma_{t}^{*})$ and size offset $(l_{t}^{*},h_{t}^{*},w_{t}^{*})$ of target box from frame ($t-1$) to frame $t$.

Please note, PROT3D is a class-agnostic 3D tracker that is able to track the target object of any categories. The loss of PROT3D is computed with loss function for final target estimation. Due to space limitation, please refer to our supplementary material for details of the loss function.

Please note again, we primary focus on experiments for 3D-SOT${}_{\text{PC}}$ trackers, as most currently open-sourced 3D trackers with available implementations belong to 3D-SOT${}_{\text{PC}}$.

In this paper, we introduce GSOT3D, a new benchmark for generic 3D SOT. It contains 620 multimodal sequences with over 123K frames, and supports different 3D single object tracking tasks. To the best of our knowledge, GSOT3D is the largest benchmark to date dedicated to 3D SOT. Besides, we assess several representative trackers on GSOT3D to understand their performance and to offer comparison for future research. Furthermore, we present a simple yet effective progressive tracker PROT3D and obtain state-of-the-art result. We believe that, our benchmark, evaluation, and new baseline will inspire more research towards generic 3D object tracking and facilitate its real-world applications.

Despite contributions, there exist a few limitations. First, the experiments are mainly focused on the 3D-SOT${}_{\text{PC}}$, and study on 3D-SOT${}_{\text{RGB-PC}}$ and 3D-SOT${}_{\text{RGB-D}}$ is not provided. Second, the sequences in GSOT3D are relatively short, and not suitable for long-term tracking. Given 3D-SOT${}_{\text{PC}}$ is the current research focus and our major goal is to offer a new benchmark for generic tracking, we leave study of more 3D tracking tasks and long-term 3D tracking to the future work.

• S1   Mobile Robotic Platform
  In this section, we demonstrate more details of our mobile robotic platform used for multimodal data collection.

• S2   Annotation Tool
  We display more details of the annotation tool in labeling sequences with 9DoF 3D bounding boxes and its reliability analysis for high-quality annotation.

• S3   More Statistics
  We demonstrate more statistics on GSOT3D regarding sequence length and per-category point density .

• S4   Evaluation Metrics and 3D IoU
  We demonstrate detailed process on how to calculate the evaluation metrics and 3D IoU.

• S5   Formulation of Different 3D SOT Tasks
  We describe the formulation of different 3D SOT tasks.

• S6   Details of Feature Transformation Block
  We present the details of the feature transformation block adopted in our PROT3D.

• S7   Loss Function
  We present details of the loss function to train PROT3D.

• S8   Summary of Evaluated Trackers
  We offer a summary for trackers assessed on GOST3D.

• S9  Qualitative Results
  We offer more qualitative analysis of our PROT3D and its comparison to other trackers on GSOT3D.

• S10   Maintenance and Responsible Usage of GSOT3D for Research
  We discuss the maintenance and responsible usage of our proposed GSOT3D for research.

To collect multimodal data for GSOT3D, we build a mobile robotic platform based on Clearpath Husky A200. Multiple sensors, including a 64-beam LiDAR, an RGB camera and a depth camera, are deployed on the platform with careful calibration using the tool from [6]. Fig. 8 shows the picture of our mobile robotic platform for multimodal data acquisition in developing GSOT3D, and the specific configuration of sensors and robot chassis are listed in Tab. 8.

For data labeling, we use the annotation tool provided by a company. Fig. 9 shows the interface for 3D bounding box annotation. Specifically, for each point cloud frame, we perform initial annotation of the target object by drawing a 3D bounding box in the annotation region (note, this region can be flexibly zoomed in or out). Then, the initial 3D bounding box is refined by adjusting the 2D boxes on each projected view on XY, XZ, and YZ planes. In the annotation tool, a preview of the 3D box in the RGB image is provided for visual inspection of the refined box. By doing this, we can ensure the obtained annotation is reliable. Please note that, all the annotations from the labeler will be inspected careful by the experts (see this part in the main text) and further refined (by the same labeler) if necessary for high quality.

In this section, we demonstrate more statistics of GSOT3D. In specific, Fig. 10 (a) shows distribution of sequence length on GSOT3D. Although the average length of GSOT3D is 198 frames, there exist several relatively longer ones with sequence length larger than 600 frames, which can be used for analyzing trackers on relatively longer sequences. Besides, Fig. 10 (b) demonstrates the average number of points for each category. We can clearly see that, the categories of bus, car, and van on average contain the most number of points, while the categories of dog and mineral_water consist of the least number of points. We hope this statistics can help readers better understand our GSOT3D.

Inspired by [14], we utilize mean Average Overlap (mAO) and mean Success Rate (mSR) to measure different tracking algorithms. Specifically, mAO is calculated by averaging the class-wise overlaps, i.e., 3D Intersection over Union (3D IoU, which will be detailed later), between all tracking results and the groundtruth, and mSR computes the class-wise percent of successful frames in which 3D IoU is larger than a threshold. mAO and mSR can be obtained as follows,

	mAO	$\displaystyle=\frac{1}{C}\sum_{c=1}^{C}\left(\frac{1}{\left|S_{c}\right|}\sum_{i\in S_{c}}\text{AO}_{i}\right)$		(8)
	mSR	$\displaystyle=\frac{1}{C}\sum_{c=1}^{C}\left(\frac{1}{\left|S_{c}\right|}\sum_{i\in S_{c}}\text{SR}_{i}\right)$

where $C$ is the total number of object categories in GSOT3D, $S_{c}$ the set of all sequences belonging to category $c$. $\text{AO}_{i}$ represents the Average Overlap (AO) for the $i^{\text{th}}$ sequence in $S_{c}$, and $\text{SR}_{i}$ denotes Success Rate (SR). $\text{mSR}_{50}$ and $\text{mSR}_{75}$ refers to mSR with thresholds of 0.5 and 0.75, respectively, when computing success rate.

Therefore, for non-symmetric targets, we use method as in KITTI [11] for 3D IoU calculation, while for symmetric targets, we use strategy as in [1, 4] for 3D IoU computation.

GSOT3D is a unique platform to broaden research direction in 3D SOT by supporting different tasks, comprising single-modal 3D object tracking, i.e., 3D SOT on Point Cloud (PC) (3D-SOT${}_{\text{PC}}$), and multi-modal 3D tracking, i.e., 3D SOT on RGB-PC (3D-SOT${}_{\text{RGB-PC}}$) or RGB-Depth (3D-SOT${}_{\text{RGB-D}}$).

3D-SOT${}_{\text{PC}}$ aims at locating the target object on the point clouds. Given the PC sequence and the initial 9DoF 3D target box, the goal is to estimate a set of 3D bounding boxes to represent the target positions in the sequence. This process can be formulated as follows,

$$\{b_{i}\}_{i=2}^{N}\leftarrow\mathcal{T}_{\text{PC}}(\{\textbf{p}_{i}\}_{i=1}^{N},b_{1})$$

(9)

where $b_{i}=(x_{i},y_{i},z_{i},w_{i},h_{i},l_{i},\alpha_{i},\beta_{i},\gamma_{i})$ is the 9DoF 3D box in frame $i$ $(1\leq i\leq N)$, with $(x_{i},y_{i},z_{i})$, $(w_{i},h_{i},l_{i})$, and $(\alpha_{i},\beta_{i},\gamma_{i})$ the target position, scale, and rotation angle. $b_{1}$ is given in the first frame and $\{b_{i}\}_{i=2}^{N}$ are predicted by the tracker $\mathcal{T}_{\text{PC}}$. $\{\textbf{p}_{i}\}_{i=1}^{N}$ represent the PC sequence, and $N$ is the number of frames in the sequence.

Different from 3D-SOT${}_{\text{PC}}$, 3D-SOT${}_{\text{RGB-PC}}$ integrates the point clouds and RGB images for to locate target, aiming to improve 3D tracking using appearance information. It can be formulated as follows,

$$\{b_{i}\}_{i=2}^{N}\leftarrow\mathcal{T}_{\text{RGB-PC}}(\{\textbf{p}_{i}\}_{i=1}^{N},\{I_{i}\}_{i=1}^{N},b_{1})$$

(10)

where $b_{1}$ is the initial 9DoF 3D box, $\{b_{i}\}_{i=2}^{N}$ the predicted results by the tracker $\mathcal{T}_{\text{RGB-PC}}$, $\{\textbf{p}_{i}\}_{i=1}^{N}$ and $\{I_{i}\}_{i=1}^{N}$ the PC and RGB image sequences, respectively.

Different than using PC, 3D-SOT${}_{\text{RGB-D}}$ exploits a more economic way using RGB and depth images for 3D tracking, and can be formulated as follows,

$$\{b_{i}\}_{i=2}^{N}\leftarrow\mathcal{T}_{\text{RGB-D}}(\{D_{i}\}_{i=1}^{N},\{I_{i}\}_{i=1}^{N},b_{1})$$

(11)

where $\mathcal{T}_{\text{RGB-D}}$ denotes the 3D tracker, $\{D_{i}\}_{i=1}^{N}$ are the depth image sequence, and all others are the same as in Eq. (10).

By supporting different tracking tasks, GSOT3D expects to expand research directions in 3D SOT.

Fig. 11 displays feature transformation block (FTB) applied in each stage of our PROT3D. The feature transformation block is borrowed from [37] for its effectiveness. In specific, we first send the targetness mask $M_{t}^{i}$ and the point feature $\bar{C}_{t}^{i}$ to the Point-to-Reference operation, which is composed of a concatenation operation, a MLP, and an EdgeConv layer [32] for feature aggregation, as follows,

$$\begin{split}\hat{g}_{t}^{i}&=\text{Point-to-Reference}(\bar{C}_{t}^{i},M_{t}^{i})\\ &=\text{EdgeConv}(\text{MLP}(\text{Concatenate}(\bar{C}_{t}^{i},M_{t}^{i})))\end{split}$$

(12)

After this, the resulted feature $\hat{g}_{t}^{i}$ is fed into a 3D CNN network to generate point-wise feature. Fig. 11 illustrates FTB. For more details, please kindly refer to [37].

In this section, we present details regarding the loss function for training PROT3D. Specifically, after the $N^{\text{th}}$ stage, the final feature $\textbf{x}_{t}^{N+1}$ is sent to the MLP layer for prediction. Similar to previous work [37], we use the following loss function for end-to-end training,

$$\mathcal{L}_{\text{total}}=\lambda_{\text{m}}\mathcal{L}_{\text{m}}+\lambda_{\text{c}}\mathcal{L}_{\text{c}}+\lambda_{\text{p}}\mathcal{L}_{\text{p}}+\lambda_{\text{s}}\mathcal{L}_{\text{s}}+\mathcal{L}_{\text{bbox}}$$

(13)

where $\mathcal{L}_{\text{total}}$ represents the total training loss, $\mathcal{L}_{\text{m}}$ the standard cross-entropy loss to supervise the targetness mask, $\mathcal{L}_{\text{c}}$ the mean square loss to supervise the target center, $\mathcal{L}_{\text{p}}$ the cross-entropy loss to supervise proposal score, $\mathcal{L}_{\text{s}}$ the cross-entropy loss to supervise the targetness score $\mathcal{S}_{\text{t}}$, and $\mathcal{L}_{\text{bbox}}$ the smooth-L₁ loss to supervise the 9DoF box $\mathcal{B}_{t}$ (including 3D center offset and 6D pose offset of size and angle). $\lambda_{\text{m}}$, $\lambda_{\text{c}}$, $\lambda_{\text{p}}$, $\lambda_{\text{s}}$ are hyper-parameters to balance different losses and are set to 0.2, 10.0, 1.0, and 1.0, respectively.

Our code will be publicly released, and more details can be found in our implementation.

To understand how existing trackers perform on GSOT3D and to provide comparison for future research, we assess eight representative trackers, including P2B [28], BAT [39], PTT [29], M2-Track [40], CXTrack [36], MBPTrack [37], SeqTrack3D [21], and M3SOT [22]. Please note that, these evaluated 3D trackers are point cloud-based, as almost all current 3D object trackers that share their implementations belong to this category. Tab. 9 summarizes these trackers.

In this section, we show qualitative results of different trackers and our PROT3D on GSOT3D in Fig. 12. From Fig. 12, we can see that, existing state-of-the-art trackers such as M2-Track, MBPTrack fail to accurately localize the target object in challenging scenarios with frequent occlusions and similar distractors, while our PROT3D can robustly locate the target in these cases owing to its progressive refinement strategy, showing its efficacy for generic 3D tracking.

Maintenance. Our GSOT3D will be hosted on the popular Github (all download links and our models will be publicly released). This enables conveniently checking the feedback from the community, and thus allows for improvements via necessary maintenance and updates by the authors. Besides, the authors will try their best to collect evaluation results of future trackers, aiming at providing up-to-date analysis and comparison on GSOT3D. Our ultimate goal is to develop a long-term and stable platform for 3D object tracking.