Towards Class-agnostic Tracking Using Feature Decorrelation in Point Clouds

LiDAR point cloud characterizes by long-range spatial information, which has promoted a deep research history in 3D tracking. Held et al.[16] constructed an up-sampled point cloud with color information, and then used a color-augmented alignment strategy to estimate the velocity of the tracked vehicle. Dewan et al.[17] utilized the RANSAC algorithm to estimate motions of sensor and moving objects, and tracked objects by updating motion based on the Bayesian approach. However, these methods resort to data association or segmentation and just obtain object points instead of 3D bounding box indicating size and orientation. Subsequently, Asvadi et al.[18] proposed a parallel mean-shift algorithms to predict 3D bounding box representation, and Li et al.[19] introduced an optimization-based algorithm that takes shape and motion priors into account.

Recently deep learning brings new energy for 3D object tracking in point clouds, and some data-driven methods have been achieving impressive performance. For example, as a counterpart of 2D visual tracking using Siamese network [20], SC3D [8] is the pioneer to learn a discriminative feature space for calculating cosine similarity between template and candidates. However, this method has the following drawbacks: 1) only using global feature embedding ignores the object relations in the search region; 2) the candidate generation is limited by the probability sampling; 3) the state estimation is not considered. Accordingly, Feng et al.[21] and Tian et al.[22] introduced coarse-to-fine and warp learning strategies to adjust the state of the tracked vehicle, respectively. Afterward, a milestone work in this area is that P2B [9] takes inspiration from the deep Hough voting [23], and designs a feature embedding module to fulfill the Siamese-based region proposal network (RPN) [24] in 3D case. Taking the P2B as the baseline, Tian et al.[25] integrated the philosophy of supervised descent method into PointNet++ to refine the predicted bounding boxes progressively, Hui et al.[26] substituted the point-based RPN head with a BEV-based module. Cui et al.[27] and Shan et al.[28] leveraged the popular transformer to promote information embedding procedure by attention. Considering that mostly single-view scanned data have similar patterns but different sizes, BAT [10] exploits the Box-Cloud representation as an additional clue to guide feature embedding. All these methods focus only on class-specific tracking, whereas we aim to put forward a class-agnostic tracking approach in point clouds.

Some point cloud tasks based on the deep Siamese network have obtained impressive performances. In particular, point cloud registration is the most related topic with 3D SOT, which seeks the best transformation matrix to align the source and target point clouds. Recently, Aoki et al.[29] incorporated the Lucas & Kanade algorithm with PointNet [6] to devise a recurrent trainable network, called PointNetLK. Wang et al.[30] used the PointNet to generate soft corresponding points and then solve the rigid transformation by a differentiable singular value decomposition. Choy et al.[31] used a sparse convolution network to propose deep global registration with correspondence confidence prediction. Huang et al.[32] presented an overlap-attention block based on a graph neural network, dedicated to the registration problem with low overlap. As for the partially scanned objects in the wild, Groß et al.[33] proposed AlignNet to learn to align scans in a coarse-to-fine manner. Actually, such Siamese network-based approaches also involve the generalization ability on unseen categories but are overlooked by them. Thus, based on the PointNetLK, Li et al.[34] claimed that the Jacobian matrix with respect to the input coordinates can improve the generalization ability for point cloud registration. Although the fusion and matching technologies that appear in these Siamese networks can provide some useful references to 3D SOT, how to implement a general tracking algorithm is still an open problem to be solved.

Towards the real-world applications, some works have paid attention to eliminating the spurious association between variables during the training. For example, with the decorrelation regularizer, Kuang et al.[35] constrained the generation of learned weights to improve the stability of the linear regression model. Shen et al.[36] theoretically explained the problem occurring in distribution shifts between training and testing sets. Afterward, Shen et al.[37] proposed group-based variables partition to alleviate the problem of the over-reduced sample size. Nevertheless, these works are aimed at the linear model. He et al.[38] thus proposed a Non-I.I.D. dataset for images recognition, and presented a batch balancing method. Furthermore, Zhang et al.[39] leveraged Random Fourier Features and iterative weight update to address the distribution shift in images classification. Although these methods have shown the effectiveness by disentangling the features, there are still no attempts that handle the distribution shifts for 3D tracking in point clouds. In our work, we present a feature decorrelation module for class-agnostic tracking, which is specially designed for the information embedding state.

Let $P=\{p_{i}=(x_{i},y_{i},z_{i})\}_{i=1}^{N}$ represent a point cloud with $N$ points captured by LiDAR. Given a specified target $P_{T}$ (also dubbed as template) in the first frame, 3D SOT aims to generate a 3D bounding box to accurately localize the target in each coming frame. In practice, considering that each scan frame covers a very large scenario, we usually crop a search region $P_{S}$ around the previous tracking result. Here, the template $P_{T}$ and search region $P_{S}$ are composed of $N_{T}$ and $N_{S}$ points, respectively. In addition, 3D bounding box $\mathcal{B}$ is used to represent the state of the target because it includes the center position $(x,y,z)$, size $(l,w,h)$ and yaw angle $\theta$, i.e., $\mathcal{B}=(x,y,z,l,w,h,\theta)$. Briefly Speaking, during online tracking, the proposed method takes the template $P_{T}$ and the search region $P_{S}$ as input and predicts a bounding box $\mathcal{B}$ for the target in each frame.

As a milestone, P2B [9] proposes a point-to-box network for class-specific tracking, attracting much attention in the research forum of 3D SOT. Based on the P2B, there are many improvements [25, 10, 28, 26, 40], among which BAT [10] introduces geometric information, Box-Cloud, to achieve considerable results. Therefore, we take these two representative trackers (P2B and BAT) as the baselines.

Mathematically, let $\psi$ be the feature extractor, which can be PointNet++ [7]. The intermediate features of the template $P_{T}$ and search region $P_{S}$ can then be denoted as $F_{T}=\psi(P_{T})\in R^{M_{T}\times D}$ and $F_{S}=\psi(P_{S})\in R^{M_{S}\times D}$, respectively. Here, $M_{T}$ and $M_{S}$ are the number of the down-sampled points. The information embedding module $\phi$ consumes $F_{T}$, $F_{S}$ and other geometric information $G$, and then produces fused features $Z=\phi(F_{T},F_{S},G)$. As for P2B, $G$ is point coordinates, i.e., $G=(P_{T},P_{S})$. For BAT, $G$ consists of template coordinates $P_{T}$ and its Box-Cloud $C_{T}\in R^{N_{T}\times 9}$, i.e., $G=(P_{T},C_{T})$. Finally, the proposal generation module $\gamma$ utilizes the deep Hough voting mechanism [23] to output candidate bounding boxes $\{\hat{\mathcal{B}}_{i}\}_{i=1}^{N_{\mathcal{B}}}$ and its corresponding confidence scores $\{s_{i}\}_{i=1}^{N_{\mathcal{B}}}$. The target location is determined by $\hat{\mathcal{B}}_{i}$ with the highest score.

To train the network, the loss function has the four constraints, i.e.,

Concretely, $\mathcal{L}_{vote}=\frac{1}{M_{S}}\sum_{i=1}^{M_{S}}\mathbb{I}(p_{i})\|p^{*}-(p_{i}+\Delta p_{i})\|$ is responsible for deep Hough voting, where $p^{*}$ is the ground-truth of target center, $\Delta p_{i}$ is the predicted offset, and $\mathbb{I}(p_{i})$ is the indicator function that outputs 1 if $p_{i}$ belongs to the target point, otherwise 0 to non-target point. $L_{cls}=-\frac{1}{M_{S}}\sum_{i}\mathbb{I}(p_{i})\log(r_{i})$ is for foreground classification, where $r_{i}$ is generated by an MLP based on $Z$. In addition, the predicted boxes and confidence score are supervised by $L_{1}$ smooth function and binary cross entropy, respectively. That is, $L_{box}=\frac{1}{N_{\mathcal{B}}}\sum_{i}\|\hat{\mathcal{B}}_{i}-\mathcal{B}^{*}\|$, where $\mathcal{B}^{*}$ is ground-truth; $L_{score}=-\frac{1}{N_{\mathcal{B}}}\sum_{i}\mathbb{I}(\hat{\mathcal{B}}_{i},\mathcal{B}^{*})\log(s_{i})$, where $\mathbb{I}(\hat{\mathcal{B}}_{i},\mathcal{B}^{*})$ is 0 when the interaction-of-union between $\hat{\mathcal{B}}_{i}$ and $\mathcal{B}^{*}$ is less than 0.3, and 1 when it is larger than 0.6.

The 3D SOT task can often be divided into three modules: feature extraction, target information embedding, and proposal generation. Currently, most class-specific trackers focus on how to conduct the fusion between the template and search region in observed distribution, but ignore the generalization when confronted with out-of-distribution (OOD) tracklet. The main gap is that when tracking the OOD instances during testing stage, the background points usually have spurious correlations with the target’s location (see Fig.3-(a)(b)(c)). This correlations mainly comes from the fused features trained on limited classes, which easily embed template’s features into irrelevant features of search region of unseen category.

Therefore, our original motivation is to keep the two basis properties for class-agnostic tracking, regardless of whether the tracked class was observed or not. One of the properties is consistency which requires that the foreground features of the target should be as close as possible. The other one is discrimination which requires that the features between the foreground and background need to be as far away as possible. When the fused features belong to such pattern (see Fig.3-(d)), the last module, i.e., proposal generation, can learn to assemble the foreground points together and generate a good bounding box. Intrinsically, to attain this goal, the core lies in how to alleviate the correlation. Recently, the literature [14, 39] points out this spurious correlations are caused by the statistical dependence among features. Along this mentality, we try to eliminate the dependence in fused features and propose a feature decorrelation method for class-agnostic tracking, which treats each channel of the fused features as a variable and removes the variable dependence via a set of learned sample weights.

In detail, we first transform the fused features into $Z^{\prime}=\tau(Z)$, where $\tau:R^{M_{S}\times D}\rightarrow R^{D}$ is the aggregation function on the search region. In this work, we simply utilize Average Pooling as $\tau$. Next, we can use the transformed features $Z^{\prime}$ to calculate sample weights. Unlike the linear model used in the previous methods [35, 37, 36], we deal with the non-linear features extracted by deep learning whose dependence is challenging to be measured. In particular, we adopt Random Fourier Features (RFF) [15] for each variable $Z^{\prime}_{i}$, inspired by [39]. Theoretically, to test the independence of two variables $Z^{\prime}_{i}$ and $Z^{\prime}_{j}$, we need to judge whether the cross-covariance operator $\Lambda_{Z^{\prime}_{i},Z^{\prime}_{j}}$ with respect to $u(Z^{\prime}_{i})$ and $v(Z^{\prime}_{j})$ is 0, where $u$ and $v$ are the element of Reproducing Kernel Hilbert Space $\mathcal{H}$. This is gurateened by the following proposition [39]: Assuming $\kappa_{Z^{\prime}_{i}}$ and $\kappa_{Z^{\prime}_{j}}$ are measurable, positive definite kernels defined on the variable $Z^{\prime}_{i}$ and $Z^{\prime}_{j}$ respectively, if the product $\kappa_{Z^{\prime}_{i}}\kappa_{Z^{\prime}_{j}}$ is characteristic, $\mathbb{E}(\kappa_{Z^{\prime}_{i}}(\cdot,\cdot))<\infty$, $\mathbb{E}(\kappa_{Z^{\prime}_{j}}(\cdot,\cdot))<\infty$, we have $\Lambda_{Z^{\prime}_{i},Z^{\prime}_{j}}=0\Leftrightarrow Z^{\prime}_{i}\perp Z^{\prime}_{j}$.

RFF can provide such function space (denoted as $\mathcal{H}_{RFF}$) to attain this goal. More specifically, in a mini-batch, we can have $B$ samples $(Z^{\prime}_{1,i},Z^{\prime}_{2,i},\dotsc,Z^{\prime}_{B,i})$, which are derived from the distribution of variable $Z^{\prime}_{i}$. Let $\mathbf{w}=(\omega_{1},\dotsc,\omega_{B})$ represent the sample weights to be learned, and satisfy $\sum\omega_{m}=1$. The weighted cross-covariance matrix can be estimated as follows:

where $\mathbf{u}=(u_{1},u_{2},\dotsc,u_{N_{u}})$ with $u_{i}\in\mathcal{H}_{RFF},i=1,\dotsc,N_{u}$ and $\mathbf{v}=(v_{1},v_{2},\dotsc,u_{N_{v}})$ with $v_{i}\in\mathcal{H}_{RFF},i=1,\dotsc,N_{v}$. Thereafter, we can optimize the Frobenius norm of the $\hat{\Lambda}_{Z^{\prime}_{i},Z^{\prime}_{j}}$ to solve $\omega_{m}$,

The learned weights $\mathbf{w^{*}}$ are then applied into the final loss function to balance the distribution of each variable in the fused feature, which will enable the fused feature to be foreground consistent. Finally, the whole procedure can be summarized as alternative and iterative optimization (See Algorithm 1):

where $Z^{\prime,(t-1)}$ is the fused features calculated by the network parameters of the last epoch, i.e.,

In this way, we can reduce the dependence on background points in the information embedding process, so that the fused features remain consistency and discrimination, and then perform class-agnostic tracking.

To promote the development of single object tracking in point clouds, Giancola et al.[8] created a benchmark: KITTI Tracking dataset. On each category, they selected some sequences as training set and others as testing set. Following this, there have been many efforts [21, 9, 25, 10, 28, 26] to improve the success and precision ratios on the dataset. It is worth noting that all these methods lack the specific designs for class-agnostic tracking and mainly focus on training different models for each category. However, as an extension of 2D visual tracking, a 3D SOT method should also satisfy the characteristic that it is capable of tracking any object as long as the target is specified in the first frame. For understanding, we illustrate class-specific and class-agnostic tracking in the Figure 5.

For class-agnostic tracking, we introduce four settings according to the mainstream datasets for LiDAR point clouds. Specifically, on KITTI, we use samples from four categories, including cars, pedestrians, vans and cyclists. On the first setting, we use the samples of pedestrians, vans and cyclists to train the models, and during the testing stage, the samples of cars are exposed to the models in addition to the three observed categories. For a well-round comparison, we swap the roles of cars and pedestrians in the second setting. Analogously, on NuScenes, we construct another two settings based on the categories of pedestrians, cars, trunks and bicycles. The first evaluates the class-agnostic performance by unseen category of cars while the second by pedestrians. We may want to traverse all permutations of different categories to obtain a more comprehensive experimental setup. In fact, cars and pedestrians account for the vast majority in the datasets, and these two classes have obvious distribution discrepancies. Therefore, it has been representative to take them as the unseen domain to measure the class-agnostic ability of 3D trackers. We also summarize those class-agnostic settings in the Table I.

Datasets. Following previous methods, we adopt the most popular dataset for LIDAR point cloud, KITTI Tracking Set[12]. Its samples are captured under 21 different scenes, totally yielding 41,713 frames and 808 tracking instances for training and testing. The target categories in this dataset mainly includes cars, pedestrians, vans, and cyclists. The coordinates of each sample are stored in terms of the camera coordinate system.

In order to avoid over-fitting on a single dataset, we also consider another large-scale dataset, NuScenes[13], which is composed of 1,000 scenes. The coordinates of each sample are stored in terms of the LiDAR coordinate system. This dataset is very challenging because it gathers abundant samples from different traffic situations, weather conditions, vehicle types, and so on. Considering that the ground-truth of the testing data is not accessable, we evaluate the trackers on its validation set.

Evaluation Metrics. Like 2D visual tracking, the performance metrics for 3D tracking are also success and precision ratios [41]. The difference with 2D tracking lies in the calculations are extended from 2D to 3D. In particular, the success ratio is related to the interaction-of-union(IoU) between the predicted bounding box and ground-truth bounding box. It statistics the percentage of frames where the IoU is larger than the given thresholds. The precision ratio is dependent on the center distance (CD) between the predicted bounding box and ground-truth bounding box. It also calculates the percentage of frames where the CD is smaller than the given thresholds. By densely drawing the thresholds on the X-axis, we can obtain two specific curves whose Y-axis are success ratio and precision ratio respectively. Following SC3D[8], their area-of-curve(AUC) is treated as the final metrics, denoted as “Success” and “Precision” . The larger the two metrics, the better this tracker is.

Network architecture. For both P2B and BAT, three set-abstraction (SA) layers of PointNet++ are utilized to extract high-level semantic features of key points. Its receptive radius is set to 0.3, 0.5, and 0.7 meters. To down-sample key points, the stride steps are 2, 4, and 8. And the dimension $D$ of is set to 256. For example, it can map the template of shape $N_{T}\times 3$ to a tensor of shape $\frac{N_{T}}{8}\times 256$ (i.e., $M_{t}=\frac{N_{T}}{8}$). For information embedding, P2B uses the target-specific feature augmentation (TFA) module to obtain the fused features $Z$ when BAT leverages a box-aware feature fusion (BAFF) module. The number of proposals, $N_{\mathcal{B}}$, is set to 64 by following the configuration of P2B and BAT. During feature decorrelation phase, we sample 5 basis functions $\{u_{i}\}_{t=1}^{N_{u}=5}$ from RKHS $\mathcal{H}_{RFF}$. Let $N(0,1)$ represent Gaussian distribution and $U(0,2\pi)$ uniform distribution, the function space $\mathcal{H}_{RFF}$ can be written as

Similarly, we can also sample basis functions from $\mathcal{H}_{RFF}$ to obtain $\{v_{t}\}_{t=1}^{N_{v}=5}$.

Training. For class-agnostic tracking, there are a total of four settings: KITTI Setting-1, KITTI Setting-2, NuScenes Setting-1, and NuScenes Setting-2. On each setting, the instances from the observed categories are only used to train the models. We create the template and search region from every instance tracklets. Specifically, for each frame, we add Gaussian perturbation to ground-truth bounding box, yielding a new bounding box. Centered on this box, we crop the search region with an offset of 2 meters. The template is then generated by aggregating the points inside the ground-truth boxes of the first and previous frames. Before sending these two sets of point clouds (template and search region) into the network, we transform them into a local coordinate system according to the corresponding bounding box. During training, a batch of such paired samples is selected randomly from the hybrid set consisting of the observed categories. In addition, when optimizing the Eq. 3, we adopt SGD algorithm to solve the weights $\mathbf{w}$ with an iteration number of 20. Its learning rate is set to 1.0 and decayed 10 every 10 iterations. Regarding other hyper-parameters, we simply follow the baselines. Our experiments run on NVIDIA RTX-2080 GPUs with the help of pyTorch 1.7 and CUDA 10.1.

Testing. We use well-trained models to track both observed and unseen objects. The template is updated by appending the tracked target into the initial one of the first frame. Considering the small motion increment between two consecutive frames, the search region is cropped around the tracking result of the previous frame. Both the template and search region are required to be transformed into their box coordinates system. Because the feature decorrelation method is committed to class-agnostic tracking by ameliorating the offline training scheme, it is not carried out during the testing phase. In other words, it will not affect the efficiency of the baselines during online tracking.

Since P2B[9] was proposed for 3D SOT, there have sprout many improved algorithms, including LTTR[27], V2B [26], DSDM[25] and BAT[10]. These methods have good performance under the class-specific tracking mode. Based on them, we conduct some preliminary experiments, which is used to show the performance change when current methods are shifted from class-specific to class-agnostic tracking mode. Specifically, taking P2B and BAT as the baselines, we retrain them use the class-agnostic mode mentioned in Section III-D, and the resulting models are denoted as P2B-U and BAT-U. We compare them on both observed and unseen categories.

Table II reports the performance of P2B-U and BAT-U on unseen categories of KITTI Setting-1 and Setting-2, and compare them with methods using class-specific tracking mode. Generally, class-specific trackers achieve high performance because the samples of training and testing sets are derived from the same category. For example, according to the success/precision metrics, P2B reaches 56.2% /72.8% on cars and 28.7%/49.6% on pedestrians. Its improved versions like BAT and LTTR can further achieve significant improvements. In particular, BAT surpasses P2B by 9.2%/6.1% on cars and by 17%/23.9% on pedestrians. However, despite these improvements, they still have a long path towards class-agnostic tracking. From the last two rows of Table II, P2B-U only obtains the performance of 32.4%/41.0% on cars, which significantly decreases by 13.8%/31.8% compared with P2B. BAT-U and BAT even differ by 30.5%/45.2% on cars. A similar phenomenon also occurred on pedestrians. In light of this, it is very necessary and meaningful to study some methods to promote the development of class-agnostic tracking. Moreover, there is another finding that BAT-U gets worse performance than P2B-U although its original version, BAT, is greatly superior to P2B. The reason is that it may confuse the network if the Box-Cloud distributions are regressed forcibly on all classes without any prior. This illustrates that the accuracy of class-agnostic tracking is not positively correlated with that of the original class-specific tracker.

While unseen categories can evaluate the generalization ability, we still expect that the trackers can remain original accuracy on observed categories. Therefore, Table III summarizes the tracking results on observed categories of KITTI Setting-1 and Setting-2. Herein, the class-agnostic trackers learn a general model using mixed samples of different categories, while the class-specific trackers separately train multiple models for different categories. As shown in this table, class-specific trackers, BAT and V2B, win the top two places. A more interesting finding is that P2B-U and BAT-U have opposite performance changes after training on KITTI Setting-1 and Setting-2. Specifically, when tracking the objects from pedestrians, vans, and cyclists (Setting-1), P2B-U brings the improvement of 7.8%/8.2% in terms of success/precision metrics compared to its original version P2B, but BAT-U decreases by 12.4%/6.5% compared to BAT. Similarly, when evaluating on the Setting-2, the gain of P2B-U is 2.7%/2.2%, but BAT-U is reduced by 2.5%/2.6%. The reason may lie in that the information embedding of BAT is sensitive when training multiple categories together. This stimulates our curiosity to explore the effects of experimental configurations by mixing different categories. Analysis on this part will be shown in the next section IV-G.

To sum up, according to the above preliminary experiments, the recently advanced trackers still require great efforts to simultaneously obtain accurate tracking results on unseen and observed classes, which validates the standpoints that we put forward in the introduction. Next, we will verify the proposed feature decorrelation method for class-agnostic tracking.

In this subsection, we validate the effectiveness of the proposed module by comparing it with P2B-U and BAT-U. For clarity, we use the suffix “-W” to represent the variations that equip with the feature decorrelation method proposed in Section III-C. Under our new setup, all trackers are supposed to be tested on both unseen and observed categories. To comprehensively compare with different methods, we also consider another large-scale dataset NuScenes in addition to KITTI.

Results on KITTI. As reported in Table IV, when tracking unseen classes on KITTI, P2B-W achieves the best performance. When tracking observed classes, BAT-W ranks first on Setting-2 and P2B-W wins first place on setting-1. Concretely, compared with P2B-U in success/precision metrics, P2B-W can track unseen shapes with the considerable improvements of 2.3%/4.6% on Setting-1 and 2.4%/2.4% on Setting-2; and even on observed shapes, it can reap the rewards of 2.4%/4.1% on Setting-1. Similarly, for unseen shapes, BAT-W outperforms BAT-U by 3.2%/3.6% on Setting-1 and 1.7%/6.1% on Setting-2. As for observed shapes, BAT-W can still obtain 35.7%/50.3% and 61.8%/74.5% performance on Setting-1 and Setting-2 respectively, which are 1.8%/0.5% and 2.2%/1.4% higher than BAT-U. This demonstrates that the proposed module plays an important role in class-agnostic tracking. From another perspective, it is worth mentioning that P2B-W overpasses BAT-W by 6.6%/8.3% and 12%/19.9% on two unseen settings although BAT is better than P2B in class-specific tracking, which tells us strong baselines may have inconsistent performance in class-agnostic tracking.

Results on NuScenes. In the last four columns of Table IV, we report the tracking results on NuScenes Setting-1 and Setting-2. Overall, P2B-W and BAT-W achieve the best results on unseen categories. In particular, compared with P2B-U, P2B-W increases by 3.1%/4.5% and 1.6%/2.1% on Setting-1 and Setting-2, respectively. And BAT-W also achieves the improvements of 2.4%/2.0% and 2.1%/1.0% in comparison to its baseline BAT-U. These results further illustrate the effectiveness of the feature decorrelation based on sample weighting. As for observed categories, both methods with suffix “-W” obtains competitive performance against their baselines. For example, on setting-1, BAT-U is enhanced from 30.9%/44.0% to 33.4%/47.8% in success/precision metrics, while P2B-U from 31.1%/46.1% to 32.6%/48.3%.

From a whole point of view, we further draw the following conclusions. Firstly, according to the overall performance, the variations of P2B family are better than BAT family. Secondly, even though our proposed method is demonstrated to be conductive to class-agnostic tracking, there is still potential room to catch up with the performance of class-specific tracking. Lastly, similar shapes may provide some useful prior for tracking unseen categories. For instance, the distribution of vans is similar to that of cars on KITTI and the distribution of trucks resembles that of cars on NuScenes, which generally brings about higher results when cars are unseen class on Setting-1.

Sharpness-Aware Minimization (SAM)[42] recently has attracted increasing attention in model generalization. The theoretical support behind this method is that when optimizing the non-convex loss function of the deep neural network, the parameters under flat extreme usually have more generalization ability than those under sharp extreme. Therefore, SAM not only pursues the minimum loss value but also considers the flatness of the loss landscape, where the flatness means that a $\delta$-neighborhood of the solved parameters ought to have very approximate loss values. In this work, this SAM method also is taken into account so as to deal with the class-agnostic tracking problem. Following the previous framework, we apply it into the two baselines—P2B and BAT—under our proposed settings. The suffix “-S” is utilized to denote the corresponding variations trained with SAM. For well-round comparisons, we evaluate them on both KITTI and NuScenes.

As we can see from Table IV, the methods with the suffix “-S” do not have stable performance on unseen categories. For example, BAT-S achieves a satisfactory performance solely on NuScenes Setting-2, while all others are inferior to P2B-W and BAT-W. Additionally, for observed categories, P2B-S and BAT-S obtain worse tracking results than P2B-W and BAT-W respectively. Especially, on KITTI Setting-2, BAT-W outperforms BAT-S in success/precision metrics, up to 18.3%/16.2%. Therefore, the robustness of such variants is a key problem worth considering for class-agnostic tracking. We conjecture that the solution of SAM may approach the edge of flatness, and gathering stochastic weights densely may be able to explore more flatter extreme [43].

Phenomenon of the spurious correlation. One may think that the class-specific tracking also involves the spurious correlation from irrelevant features. But in fact, the class-specific tracking is to train and test a model on the same class, which can handle the observed distribution of a category well. When confronted with unseen classes (i.e., class-agnostic tracking), the model becomes confused and the impact of irrelevant features will be amplified. To validate this, we show the visualization results of the subsequent voting process in Figure 8. As we can seen, for the same instance, the class-specific tracking is slightly affected by the irrelevant features, whereas and the class-specific tracking even clusters the background points together.

Effectiveness for the fused features. As mentioned before, we aim at making the fused features consistent among foreground points while being able to distinguish background points. To verify whether this goal is achieved, we visualize semantic information based on the fused features. Figure 6 shows the visualization results of P2B-U, P2B-W, and P2B-S. Figure 7 plots the results of BAT-U, BAT-W, and BAT-S. All the models are evaluated on unseen categories. In these figures, the more yellow the point is, the greater the probability that the point lies on the target surface. We cover the targets with the orange rectangles. From Figure 6, we can observe that P2B-W has a much brighter yellow on the scanned points of the target compared with P2B-U and P2B-S. In particular, P2B-S even generates a high similarity for those features derived from background points (labeled in circles), which will mislead the subsequent tracking procedure. In addition, as shown in Figure 7, BAT-W can produce more discriminative cases where the foreground points tend to be yellow and the background points are dark consistently. Note that, both BAT-U and BAT-S have the phenomenon of outputting spurious features for background points (labeled in circles).

What kind of baselines can work well for class-agnostic tracking? Like 2D visual tracking, the ultimate goal of 3D SOT is to train a general model capable of tracking any specified target no matter whether it is observed or not during training. Therefore, it is inevitable to offline train a model using mixed samples from different categories. Here, we discuss the impact of experimental configurations that mix different categories. On both KITTI and NuScenes, we run three different combinations and test the trained models on the same categories. As reported in Table V, we observe that P2B-W has a better expansibility than BAT-w in all configurations. We deem that a good 3D tracker should balance the two aspects well. Firstly, for intra-class objects, the point clouds scanned by LiDAR from a single view usually have the same patterns, a tracker thus should be able to discern the target from these inter-class distractors. Secondly, for inter-class objects, the distribution and size often have obvious discrepancies, which requires a tracker to be robust to the shift of inter-class objects. P2B and BAT overlook the first and second points, respectively. In particular, BAT is easy to confuse the network if the Box- Cloud is forced to be regressed on all kinds of classes without any prior.

Limitations and Future works. The sample weighting method is originally designed for out-of-distribution generalization by balancing each variable in each sample. In this work, we follow the literature [39] to iteratively adjust the training instances in a mini-batch. It can be explored to adopt other strategies to update global weights. In addition, there are several other ways to be worth studying for class-agnostic tracking apart from sample weighting. For instance, one of the solutions is that the network can learn to learn from the shift between observed and unseen domains. Therefore, we can let meta-learning[44] simulate the “learning-then-generalization” procedure, and then apply it to online tracking. In the realm of 2D visual tracking, some researchers [45, 46] treated each video clip as one domain, and leveraged meta-learning to learn fast adaptation ability. However, the characteristic of point cloud data is that the obvious difference between categories is only reflected in the spatial distribution. When each point cloud sequence is treated as a domain, especially when the number of categories is unbalanced, it cannot give full play to its advantages. Alternatively, taking inspiration from [47, 48], we can view each scanned category as one domain to design class-agnostic tracking. Another solution is that we can use data generation to enhance the generalization in real-world scenarios. For example, the deformed point cloud learned from adversarial attack [49] can be utilized to expand the original data distribution. And based on the idea of PointAugment [50], we can even control the generation of more diverse samples during training.