Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking

Advances in 2D video [54, 27, 26] and object tracking approaches [31, 16, 37, 23, 35, 49] stimulate the development of the 3D tracking, and the methods are evolving all the time. Early 2D SOT methods mainly focus on the classifier design like structured output SVM [37, 14], correlation filter [17, 25]. With the prevalence of deep learning, end-to-end trainable trackers emerged in the visual community. The Siamese-like structure [6, 18, 15, 23, 47, 43, 29] based methods have been popular in the tracking field and have numerous variants. SiamFC [1] is a pioneering work integrating feature correlation into a fully convolutional Siamese network for visual tracking. Subsequently, improvements like introducing detection components such as region proposal network [23, 22, 8], discriminating fore and back-ground [50], anchor-free detecting [13] etc.. Recently, vision transformers have been introduced into 2D SOT [4, 39, 44, 42] to exploit the long-range modeling of the attention mechanism to fuse the features effectively. Moreover, the one-stream network trackers based on transformers are proposed in 2D tracking. The OSTrack [45] proposes joint feature learning and relation modeling, masking a proportion of image patches to save computational overheads. SimTrack [3] concatenates the template and search input, improving the patch embedding method with a Foveal window strategy. Our work draws inspiration from the aforementioned trend in 2D vision. However, it is specifically tailored to address the challenges of the 3D Single Object Tracking (SOT) problem, taking into account the unique characteristics of point clouds. Furthermore, we analyze the success of the transformer-based single-branch backbone, attributing to the dynamic affinity of the attention mechanism, which enables the synchronization of feature extraction and matching processes.

In this paper, we only discuss LiDAR-based 3D object tracking. Till to now, almost all the 3D tracking methods [12, 33, 9, 46, 5, 34, 53, 51, 19, 20, 38] are based on the Siamese structure. The pioneering work in this field is the SC3D [12], which initially defines the task. SC3D employs cosine similarity to measure the resemblance between template and search region features, and incorporates shape completion during training to enhance appearance refinement. Trackers following SC3D make advancements from two perspectives. Firstly, they enhance the matching network [33, 51, 19, 53, 5, 40, 52]. For instance, MLVSNet [40] uses the CBAM module [41] to enhance the vote cluster features with both channel and spatial attention. STNet [20] employs cross- and self-attention modules to enhance the interaction between the extracted template and search region features, boosting their feature-level integration. M2Track introduces a motion-centric paradigm and motion estimation module to correlate the template and search features instead of appearance matching. All these methods depend on a standalone module to match the features. Secondly, the trackers [33, 9, 19, 40, 34] have attempted to improve the prediction head part. P2B [33] employs Hough Voting to predict the target location, and  [51, 34, 40] all follow the voting strategy to make predictions. 3D-SiamRPN [9] uses an RPN head to predict the final results. LTTR [9] and V2B [19] use center-based regressions to predict several object properties. However, few efforts are made in exploring the encoder/backbone of trackers as PointNet++ [32] is the feature extractor by default. In this study, we focus on the backbone design and incorporate the matching process directly into the backbone, significantly streamlining the tracker network.

In the configuration of 3D LiDAR single object detection (SOT) task, the 3D bounding box (BBox) is defined as $(x,y,z,w,l,h,\theta)\in\mathbb{R}^{7}$ , where the $(x,y,z)$ represents the coordinate center of the BBox and $(w,l,h),\theta$ stand for the BBox size and heading angle (the rotation around the up-axis) respectively. Generally, the BBox size is assumed to be fixed by default even when the target object is non-rigid, thus minimizing the dimensions of BBox from $\mathbb{R}^{7}$ to $\mathbb{R}^{4}$. Given a sequence of temporally-connected point clouds $\mathcal{\{P}_{i}\}_{i=1}^{T}$ ($T$ is the number of points in each frame) and a initial BBox $\mathcal{B}_{1}$ of the target, the goal of SOT is to localize the target BBoxes $\mathcal{\{B}_{i}\}_{i=2}^{T}$ in all frames online. Following the previous manner, a template point cloud $\mathcal{P}^{t}=\{p_{i}^{t}\}_{i=1}^{N_{t}}$ and a search region $\mathcal{P}^{s}=\{p_{i}^{s}\}_{i=1}^{N_{s}}$ are generated, where $N_{t}$ and $N_{s}$ are number of template and search region points. The template $P^{t}$ is generated by cropping and centering the target in the initial frame based on the initial BBox.

We propose to replace the conventional Siamese-like backbone paradigm with a sole backbone, eliminating the double forward process of Siamese structure. Therefore, template and search region seeds are concatenated to do forward propagation. The Transformers’ property of long-range relation-modeling is the intrinsic merit to tackle the concatenated template and search region seeds. Based on that, we leverage the self-attention modules to build the single-branch backbone as shown in Fig. 2(a). In our approach, we utilize a Query & Group module to sample the template seeds, denoted as $\mathcal{P}^{t}\in\mathbb{R}^{N_{t}\times C}$, using the FPS method before jointly forwarding. This module also groups the k-nearest points to aggregate features. However, when it comes to the search region point cloud, represented as $\mathcal{P}^{s}\in\mathbb{R}^{N_{s}\times C}$, we solely employ the Group module to aggregate neighbor information without reducing the number of search points. Subsequently, the template and search seeds are concatenated, incorporating a joint parametric positional embedding for the localization of tokens. This process is as follows:

	$\displaystyle\mathcal{T}^{t}=$	$\displaystyle{\rm kNN(FPS}(\mathcal{P}^{t})),\mathcal{P}^{t}\in\mathbb{R}^{N^{\prime}_{t}\times C^{\prime}},N^{\prime}_{t}<N_{t},$		(1)
	$\displaystyle\mathcal{T}^{s}$	$\displaystyle={\rm kNN}(\mathcal{P}^{s}),\mathcal{P}^{s}\in\mathbb{R}^{N_{s}\times C^{\prime}},$
	$\displaystyle\mathcal{T}^{ts}$	$\displaystyle=[\mathcal{T}^{t};\mathcal{T}^{s}]+\mathbf{p_{e}},\mathcal{T}^{ts}\in\mathbb{R}^{(N_{t}+N_{s})\times C^{\prime}}.$

Afterward, linear layers are leveraged to project the input tokens into query, key, and value latent and the head-wise joint attention map is calculated to model the intra- & inter-relations of template and search tokens.

	$\displaystyle Q^{ts},K^{ts},V^{ts}$	$\displaystyle=W_{q}(\mathcal{T}^{ts}),W_{k}(\mathcal{T}^{ts}),W_{v}(\mathcal{T}^{ts}),$		(2)
	$\displaystyle\mathcal{A}_{m}^{ts}=$	$\displaystyle{\rm Softmax}\frac{Q_{m}^{ts}(K_{m}^{ts})^{\top}}{\sqrt{C^{\prime}/M}}.$

Based on the joint head-wise attention map $\mathcal{A}_{m}^{ts}$ of template and search tokens, the features extracted by multi-head attention are:

$$\mathcal{T^{\prime}}^{ts}=[\mathcal{A}_{1}^{ts}V^{ts}_{1},\mathcal{A}_{2}^{ts}V^{ts}_{2},\dots,\mathcal{A}_{M}^{ts}V^{ts}_{M}]\mathcal{W}+\mathcal{T}^{ts},$$

(3)

where $M$ is the number of the attention heads and $\mathcal{W}$ is the weight of a MLP.

We illustrate how the single-branch structure can synchronize the process of feature extracting and matching with a simple backbone. The formula of joint attention $\mathcal{A}_{m}^{ts}$ in Eq. 2 can be expanded as ($\sigma$ represents Softmax operation):

		$\displaystyle=\bm{\sigma}\frac{[Q_{m}^{t};Q_{m}^{s}][K_{m}^{t};K_{m}^{s}]^{\top}}{\sqrt{C^{\prime}/M}}$		(4)
	$\displaystyle=[\bm{\sigma}\frac{Q_{m}^{t}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}},$	$\displaystyle\bm{\sigma}\frac{Q_{m}^{t}(K_{m}^{s})^{\top}}{\sqrt{C^{\prime}/M}};\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}},\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{s})^{\top}}{\sqrt{C^{\prime}/M}}].$

The new extracted search region features can be obtained based on Eq 4 as:

	$\displaystyle\mathcal{T^{\prime}}_{m}^{s}=\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}}$	$\displaystyle V_{m}^{t}+\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{s})^{\top}}{\sqrt{C^{\prime}/M}}V_{m}^{s},$		(5)
	$\displaystyle[V_{m}^{t};V_{m}^{s}]$	$\displaystyle=V_{m}^{ts},$

depending on the projected features $[V_{m}^{t},V_{m}^{s}]$ of both the template and search region from last layer. The attention queried from search region to template, $\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}}$, is the matching matrix that guides to aggregate highly-relevant template features. Moreover, the matching matrix is dynamic, which is determined by the changing query and key latent of search and template features as:

		$\displaystyle\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}}=\bm{\sigma}\frac{W_{q,m}^{s}\mathcal{T}_{m}^{s}(\mathcal{T}_{m}^{t})^{\top}(W_{k,m}^{t})^{\top}}{\sqrt{C^{\prime}/M}},$		(6)
		$\displaystyle i.e.~{}~{}~{}~{}~{}~{}~{}\bm{\sigma}\frac{Q_{m}^{s}(K_{m}^{t})^{\top}}{\sqrt{C^{\prime}/M}}\propto[\mathcal{T}_{m}^{s};\mathcal{T}_{m}^{t}].$

To conclude, the synchronization attributes to the dynamic mechanism of matching matrix, continuously adapting the matching relations according to the extracted features of template and search region.

Based on the synchronizing mechanism of feature extracting and matching, we propose to replace the Point-wise Transformer [48] with a novel Attentive Points-Sampling Transformer (APST). APST is proposed based on the observation that backbones utilized in previous 3D LiDAR trackers always adopt a non-parametric strategy of down-sampling the search region features/tokens with farthest point sampling (FPS) or randomly sampling to reduce the points as introduced in PointNet++ [32]. Nevertheless, this non-parametric sampling method lacks learnability and controllability since no parameters associated with the sampling process are updated during model training. Consequently, the final performance is compromised due to the significant influence of the points-center in LiDAR, which determines the effectiveness of extracting features around the foreground, as shown in Fig. 3.

Therefore, we introduce the APST, which involves the selection of points based on attentive relations between the tokens of the template and search region as illustrated in sub-figure (b) of Fig. 2. The attentive response of template tokens reacting to search region tokens is considered as positively responding search tokens are more likely to be in the foreground. In that case, we segment the attention map and separate the $\mathcal{A}_{m}^{t\rightarrow s}=\bm{\sigma}\frac{Q_{m}^{t}(K_{m}^{s})^{\top}}{\sqrt{C^{\prime}/M}}$ from the Eq. 4, averaging along the dimension of template-tokens attention among all transformer heads to acquire the response scores. To ensure the maximum response of search tokens to template ones, a group of search region indexes, named $\Omega_{s}$, is dynamically updated as:

$$\Omega_{s}^{*}=\mathop{\arg\max}_{\Omega_{s}}\frac{1}{N_{t}}\frac{1}{M}\sum\limits_{t=0}\limits^{N_{t}}\sum\limits_{m=0}\limits^{M}\frac{Q_{m}^{t}}{\sqrt{C^{\prime}/M}}(K_{m}^{s})^{\top}\bigg{|}_{s\in\Omega_{s}},$$

(7)

Based on the optimal solution $\Omega_{s}^{*}$, the search region tokens are sampled to decrease the number of tokens after multi-head self-attention and concatenate with template ones. Note that the template tokens are sampled with the FPS method as introduced in Sec. 3.2, and only the search region tokens are sampled with the attentive sampling method.

The dynamic-affinity property of the Transformer suggests that the attention map is influenced by the latent tokens projected through learnable linear layers. Hence, it is reasonable to assert that sampling tokens guided by the attention map is linked to model learning, as the attention map is generated based on updated parameters. As a result, this type of token/point sampling proves advantageous in providing prior knowledge for centroid selection and enhancing the efficient aggregation of representational information, as shown in Fig. 3.

With the encoded point-wise features of various scales, a multi-scale feature fusion module is adopted to fuse these features and output the features with the same origin input size, feeding into the decoder part for final predictions. Following the V2B [19], the features are voxelized as a volumetric representation and 3D convolutions are utilized on the encoded features. Afterward, the BEV feature maps are acquired by pooling on the z-axis for regression. Focal loss and L1 loss are leveraged for classification and BBox center offset and rotation regression, respectively. A detailed introduction is presented in the supplementary.

We compare the SyncTrack with other state-of-the-art methods from the pioneering SC3D [12] to the most recent siamese network STNet [20] and single branch network M2Track [52], as shown in Table 1. We separate the trackers into Siamese Network and Single Branch Network categories to compare with our proposed SyncTrack. Compared with siamese structured trackers, the SyncTrack achieves the best results on rigid and non-rigid object tracking, outperforming current tracking methods based on siamese networks on most specific categories and the overall mean results. STNet [20] is the state-of-the-art siamese-based tracking method with self-attention and cross-attention modules to match the template and search-region features. Our SyncTrack outperforms the STNet by a relatively large margin on Car, Van and Pedestrian categories under the evaluation of Success metric. Also, SyncTrack surpasses the previous best Mean results by $2.8\%$ and $1.8\%$ on Success and Precision metrics, respectively.

We also compare the SyncTrack with the only current single-branch tracker M2Track [52]. Our SyncTrack outperforms M2Track by a large margin, up to $7.8\%$ in Success in the category of Car whereas the M2Track is better in the category of Pedestrian. However, for the comprehensive evaluation, the Mean performance of all frames, our SyncTrack outperforms the M2Track by $1.2\%$ on the Success metric.

Note that nuScenes dataset only annotates keyframes and provides official interpolated results for the remaining frames, so there are two configurations for this dataset. The first is from [51, 52] which trains and tests both only on the keyframes. The second one is from [19, 20], which trains and tests on all the frames. The results based on these two configurations are different. We believe that the motion in key frames is substantial, which does not conform to the practical applications. Therefore, we train and test all the frames in this paper. We train the previous methods on the nuScenes by ourselves using their official codes to compare with our SyncTrack as many results are missing or only reported by testing nuScenes test-split with a pre-trained KITTI model.

As shown in Table 2, SyncTrack performs significantly better than other trackers on the mean results of four categories. Specifically, the SyncTrack yields the best results on both metrics and on most categories except the Pedestrian, which is lower than M2Track [52] and BAT [51] by a minor margin ($1.6\%$ and $0.4\%$). However, in the Truck category, our SyncTrack outperforms state-of-the-art by a large margin, up to $5.9\%$ and $6.2\%$ on Success and Precision respectively.

To evaluate the generalization ability of SyncTrack, we pre-train the model on the KITTI dataset and test it directly on the nuScenes dataset without fine-tuning. The results are shown in Table 4. It can be observed that SyncTrack outperforms other methods on the mean results of four categories by a large margin. SyncTrack not only achieves a good balance between inference speed and tracking accuracy, but also generalizes very well to new domains.

We prove that our SyncTrack has good scalability in large-scale datasets like nuScenes. The basic model of SyncTrack has merely one Transformer layer in each stage to ensure real-time performance for tracking. However, the backbone of SyncTrack is scalable in both depth and width. We name the basic model as SyncTrack-Small. Furthermore, the SyncTrack-Mid has three Transformer layers at every stage, with nine layers total. As for the SyncTrack-Large, it doubles the number of feature channels of the SyncTrack-Mid to [256, 128, 64] for every stage. Table 5 reveals that performance improves when scalability increases not only in depth (Small v.s. Mid), but also in width (Mid v.s. Large).

We conduct comprehensive ablations to evaluate the components of SyncTrack.

In Figure 4, we present visualization results obtained from LiDAR video sequences taken from the KITTI dataset. These visualizations show the motion pattern of objects belonging to four categories wihtin the KITTI dataset. Obviously, our SyncTrack excels in accurately tracking the intended target and predicting bounding boxes when compared to STNet [20]. This achievement is primarily attributed to the dynamic and abundant feature interactions that occur between the template and search region seeds in SyncTrack. These interactions enable our algorithm to effectively distinguish the foreground from the background, leading to superior performance.