Joint 3D Object Detection and Tracking Using Spatio-Temporal Representation of Camera Image and LiDAR Point Clouds

3D object detection methods can be divided into two categories: (i) LiDAR-only and (ii) sensor fusion-based methods. The LiDAR-only 3D detectors encode point clouds using the PointNet (Qi et al. 2017a, b) and detect objects by applying a detection-head network. These methods include PointRCNN (Shi, Wang, and Li 2019), Part A² (Shi et al. 2019), STD (Yang et al. 2019), 3DSSD (Yang et al. 2020), VoxelNet (Zhou and Tuzel 2018), SECOND (Yan, Mao, and Li 2018), PointPillar (Lang et al. 2019), and CIA-SSD (Zheng et al. 2021). To overcome the limitations of the LiDAR-only approach, many sensor fusion-based 3D object detection methods have been developed, which enhance object features by combining intermediate features obtained from the camera image and LiDAR point clouds. These methods include ContFuse (Liang et al. 2018), MMF (Liang et al. 2019), CLOCs (Pang, Morris, and Radha 2020), and 3D-CVF (Yoo et al. 2020). In our method, we adopt a 3D-CVF (Yoo et al. 2020) as the baseline detector.

Numerous 3D MOT methods follow the tracking-by-detection paradigm (Weng et al. 2020a; Chiu et al. 2020; Weng et al. 2020b; Zhai et al. 2020; Zhang et al. 2019; Baser et al. 2019), in which the objects are first detected in 3D, the tracker then associates the detected objects. Since a simple but effective baseline 3D MOT method, AB3DMOT was proposed in (Weng et al. 2020a). Various 3D MOT methods have been proposed including mmMOT (Zhang et al. 2019), FANTrack (Baser et al. 2019), and GNN3DMOT (Weng et al. 2020b). These methods do not consider the co-optimization of detection and tracking, limiting their potential for further performance gains.

Various JoDT methods have been proposed in (Zhang et al. 2021; Wang et al. 2020; Wang, Kitani, and Weng 2020; Peng et al. 2020; Hu et al. 2019; Shenoi et al. 2020; Kim and Kim 2016; Kieritz, Hubner, and Arens 2018; Ke et al. 2019; Munjal et al. 2020). In the early phase, the idea of JoDT was adopted for 2D MOT. CDT (Kim and Kim 2016) restored undetected objects by examining forward and backward tracing of tracking results. JDE (Wang et al. 2020) and FairMOT (Zhang et al. 2021) incorporated a re-identification model of the tracker into the detector. GSDT (Wang, Kitani, and Weng 2020) performed simultaneous object detection and tracking using GNN.

Recently, JoDT has been extended to 3D MOT. Mono3DT (Hu et al. 2019) enhanced the performance of monocular camera-based 3D object detection by tracking moving objects across frames via occlusion-aware association and depth-ordering matching. JRMOT (Shenoi et al. 2020) combined re-identification, detection and tracking steps into a joint probabilistic data association framework. The proposed 3D DetecTrack differs from the aforementioned methods in that the detector and the tracker cooperatively generate strong spatio-temporal features, which are then used to perform 3D object detection and tracking.

The overall architecture of the proposed 3D DetecTrack is depicted in Figure 1. The proposed JoDT consists of i) detection stage and ii) tracking stage. We denote the tracklet at the $(t-1)$th time step as $T_{t-1}=(T_{t-1,1},...,T_{t-1,N_{T}})$, where $N_{T}$ is the number of tracked objects. The objective of JoDT is to produce the detection results $\mathcal{D}_{t}=(\mathcal{D}_{t,1},...,\mathcal{D}_{t,N_{D}})$ and the tracklet $T_{t}$ given $T_{t-1}$ and the sensor measurements $Y_{t}$ acquired at the $t$th time step, where $N_{D}$ is the number of detected objects. Note that $N_{T}$ and $N_{D}$ can vary over time.

Detection stage: We adopt the two-stage detection model, 3D-CVF (Yoo et al. 2020) as our baseline 3D object detector. We choose the 3D-CVF because it can generate both camera-view 2D features and bird’s eye view (BEV) LiDAR features. In our work, we modify the 3D-CVF to produce spatio-temporal features. Let ($F_{t-1}^{(v)},F_{t}^{(v)}$) be the camera-view features extracted from the images ($I_{t-1},I_{t}$) via the shared CNN backbone network. Similarly, let ($F_{t-1}^{(BEV)},F_{t}^{(BEV)}$) be the BEV features obtained by the voxelization process followed by the 3D CNN. The spatio-temporal feature aggregation network (SFANet) aggregates two concatenated features $(F_{t-1}^{(v)},F_{t-1}^{(BEV)})$ and $(F_{t}^{(v)},F_{t}^{(BEV)})$ to produce the spatio-temporal features $(F_{t}^{(v,a)},F_{t}^{(BEV,a)})$. Because these two features are temporally correlated but not exactly identical, they should be combined in different proportions, depending on their relevance to the end task. For this goal, we employ a gated attention mechanism, which adaptively balances the contributions of $(F_{t-1}^{(v)},F_{t-1}^{(BEV)})$ and $(F_{t}^{(v)},F_{t}^{(BEV)})$ by multiplying the learnable weight maps $A_{t-1}$ and $A_{t}$. This operation will be described in detail later. The detector then fuses the camera features ${F}_{t}^{(v,a)}$ and LiDAR features ${F}_{t}^{(BEV,a)}$ into ${F}_{t}^{(a)}$, following the procedure of 3D-CVF (Yoo et al. 2020).

Based on the features ${F}_{t}^{(a)}$ produced by SFANet, the RPN predicts the object box and objectness scores for each anchor box. Subsequently, the refinement stage refines the box coordinates and computes the classification scores. Our 3D DetecTrack enhances the RPN and refinement stages by utilizing the information obtained from the tracklet $T_{t-1}$. The Trk-RPN calibrates the objectness score based on the IoU between each anchor and its nearest tracklet in the BEV domain. The objects maintained in the tracklet are supposed to increase the likelihood of objects being detected in the vicinity, which implies that the tracker assists with the detection task. The Trk-RefNet aggregates the instance-level features at time $t$ and $t-1$ based on the cosine similarity-based attention and adjusts the classification score based on IoU between the anchor and the nearest tracklet.

Tracking stage: The tracking stage associates the detection results $\mathcal{D}_{t}$ with the tracklet $T_{t-1}$ based on the spatio-temporal features obtained in the detection stage. The 3D boxes for the detected objects in $\mathcal{D}_{t}$ are projected into the BEV domain to produce 2D boxes. Then, 2D RoI pooling is performed for the $i$th object to extract the RoI-aligned features, $f_{t,i}^{(v,a)}$ and $f_{t,i}^{(BEV,a)}$ from $F_{t}^{(v,a)}$ and $F_{t}^{(BEV,a)}$, respectively. In addition, the tracker pools the point-encoded features $f_{t,i}^{(p)}$ via 3D-RoI alignment method proposed in (Yoo et al. 2020). These features are concatenated as $f_{t,i}^{(\mathcal{D})}=(f_{t,i}^{(v,a)},f_{t,i}^{(BEV,a)},f_{t,i}^{(p)})$. Similarly, a similar feature pooling procedure is applied for the $i$th object in the tracklet $T_{t-1}$. This yields the concatenated features $f_{t-1,i}^{(T)}=(f_{t-1,i}^{(v,a)},f_{t-1,i}^{(BEV,a)},f_{t-1,i}^{(p)})$. Two feature groups $f_{t,i}^{(\mathcal{D})}$ and $f_{t-1,i}^{(T)}$ serve for the discriminative features used for object association.

Two features $(f_{t,1}^{(\mathcal{D})},...,f_{t,N_{D}}^{(\mathcal{D})})$ and $(f_{t-1,1}^{(T)},...,f_{t-1,N_{T}}^{(T)})$ are fed into the SG-GNN. These features are represented by nodes in the graph. The SG-GNN associates the objects in $\mathcal{D}_{t}$ with those in $T_{t-1}$ by matching $(f_{t,1}^{(\mathcal{D})},...,f_{t,N_{D}}^{(\mathcal{D})})$ and $(f_{t-1,1}^{(T)},...,f_{t-1,N_{T}}^{(T)})$. We consider a fully connected graph to model the spatial and temporal relations among the objects in $\mathcal{D}_{t}$ and $T_{t-1}$. Because these dense connections can lead to unnecessary feature exchanges between nodes, we devise rule-based pruning and attention-based edge gating to improve SG-GNN. Finally, the SG-GNN produces an affinity matrix based on the pairwise association score for all edges connecting the nodes from $\mathcal{D}_{t}$ and those from $T_{t-1}$. The affinity matrix is then processed by the Hungarian algorithm (Kuhn 1955) to output the final tracklet $T_{t}$. The entire procedure is repeated until the input sequence is complete.

The SFANet selectively combines two spatial feature maps obtained in two adjacent time steps.

Suppose that there exist two feature maps ${F}_{t-1}^{(\cdot)}$ and ${F}_{t}^{(\cdot)}$ of the same size $C\times X\times Y$. The SFANet applies the weighted aggregation of ${F}_{t-1}^{(\cdot)}$ and ${F}_{t}^{(\cdot)}$ as

$\displaystyle{F}_{t}^{(\cdot,a)}={A}_{t}\otimes{F}_{t}^{(\cdot)}+{A}_{t-1}\otimes{F}_{t-1}^{(\cdot)},$

(1)

where $\otimes$ denotes pixel-wise multiplication and the attention maps ${A}_{t}$ and ${A}_{t-1}$ have a size of $1\times X\times Y$. The attention maps ${A}_{t}$ and ${A}_{t-1}$ are computed as

	$$\displaystyle{A}_{t}=\sigma(\mathrm{conv_{3\times 3}}({F}_{t}^{(\cdot)}\oplus{F}_{t-1}^{(\cdot)}))$$		(2)
	$$\displaystyle{A}_{t-1}=1-{A}_{t}$$		(3)

where $\sigma(\cdot)$ is the logistic-sigmoid function, ${\mathrm{conv_{3\times 3}}}$ are the convolutional layers with $3\times 3$ kernels, and the operation $\oplus$ denotes concatenation.

The detailed structures of Trk-RPN and Trk-RefNet are depicted in Figure 2 (a) and (b), respectively.

Trk-RPN: Trk-RPN produces $P$ region proposals $\mathcal{R}_{1},...,\mathcal{R}_{P}$ based on the feature maps ${F}_{t}^{(a)}$ obtained by the SFANet. For each anchor, the initial objectness score and 3D box coordinates are predicted through a 1$\times$1 convolution. The objectness score is calibrated using the IoU between the anchor box and tracklet boxes in $T_{t-1}$ in BEV domain. For each anchor, IoU is obtained as the highest IoU among all tracklet boxes. IoU is set to 0 if the anchor does not overlap with any tracklet box. The input feature, initial objectness score, and IoU are concatenated and passed through the additional feedforward neural network to output the adjusted objectness score. Because higher IoU values indicate a higher chance of an anchor being positive, they are expected to increase the objectness score of the anchor.

Trk-RefNet: Trk-RefNet refines the 3D box prediction and classification score for all region proposals found by Trk-RPN. Based on the $j$th region proposal $\mathcal{R}_{j}$, the RoI-aligned features $g_{t,j}^{(v,a)}$ and $g_{t,j}^{(BEV,a)}$ are pooled from $F_{t}^{(v,a)}$ and $F_{t}^{(BEV,a)}$. In addition, 3D-RoI alignment (Yoo et al. 2020) is applied to extract the point-encoded features $g_{t,j}^{(p)}$. These features are concatenated as $g_{t,j}^{(\mathcal{D})}=(g_{t,j}^{(v,a)},g_{t,j}^{(BEV,a)},g_{t,j}^{(p)})$. Whereas SFANet combines feature maps without any alignment, Trk-RefNet spatially aligns object features before performing feature aggregation. Treating $\mathcal{R}_{j}$ as an anchor, Trk-RPN predicts the 3D bounding box for the $(t-1)$th time step based on $F_{t-1}^{(a)}$. Then, the concatenated features for the $(t-1)$th time step are obtained as $g_{t-1,j}^{(\mathcal{D})}=(g_{t-1,j}^{(v,a)},g_{t-1,j}^{(BEV,a)},g_{t-1,j}^{(p)})$. Finally, the instance-level features, $g_{t,j}^{(\mathcal{D})}$ and $g_{t-1,j}^{(\mathcal{D})}$ are aggregated as

$\displaystyle h_{t,j}=g_{t,j}^{(\mathcal{D})}+w(g_{t,j}^{(\mathcal{D})},g_{t-1,j}^{(\mathcal{D})})g_{t-1,j}^{(\mathcal{D})},$

(4)

$w(g_{t,j}^{(\mathcal{D})},g_{t-1,j}^{(\mathcal{D})})$ denotes the cosine similarity defined as

$\displaystyle w(g_{t,j}^{(\mathcal{D})},g_{t-1,j}^{(\mathcal{D})})=\frac{\phi(g_{t,j}^{(\mathcal{D})})\odot\phi(g_{t-1,j}^{(\mathcal{D})})}{\left\lVert\phi(g_{t,j}^{(\mathcal{D})})\right\rVert_{2}\left\lVert\phi(g_{t-1,j}^{(\mathcal{D})})\right\rVert_{2}},$

(5)

where $\phi(\cdot)$ denotes the global average pooling operation, and $\odot$ denotes the inner product operation. The cosine similarity measures the correlation between two detection results and is used to adjust the weight applied to the features at the $(t-1)$th time step.

Trk-RefNet calibrates the classification score by applying the fully-connected layers to the initial classification score concatenated with the IoU between the region proposal and the nearest tracklet box.

Spatio-temporal graph: The graph models the relation among the objects in $\mathcal{D}_{t}$ and $T_{t-1}$. According to the 3D boxes in $\mathcal{D}_{t}$ and $T_{t-1}$, RoI pooling is performed to obtain the RoI-aligned features $(f_{t,1}^{(\mathcal{D})},...,f_{t,N_{D}}^{(\mathcal{D})})$ and $(f_{t-1,1}^{(T)},...,f_{t-1,N_{T}}^{(T)})$. These RoI-aligned features are represented by nodes, and their explicit pairwise relationships are encoded by the edges. Previous works (Weng et al. 2020b; Wang, Kitani, and Weng 2020) used a GNN connecting the nodes at the $t$th time step with those at the $(t-1)$th time step. In our work, we consider a spatio-temporal graph, which additionally connects the nodes within the nodes at the $t$th time step as well as those within the nodes at the $(t-1)$th time step. This captures the spatial relation between the objects. Our fully-connected graph is motivated by the idea that understanding the spatial relation between objects will help better associate the objects temporally. However, owing to the dense connections of the graph, a GNN may require unnecessary exchanges of the features, thereby leading to a higher convergence speed. To address this issue, we apply rule-based edge pruning and attention-based edge gating to SG-GNN, as described below.

Rule-based edge pruning: Rule-based edge pruning discards unnecessary edges from a graph based on the spatial distance between objects. Two types of rules are applied. First, the edges connecting the nodes at the $t$th time step are pruned if the Euclidean distance between the object centers in the BEV domain is larger than the threshold $L_{s}$ meter. Similarly, the edges connecting the nodes at the $(t-1)$th time step are pruned according to the same condition. This implies that the SG-GNN only attends to the spatial relation between nearby objects. Second, the edges connecting the nodes between the $t$th time step and the $(t-1)$th time step are pruned if the Euclidean distance between objects is larger than $L_{t}$ meter. This design is justified by the fact that objects that are distant from each other are less likely to be associated. These pruning rules can simplify the node connectivity, thereby requiring much fewer iterations. In our experiments, we set $L_{s}=15$ meter and $L_{t}=5$ meter based on an empirical study.

Attention-based edge gating: We also apply attention-based edge gating, in which an attention mechanism is used to attend the model only on the influential edges. This method adaptively adjusts the weight for the edge based on the similarity of the two features associated with the nodes at both ends. The basic node feature aggregation step follows the rule suggested in (Morris et al. 2019). Consider $N$ source nodes $n_{A,1},...,n_{A,N}$ and a single target node $n_{B}$. In the intermediate iteration of GNN, we have the features $f_{A,1},...,f_{A,N}$ and $f_{B}$ at the nodes $n_{A,1},...,n_{A,N}$ and $n_{B}$, respectively. The attention weight $a_{i}$ is applied to the directed edge $E_{n_{A,i}\rightarrow n_{B}}$ from $n_{A,i}$ to $n_{B}$ during node feature aggregation. Specifically, $a_{i}$ is multiplied by the features $f_{A,i}$ when aggregating the features $f_{A,1},...,f_{A,N}$ at node $n_{B}$. The attention weight $a_{i}$ is calculated by

	$\displaystyle s_{k}$	$\displaystyle=\frac{f_{A,k}\circ f_{B}}{{\left\lVert{f}_{A,k}\right\rVert_{2}\left\lVert{f}_{B}\right\rVert_{2}}}$		(6)
	$\displaystyle a_{i}$	$\displaystyle=\frac{e^{w\cdot s_{i}}}{\sum_{k=1}^{N}e^{w\cdot s_{k}}}$		(7)

where $\circ$ denotes the dot product, and $w$ is the learnable parameter. This attention weight reduces the influence of edges in which the two features are not aligned well. This enables the SG-GNN to focus only on the association of important features.

After a fixed number of iterations, the SG-GNN ends up with the features $(f^{\prime}_{t,1},\cdots,f^{\prime}_{t,N_{D}})$ and $(f^{\prime}_{t-1,1},\cdots,f^{\prime}_{t-1,N_{T}})$ at the nodes. The affinity matrix $A$ is constructed based on the association score between the objects in $\mathcal{D}_{t}$ and those in $T_{t-1}$. The $(i,j)$th element of $A$ is provided by the association score $A_{ij}$ between $f^{\prime}_{t-1,i}$ and $f^{\prime}_{t-1,j}$, which is calculated by

$$\displaystyle A_{ij}=\sigma(\mathrm{fc}(f^{\prime}_{t-1,i}\otimes f^{\prime}_{t,j}))$$

(8)

where $\mathrm{fc}$ represents fully connected layers with a depth of 3. The association score has a value between 0 and 1 and the affinity matrix $A$ is fed to the Hungarian algorithm (Kuhn 1955) to determine the tracklet $T_{t}$.

We adopt a multi-task loss to train the 3D DetecTrack. The total loss function comprises the detection loss $L_{\rm det}$ and tracking loss $L_{\rm trk}$, i.e., $L_{\rm total}=L_{\rm det}+L_{\rm trk}$. The detection loss $L_{\rm det}$ is expressed as

$\displaystyle L_{\rm det}=L_{\rm rpn}+L_{\rm ref}$

(9)

where $L_{\rm rpn}$ denotes the RPN loss used to train the network pipeline up to Trk-RPN, and $L_{\rm ref}$ denotes the refinement loss in training Trk-RefNet. Following the setup in (Yoo et al. 2020), the RPN loss $L_{\rm rpn}$ comprises the focal loss (Lin et al. 2017) for the classification task and the Smoothed-L1 loss for the regression task. The refinement loss $L_{\rm ref}$ is defined similarly.

The tracking loss $L_{\rm trk}$ measures the mean squared error (MSE) of the affinity matrix $A$, defined as

$\displaystyle L_{\rm trk}=\frac{1}{N_{D}\cdot N_{T}}\sum_{i=1}^{N_{T}}\sum_{j=1}^{N_{D}}(A_{ij}-A^{gt}_{ij})^{2}.$

(10)

Note that $A^{gt}$ represents the ground truth (GT) of the affinity matrix $A$. The GT affinity matrix $A^{gt}$ is calculated in two steps. In the first step, the 3D IoU between the output $D_{t}$ and its GT boxes is calculated and the object ID of the GT box is assigned to the object in $D_{t}$ if the highest 3D IoU is above 0.5. Comparing the object IDs in $T_{t-1}$ and those assigned to $D_{t}$, we assign 0 or 1 to the GT affinity matrix $A^{gt}$. In the training phase, our 3D DetecTrack model regards the GT for $D_{t-1}$ as the tracklet $T_{t-1}$.

Dataset and evaluation metrics: The KITTI dataset was collected from urban driving scenarios using a single Pointgrey camera and Velodyne HDL-64E LiDAR (Geiger, Lenz, and Urtasun 2012). To validate our method, we split the tracking training dataset evenly into train set and valid set by half, following (Weng et al. 2020b). Because the KITTI object detection dataset did not contain sequence data, performance was evluated only on the KITTI object tracking dataset. As a benchmark, we evaluated the 3D MOT performance on the KITTI object tracking valid dataset. As 3D MOT performance metrics, we used sAMOTA, AMOTA, and AMOTP metrics (Weng et al. 2020a) as well as the standard CLEAR metric (Bernardin and Stiefelhagen 2008). We also examine the 2D MOT performance on the KITTI object tracking test dataset for the Car category. For 2D MOT, we used the KITTI evaluation metric (Luiten et al. 2020) including HOTA, DetA, and AssA. Because our method was designed under the JoDT framework, we also evaluated the 3D detection performance. As a performance metric, we used the average precision (AP) with three difficulty levels, i.e., ”easy”, ”moderate”, and ”hard” as suggested in (Geiger, Lenz, and Urtasun 2012).

Implementation details: Our 3D DetecTrack model was trained in two steps. In the first step, we trained only the detection stage of the 3D DetecTrack using both the KITTI object detection and tracking datasets. The detection stage requires tracklet $T_{t-1}$ for training. Since the detection dataset does not contain sequence data, the sequence was created by copying the same data. We trained the detection stage following the setup suggested in (Yoo et al. 2020). In the second step, the detection stage was initialized with the pre-trained model and the entire 3D DetecTrack model was trained end-to-end on the KITTI object tracking dataset. We trained the entire network over $40$ epochs using the ADAM optimizer (Kingma and Ba 2014). The initial learning rate was set to $10^{-4}$ and decayed by a factor of $0.1$ at the $26$th and $35$th epochs. The weight decay parameter was set to $10^{-4}$, and the mini-batch size was set to $4$.

Dataset and evaluation metrics: The nuScenes dataset is a large scale autonomous driving dataset which contains more than 1,000 driving scenarios. The dataset was collected using six multi-view cameras, 32-channel LiDAR, and 360-degree object annotations. We evaluated 3D MOT performance for 7 categories (bicycle, bus, car, motorcycle, pedestrian, trailer and truck), as a subset of the detection categories in (Caesar et al. 2020). We also evaluated the 3D detection performance for 10 categories, including barrier, construction vehicle, and traffic cone as well as the aforementioned 7 categories. As 3D MOT performance metrics, we used the standard CLEAR metric (Bernardin and Stiefelhagen 2008) and the sAMOTA, AMOTA, and AMOTP metrics (Weng et al. 2020a). We used the nuScenes detection score (NDS) (Caesar et al. 2020) as a 3D detection performance metric.

Implementation details: We trained the 3D DetecTrack using a similar procedure similar to that used for the KITTI benchmark. We trained the entire model over 20 epochs. The initial learning rate was set to $10^{-4}$ and decayed by a factor of 0.1 at the $13$th and $17$th epochs. The rest of configurations were the same as those for KITTI.

Performance on KITTI: Table 1 presents the 3D MOT performance and runtime evaluated on the KITTI tracking valid set. The KITTI benchmark provides a test dataset for 2D MOT, but not for 3D MOT; therefore, we strictly followed the 3D MOT evaluation procedure presented in (Weng et al. 2020a). We compared our 3D DetecTrack with several outstanding 3D MOT methods including FlowMOT (Zhai et al. 2020), AB3DMOT (Weng et al. 2020a), mmMOT (Zhang et al. 2019), FANTrack (Baser et al. 2019), GNN3DMOT (Weng et al. 2020b), the method of (Weng, Yuan, and Kitani 2020), and PC-TCNN (Wu et al. 2021). These 3D MOT methods adopt different 3D detectors. Thus, as shown in Table 1, we also evaluate the performance of mmMOT (Zhang et al. 2019) and AB3DMOT (Weng et al. 2020a) combined with the vanilla 3D-CVF to compare the ability of the trackers only. Table 1 shows that the proposed method outperforms the existing 3D MOT methods by a significant margin for all MOT metrics considered. In particular, our 3D DetecTrack performs better than the current state-of-the-art method, PC-TCNN. The proposed 3D DetecTrack also achieves better performance than AB3DMOT (Weng et al. 2020a) and mmMOT (Zhang et al. 2019) when 3D-CVF is used as a 3D detector.

We also evaluate the 2D MOT performance on the KITTI object tracking test set (refer to the official evaluation benchmark on the KITTI leaderboard). Due to space concerns, we provide the 2D MOT performance in the Appendix. 2D MOT results are obtained by projecting 3D bounding boxes to the camera domain and generating 2D bounding boxes enclosing the projected coordinates. Although the proposed 3D DetecTrack is not originally designed for 2D MOT task, it achieves comparable performance to that of the current state-of-the-art 2D MOT methods. Note that the runtime of the tracker in the proposed method (37ms) is comparable to that of the existing tracking methods.

Performance on nuScenes: Table 2 presents the 3D MOT performance on nuScenes validation set. The baseline algorithm uses the original 3D-CVF and the vanilla fully-connected GNN. The proposed approach improves on the baseline by 3.74% in mAP and 6.12%, 2.44% and 3.52% in sAMOTA, AMOTA, and AMOTP, respectively. The 3D DetecTrack achieves the best performance among the candidates in the sAMOTA, AMOTA, and MOTA metrics. The per-class performance on both 3D detection and MOT tasks is provided in the Appendix. This shows that the proposed method achieves a particularly remarkable performance gain on bicycle, motorcycle, bus, and truck categories.

In this section, we present an ablation study conducted to validate the contributions of the proposed design to our 3D DetecTrack. Experiments were conducted on the KITTI valid set. Our baseline was chosen as a combination of the original 3D-CVF and the baseline GNN. Table 3 presents how much the components of the 3D DetecTrack, SG-GNN, SFANet, Trk-RPN, and Trk-RefNet improve the performance on both 3D detection and MOT tasks. When the SG-GNN is added to the baseline, both temporal and spatial object context is exploited for object association, improving the sAMOTA performance by 2.82%. Both Trk-RPN and Trk-RefNet improve both the detection and MOT performance by utilizing the information from the tracklets. Trk-RPN yields a 0.59% gain in ${\rm AP}_{mod.}$ detection performance and a 0.59% gain in sAMOTA MOT performance. Trk-RefNet achieves a 0.36% gain in ${\rm AP}_{mod.}$ and a 0.46% gain in sAMOTA. Totally, both Trk-RPN and Trk-RefNet achieve 0.95% and 1.05% gains in ${\rm AP}_{mod.}$ and sAMOTA metrics, respectively. SFANet also improves the performance by 0.69% in ${\rm AP}_{mod.}$ and 0.47% in sAMOTA by aggregating adjacent feature maps over time. Combining all the ideas, the proposed method improves ${\rm AP}_{mod.}$ performance by 1.64% and sAMOTA performance by 4.34% over the baseline, which appears to be substantial.

Table 4 analyzes the performance gains achieved by the rule-based edge pruning and attention-based edge gating. We evaluate the performance gains achieved by adding each idea to a fully-connected GNN. The rule-based edge pruning offers a 1.48% improvement in sAMOTA by removing unnecessary connections from the fully-connected graph. The attention-based edge gating weights the GNN features according to their importance, which offers additional gain of 0.88% in sAMOTA. Totally, the combination of the two increases sAMOTA by 2.36% over the baseline GNN. We also analyze the runtime of the SG-GNN. While the rule-based pruning method reduces the computation time, the attention-based gating method increases the computation time for better performance. Overall, the SG-GNN reduces the runtime of the baseline GNN by 10%.