SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection

Existing LiDAR-based 3D object detectors commonly leverage sparse convolutions for data processing to enhance computational efficiency. There are primarily two types of sparse convolutions used: submanifold sparse convolution [31], which maintains feature sparsity between input and output feature maps, and regular sparse convolution [32], which increases the density of the feature map by expanding features into neighboring regions. Since regular sparse convolution decreases feature sparsity dramatically, it is often merely used to down-sample feature maps in existing methods [18, 5, 4, 7].

Most voxel-based methods [18, 4, 7] adopt sparse CNNs to extract features. These CNNs typically comprise a series of sparse residual blocks, where each block contains two submanifold sparse convolutions and a skip connection linking its input and output.

Since submanifold sparse convolutions preserve feature sparsity from input to output, they may impede information exchange among spatially distant features. As a result, merely stacking SRBs can result in a receptive field with limited size. HEDNet [7] addresses this by incorporating sparse encoder-decoder blocks that capture long-range dependencies among features while maintaining computational efficiency. Figure 3 illustrate a general structure of EDB. It decreases the spatial distance among distant features via feature down-sampling and subsequently restores lost details through multi-scale feature fusion. By applying 3D and 2D submanifold sparse convolutions to construct the SRB, we can obtain 3D-EDB and 2D-EDB, respectively.

Taking raw point clouds as input, the 3D sparse backbone initially extracts sparse feature maps using a voxel feature encoder (VFE) and progressively down-samples them to extract high-level features. At the end of the backbone, it incorporates a 3D-EDB to facilitate information exchange among distant features. Subsequently, the 3D sparse features are compressed into 2D sparse BEV features. This compression is implemented by using two regular sparse convolutions with stride 2 to down-sample features along the Z-axis and then aggregating features of voxels sharing the same coordinates in BEV.

Taking the BEV sparse features as input, the 2D sparse backbone begins by performing voxel classification on each voxel to determine whether the geometric center of each voxel falls within an object bounding box of a specific category or belongs to the background. Subsequently, an adaptive feature diffusion (AFD) operation, in conjunction with a 2D-EDB, is employed to propagate voxel features towards object centers.

Since most high-performing hybrid detectors utilize the center-based head introduced by CenterPoint [4] or TransFusion-L [5], we adopt a similar head in SAFDNet. However, as the designs of these detection heads are tailored for dense feature maps, we have made some adjustments to accommodate sparse features. For more details, please refer to Section 3.4.

Is it possible for detectors to extract features nearer or at object centers while maintaining feature sparsity as much as possible? An intuitive idea is to diffuse sparse features to neighboring voxels rather than all voxels like hybrid detectors. Figure 4 (a) depicts a uniform feature diffusion strategy, where input voxel features are expanded to a $K\times K$ neighborhood, with $K$ set to 5 as an example. There are two possible implementations:

1. a.

   Parameter-based (PB) UFD: employing a regular sparse convolution with kernel size $K\times K$ to spread features, in conjunction with a 2D-EDB for further transformation.

2. b.

   Parameter-free (PF) UFD: initializing zero features in neighboring regions and then incorporating a 2D-EDB to spread features progressively.

Through our analysis of the sparse voxels output by the 3D sparse backbone, we observed that: (a) fewer than 10% of the voxels fall within the bounding boxes of objects; (b) smaller objects often have voxel features near or at their centers. This indicates potential redundancy in uniformly expanding features into neighborhoods of the same size, particularly for voxels within the bounding boxes of small objects and those belonging to the background. Hence, we propose an adaptive feature diffusion strategy according to voxel positions, as depicted in Figure 4 (b). The idea is to assign a larger diffusion range to voxels within the bounding boxes of large objects to bring features nearer to object centers, while assigning a smaller range to voxels within the bounding boxes of small objects or the background to preserve feature sparsity. Implementing this idea necessitates voxel classification to determine whether a voxel’s center is within the bounding box of an object of a specific category or belongs to the background.

Given $\mathbf{P}_{i}$, a binary mask $\mathbf{M}_{i}$ that indicates voxels whether fall within any object bounding boxes of category group $i$ is calculated by threshold $t$, then the binary mask $\mathbf{R}_{i}$ that indicates the feature diffusion areas can be jointly decided by $\mathbf{M}_{i}$ and the diffusion kernel size $K_{i}\kern-1.99997pt\times\kern-1.99997ptK_{i}$. $K_{i}$ is defined as $\alpha\cdot S_{i}$, where $S_{i}$ is the average size of objects in category group $i$ and $\alpha$ controls the range of feature diffusion. The object size is defined as the maximum value between the length and width of the bounding box and is normalized by the voxel size. After that, the binary mask representing the overall feature diffusion areas can be calculated as $\mathbf{R}=\mathbf{R}_{1}\cup\mathbf{R}_{2}\cup...\cup\mathbf{R}_{G}$. For each voxel in the feature diffusion areas, if there are no features, we initialize it with zero features. Finally, a 2D-EDB is utilized for further feature transformation.

On the validation set, SAFDNet achieved 75.7% L2 mAP and 73.9% L2 mAPH, performing slightly better than the hybrid detector HEDNet and the sparse detector FSDv2. On the test set, SAFDNet yielded 1.3% L2 mAPH gains over the previous best fully sparse detector FSDv2. Notably, SAFDNet achieved significant improvements over FSDv2 on the large vehicle category (2.2% L2 mAPH on the test set), which suffers more from the center feature missing problem with the full sparse architecture. More comparison on model runtime has been presented later.

We primarily compared SAFDNet with previous top-performing LiDAR-based methods on the nuScenes test set (Table 3). On the nuScenes test set, SAFDNet achieved impressive results with 72.3% NDS and 68.3% mAP. Compared with FSDv2, SAFDNet showcased significant improvements on the categories of large objects including car (+3.6%), truck (+5.7%), and trailer (+4.6%). These results further demonstrate the effectiveness of our method.

To validate the effectiveness of SAFDNet on the long-range detection, we conducted experiments on the Argoverse2 dataset with a perception range of 200 meters (Table 4). SAFDNet achieved a gain of 2.1% mAP over the previous best sparse detector FSDv2. It is worth noting that the proposed SAFDNet also outperformed the hybrid detector HEDNet, which indicates that expanding features towards all unoccupied area without restriction may hurt the detection performance.

We compared the inference speed of SAFDNet with prior top-performing methods, as shown in Table 5. On the Waymo Open dataset with a short perception range, SAFDNet was 2$\times$ faster than the sparse detector FSDv2 and was 1.2$\times$ faster than the hybrid detector HEDNet. On the Argoverse2 dataset with a long perception range, SAFDNet yielded 2.1% mAP gains over FSDv2 while being 1.3$\times$ faster. Compared with HEDNet, SAFDNet achieved 2.6% mAP improvements while being 2.1$\times$ faster. Note that the hybrid detector HEDNet requires much more training memory than the other sparse detectors.

We performed step-wise ablation experiments from the hybrid detector HEDNet to our SAFDNet on the Waymo Open dataset (Table 6 (a)). Initially, we removed the 2D dense backbone in HEDNet and replaced all convolutions in the detection head with submanifold sparse convolutions, resulting in the first model. The accuracies across all categories dropped significantly. We observed that using the voxels within which object centers fall, as done in CenterPoint, to calculate the Gaussian heatmap during classification training led to rapid convergence of the classification loss to zero. In the second model, we used the nearest non-empty voxel as the center to generate the Gaussian heatmap. This adjustment notably increased accuracy to 71.0% mAPH. In the third model, we continued to adopt the center voxel but normalized the training target with the maximum values to prevent gradient vanishing, resulting in a slight accuracy boost. Adding the 2D-EDB in the fourth model did not lead to a performance improvement. Notably, there still existed a significant accuracy gap, particularly on large vehicles, between the fourth model and HEDNet (over 3% mAPH). Finally, incorporating the proposed adaptive feature diffusion strategy effectively bridged this gap. An ablation on different sparse backbones is presented in Appendix B.

We compared different type of diffusion strategies on the Waymo Open and Argoverse2 datasets, as presented in Table 6 (b). While the models with the two types of UFD strategies showed substantial gains over the model without feature diffusion, they introduced significantly higher computational FLOPs. In contrast, the model with the AFD strategy achieved the highest accuracy while incurring fewer computational costs.

We compared different diffusion ranges of AFD on the Waymo Open and Argoverse2 datasets. Table 6 (c) shows that, setting $\alpha$ to 1, wherein the diffusion kernel size matches the average object size, resulted in the highest accuracy for SAFDNet. On the Waymo Open dataset, a large diffusion range slightly degraded the performance, suggesting that excessive feature diffusion might be unnecessary and could harm performance.