Ada3D : Exploiting the Spatial Redundancy with
Adaptive Inference for Efficient 3D Object Detection

Voxel-based methods convert the point cloud into regular grids. SECOND [30] utilizes the 3D sparse convolution for feature extraction. CenterPoint [33] is a single-stage detector that leverages a keypoint detector to detect box centers. PV-RCNN [20] combines the point and voxel features and utilizes a two-staged framework for precise detection. While the voxel-based detectors achieve state-of-the-art results, their high computational and memory costs impede their application on self-driving cars. Ada3D aims to alleviate this issue through adaptive inference.

In the field of 2D perception, adaptive inference methods reduce spatial redundancy for 2D images. Figurnov et. al. [5] dynamically adjust depth for different regions. A series of methods [11, 29, 8] learn to adaptively skip redundant channel/pixels. GFNet [12] employs reinforcement learning to locate the discriminant regions. Ada3D applies adaptive inference to the 3D perception, and adaptively filters redundant 3D voxels and BEV features.

Some prior studies aim to enhance the efficiency of 3D detectors. SPVNAS [26] employs neural architecture search to search to find suitable depth and width for the 3D model. Lee et. al. [10] propose a point-distribution pruning method on 3D convolution kernel. SPS-Conv [16] prunes the output mapping for sparse convolution based on the feature magnitude. RSN [23] designs network module to prune unnecessary part in range view lidar image. A series of fully sparse detectors: FSD [3], FSD++ [4], VoxelNeXT [1] design novel architectures that eliminate the dense BEV backbone. These methods optimize the efficiency of 3D detectors from the perspective of compressing model redundancy. Differently, Ada3D focuses on reducing spatial redundancy and could work on par with these methods.

Figure 3 illustrates the overall framework of Ada3D. The 3D object detection task aims to predict 3D bounding boxes $\mathcal{B}=\{b_{k}\}$ from the point cloud $\mathcal{P}=\{(x,y,z,r)_{i}\}$. The voxel-based 3D detectors [20, 30, 33] quantize the point cloud into regular grids. Without loss of generality, we omit the batch dimension in the following equations. The voxelization generates sparse voxels $\mathbf{X_{3d}}\in\mathbb{R}^{N\times C}$ of voxel numbers $N$ and feature channels $C$. The 3D voxel backbone $\mathcal{F}_{3d}$ applies 3D sparse convolution [6] on the voxels to extract point cloud feature. We use the $\mathbf{X}_{i,c}$ to represent the $c$-th channel of $i$-th voxel feature, and the $c^{\prime}$ channel of the $j$-th output voxel can be described as:

$$\begin{split}\mathbf{Y}_{j,c^{\prime}}=\sum_{k}\sum_{c}W_{k,c,c^{\prime}}X_{R_{k,j},k,c},\\ \end{split}$$

(1)

where $R_{k,j}$ is the input index $i$ given the output index $j$ and kernel offset $k$, $W_{k,c,c^{\prime}}$ denotes the kernel offset $k$’s weight.

The processed 3D feature $\mathbf{\tilde{X}}_{\text{3D}}$ is then projected to the BEV plane through sum pooling along the z-axis to generate 2D features $\mathbf{X}_{\text{2D}}\in\mathbb{R}^{C\times W\times H}$. We define $\Gamma_{3D\to 2D}$ as the mapping from 3D voxels to 2D BEV pixels, and $\Gamma_{2D\to 3D}$ describes the invert mapping. The 2D BEV backbone $\mathbf{F}_{\text{2D}}$ is applied to further extract the 2D BEV feature. Finally, the detection head $\mathbf{F}_{\text{head}}$ predicts the 3D bounding box.

The adaptive inference is adopted in both the 3D and 2D backbone. For simplicity, we omit the channel dimension $C$ for feature $\mathbf{X}$ for the equations below, since all channels share the same spatial filtering pattern. We describe the layer indexes where the predictor is applied with the layer index $\mathcal{I}_{3D}=\{l_{3D}^{(1)},...,l_{3D}^{(n)}\}$ and $\mathcal{I}_{2D}=\{l_{2D}^{(1)},...,l_{2D}^{(n)}\}$. The adaptive inference for 3D backbone could be described as:

$$\begin{split}\mbox{for}\hskip 5.0ptl_{3D}^{(i)}\in\mathcal{I}_{3D}:\mathbf{X}^{(l_{3D}^{(i)})}_{3D}&=\mathbf{F}_{3D}^{(l_{3D}^{(i)})}(\mathbf{\tilde{X}}^{(l_{3D}^{(i)}-1)}_{3D}),\hskip 5.0pt\mbox{where}\\ \mathbf{\tilde{X}}^{(l_{3D}^{(i)}-1)}_{3D}=\Gamma_{2D\to 3D}(\mathbf{F}_{\text{drop}}&(\Gamma_{3D\to 2D}(\mathbf{X}^{(l_{3D}^{(i)}-1)}_{3D}),\mathbf{S}))\odot\mathbf{X}^{(l_{3D}^{(i)}-1)}_{3D},\\ \mathbf{S}=\mathbf{F}_{\text{score}}&(\Gamma_{3D\to 2D}(\mathbf{X}^{(l_{3D}^{(i)}-1)}_{3D})),\\ \end{split}$$

(2)

where the $\mathbf{S}\in\mathbb{R}^{W\times H}$ represents the importance score for BEV pixels, which is generated by $\mathbf{F}_{\text{score}}$ that combines the predictor output and 3D point cloud’s density. The $\mathbf{F}_{\text{score}}$ takes the 2D BEV input projected from the input 3D voxel feature $\mathbf{X}^{(l_{3D}^{(i)}-1)}_{3D}\in\mathbb{R}^{N}$. Given the drop ratio $R_{\text{drop}}$, the spatial filtering process $\mathbf{F}_{\text{drop}}$ drops the most redundant portion of features in the BEV space based on the importance score $\mathbf{S}$. It generates the one-hot mask that indicates whether the given location should be kept or discarded. The mask is then broadcasted back to the voxel space through $\Gamma_{2D\to 3D}$ and element-wisely multiplied with the original 3D voxel feature to generate subsampled 3D voxel feature $\mathbf{\tilde{X}}^{(l_{3D}^{(i)}-1)}_{3D}$. Note that the equation describes the algorithmic simulation of spatial filtering, while in the actual GPU processing, features with zero values in $\mathbf{\tilde{X}}^{(l_{3D}^{(i)}-1)}_{3D}$ are excluded to achieve actual hardware acceleration, i.e., their computation and storage are skipped. More details about $\mathbf{F}_{\text{drop}}$ and $\mathbf{F}_{\text{score}}$ will be discussed in Sec. 3.2 and Sec. 3.3.

Similarly, the adaptive inference for the 2D BEV backbone is applied at $\mathcal{I}_{2D}=\{l_{2D}^{(1)},...,l_{2D}^{(n)}\}$ layers with similar process described in Equ. 2 without the transformation $\Gamma_{2D\to 3D},\Gamma_{3D\to 2D}$ between the voxel and the BEV space.

As discussed in Equ. 2 in Sec. 3.1, the $\mathbf{F}_{score}$ is used for evaluating the input feature to identify its redundant parts. In Ada3D, we adopt a lightweight CNN as the spatial-wise importance predictor in BEV space to predict pixel-wise importance score from the input feature.

Inference. The predictor inference for 3D voxel feature is described as:

$$\begin{split}\mathbf{Y}_{\text{pred}}=\mathbf{F}_{\text{pred}}((\mathbf{X}_{\text{BEV}});\Theta_{\text{pred}}),\\ \end{split}$$

(3)

where $\mathbf{F}_{\text{pred}}$ is the predictor with the parameter $\Theta_{\text{pred}}$. The predictor’s output is a single channel heatmap $\mathbf{Y}_{\text{pred}}\in\left[0,1\right]^{W\times H}$. We choose to design the predictor in the BEV space instead of 3D space, as the perception is mainly conducted in the former. Intuitively, there exists less redundancy in the vertical space, and the efficiency improvement of compressing it is restricted. Also, estimating the importance of the whole 3D space is more challenging. In order to effectively and efficiently evaluate the importance, we design a lightweight predictor that is shared for different layers at both the 3D and 2D backbone. It consists of multiple group convolutions  [35] with reduced parameters and computational complexity. Besides, the resolution of the predictor is selected as 1/8 of the original original BEV resolution. The computaional cost of the predictor’s is less than 1% of the 2D backbone, thereby bringing negilible overhead.

Training. Our oracle experiment in Fig. 2 shows that the performance only decreases slightly when dropping a notable amount of points outside the ground-truth bounding boxes. It reveals that the center of the bounding box is of high importance and the importance spreads to the local region. Therefore, following CenterPoint [33], we generate the ground-truth heatmap $M_{\text{gt}}$ for the predictor by rendering a 2D Gaussian circle with a peak located at each bounding box center $(u,v)$, which could be formulated as follows:

$$\begin{split}\mathbf{M}_{\text{gt}}=\sum_{b_{i}}\mathcal{G}((u,v)_{i},\sigma),\\ \end{split}$$

(4)

where $b_{i}$ is the ground-truth bounding box, and $\mathcal{G}$ is the 2D gaussian function with radius $\sigma$. The mean squared error (MSE) loss is adopted for predictor training.

The spatial filtering $\mathbf{F}_{drop}$ in Equ. 2 in Sec. 3.1 describes the process of dropping the most redundant $R_{\text{drop}}$ of the input features based on the importance criterion $\mathbf{S}$. We combine the predictor score with the point cloud density to determine where to drop.

The predictor score $\mathbf{Y}_{pred}$ could effectively represent the relative importance of the input feature. However, due to the imaging principle of the Lidar sensor, the point cloud closer to the sensor has a larger density, and the remote part is sparse [38]. Due to the neighboring aggregation characteristic of the convolution, the predictor tends to output higher results for denser regions and could miss the remote objects (as shown in Fig. 9). To compensate for this bias, we propose density-guided spatial filtering that takes the unique properties of the Lidar point cloud into consideration. Specifically, we use the point cloud BEV density to adjust the predictor score. Therefore, the importance criterion $\mathbf{S}$ is calculated as follows:

$$\begin{split}\mathbf{S}=\mathbf{F}_{score}(\mathbf{X}_{BEV})=\mathbf{F}_{pred}(\mathbf{X};\Theta_{pred})\cdot(\mathbf{D}_{g})^{\beta},\end{split}$$

(5)

where $D$ is the density heatmap pooled with kernel size of $g$, and $\beta$ is a hyperparameter that tunes the density distribution. The value of $\beta$ is selected for each dataset with the goal of aligning the variance of the predictor score and density distribution on 10 sampled scenes. Example in Fig. 9 demostrates that the density guidance enlarges the importance score for sparser regions and avoids mistakenly dropping the remote objects.

As illustrated in Fig. 4, the 2D feature map projected from 3D voxel features in the BEV plane is sparse, only 5% and 20% features are nonzero for KITTI and nuScenes (the orange part). The rest of the background pixels (the blue ones) are initialized as zero. However, current methods do not utilize such sparsity, and the feature map loses sparsity after the first bacth normalization layer (See Fig. 2). A large amount of computation and memory is wasted for the “background” features with limited information.

A straightforward way to preserve sparsity in 2D BEV backbone is to apply batch normalization only for the nonzero elements. This approach is described as the “Nonzero BN” in Fig. 4. However, we empirically discover that replacing the “Normal BN” with “Nonzero BN” causes instability in training and moderate performance degradation when fine-tuning from dense pretrained models. We attribute this problem to the violation of the feature’s relative relations. As shown in Fig. 4, the orange part with diagonal hatching has larger values than the background features (zero), but after the “Nonzero BN”, their values are smaller than the background. The finetuning process needs to learn such distribution change thus causing instability. To address this problem, we propose a simple but effective modification and introduce the “Sparsity-preserving BN”. In order to preserve the features’ relative relations, the SP-BN leaves out the procedure of subtracting the feature’s mean. Therefore, most parts of the nonzero elements remain positive and are distinguishable from the “background”. The fine-tuning process only needs to learn the offset of the background “zero” elements. SP-BN (affine transform omitted) can be formulated as:

$$\begin{split}\hat{x}_{i}^{(k)}=\frac{x_{i}^{(k)}}{\sqrt{(\sigma_{B}^{(k)})^{2}+\epsilon}},\end{split}$$

(6)

where $\sigma_{B}^{(k)}$ is the standard deviation. Experimental results show that when replacing the normal batch normalization with SP-BN, we could increase the sparsity of 2D BEV heatmap from $0\%$ to $50\%$ without loss of performance.

KITTI and nuScenes and ONCE dataset The KITTI dataset has 7481 training images and 7518 test images with corresponding point clouds. The object to detect have 3 classes: car, pedestrian, and cyclist, the boxes are classified into three subsets: “Easy”, “Moderate” and “Hard” based on the levels of difficulty. The detection results are evaluated by average precision (AP) for each subset with IoU threshold 0.7 for cars and 0.5 for pedestrians and cyclists. The nuScenes dataset comprises 1000 driving sequences with annotations in the form of bounding boxes for 10 object classes. The commonly used metrics are the mean Average Precision (mAP) and the nuScenes detection score (NDS). NDS is the weighted average of mAP and other box characteristics, such as translation and orientation. The ONCE [18] dataset provides Lidar point clouds collected from downtown and suburban areas of multiple cities for 3D object detection. For supervised training, the training set contains 5k labelled scenes and the validation set contains 3K scenes. The commonly used mAP is adopted as the evaluation metric.

Adaptive inference design We apply Ada3D to CenterPoint [33] model on both datasets. Due to the original CenterPoint paper does not conduct experiments on KITTI, we follow the author’s released code [31] to construct the CenterPoint model on KITTI. We replace all the batch normalization layers in the 2D backbone with sparsity-preserving BN. We apply adaptive inference at the 2nd and 4th layer of the 3D and 2D backbone. The $R_{drop}$ is a hyperparameter (e.g., 25%/50%) to control how many features to drop. The predictor’s input resolution is set as the scene size divided by voxel size$\times$8 . Max poolings and same padding upsample layers are adopted to align features of different sizes. The predictor consists of 3 convolution layers with channel size $\left[16,32,16\right]$ and group size of 8. The predictor is trained with adam optimizer with one-cycle learning rate shceduling [7] of learning rate 0.003 for 10 epochs. To recover the performance, we adopt an interleaved scheme that alternates between finetuning the model with adaptive inference for 5 (2 for nuScenes) epochs and training the predictor for 1 epoch and repeat this process for a total of 5 times. The $\sigma$ for ground-truth heatmap is 5.0. The density guidance $\beta$ is set as 0.5 and 0.7 for KITTI and nuScenes/ONCE.

Hardware experiments settings We measure the latency and memory usage of convolution layers on an Nvidia RTX 3090 GPU using CUDA 11.1. We implemented sparse convolution operations using the gather-GEMM-scatter dataflow in TorchSparse v2.0.0 [25] and SpConv v.2.2.6 [6]. To measure latency, we synchronized the GPU and recorded the starting and ending times. To measure peak memory usage, we embedded the PyTorch Memory Utils [19] into the engine frontend.

We first present the performance and resource consumption of Ada3D optimized model on KITTI and nuScenes. We estimate the memory cost of the model by summing the intermediate activation sizes following recent literature [24]. As could be seen from Table. 1, the Ada3D optimized model achieves comparable performance with other methods of different paradigms while compressing the model’s computational and memory cost. In Table. 4 and Fig. 5, we present Ada3D model with different drop rates. The model size could be effectively tuned with the drop rate $R_{\text{drop}}$ to fit different resource budgets. “Ada3D-A” model only conducts adaptive inference for 2D backbone, it improves the model performance while reducing the dense rate of BEV features from 100% to 20% . “Ada3D-B” model reduces 40% 3D voxels and more than 80% 2D pixels and compresses the computaional and memory cost of the model by 5$\times$ without performance degradation. “Ada3D-C” model reduces 60% 3D voxels and more than 90% 2D pixels with moderate performance loss, and reduces the model’s computation and memory cost by an order of magnitude. Table. 2 presents the performance on nuScenes, Ada3D optimized CenterPoint model achieves 2$\sim$4$\times$ FLOPs and memory savings with less than 1% performance drop. Compared with methods that focus on reducing the model redundancy (“CenterPoint-0.5W” and “SPSS-Conv”), Ada3D achieves a larger compression rate with less performance drop.

We conduct hardware profiling of the Ada3D model using sparse convolution GPU libraries [25, 6]. Fig.6 illustrates the reduction of GPU latency and peak memory for each layer, while Fig.7 presents the end-to-end hardware specs for the 3D and 2D backbones, respectively. From the results, we draw the following conclusions. First, by using SP-BN and spatial filtering, we retain high sparsity of the 2D feature map, which brings significant reductions in peak memory and computation for the 2D backbone. For instance, the ”conv2d_1” layer shows a 2.5$\times$ latency and 8.5$\times$ memory improvement, and the overall memory of the 2D backbone is reduced by 4.5$\times$, 6.7$\times$, and 1.9$\times$ for each model. Second, the end-to-end latency of the 3D backbone aligns with the drop rate. The latency for the 3D backbone is 0.74$\times$, 0.56$\times$, and 0.77$\times$ of the pre-optimized ones, which corresponds to the drop rate (25%, 50%, 25%). Third, Ada3D is more effective for larger scenes and finer voxel sizes, since there exists more potential for exploiting the spatial sparsity. For the nuScenes Ada3D model, only peak memory optimization is achieved, but the latency remains similar. This is because that due to resource constraints, a larger voxel size is often used at the cost of inferior performance [26]. The nuScenes BEV feature map is processed in a relatively low resolution ($\left[128,128\right]$), thus the dense rates of the deeper layer’s feature maps remain high, and using sparse convolution to process them takes longer than normal convolution. Future directions to improve this include further reducing redundancy or adopting more hardware acceleration techniques. Addtionaly, improving the efficiency could enable finer voxel size, which could in turn enhances performance and safety for safety-critic autonomous driving applicaiton.

Importance predictor accurately evaluates the input features’ importance. In Table. 4, comparing “Ada3D-A” and “SP-BN”, the predictor increases the 2D feature map’s sparsity from 50% to 80% upon SP-BN. As shown in Table. 6, Among the least important 25%/50% predicted, only 1.5%/7.8% features are mistakenly evaluated. Fig. 9 and Fig. 8 present the visualization of predictor heatmaps in both the BEV and voxel space. The predictor recognizes features within the box and avoids dropping them.

Density guidance avoids dropping the remote small-sized objects. In Table. 4, comparing the “Ada3D-B” models with and without density guidance, simply using predictor scores causes notable performance degradation, especially for the pedestrian (-2.4%) with smaller sizes. Fig. 9 shows the example of density guidance correcting the drop of remote small objects. The predictor fails to correctly detect features for box-1,2,5 due to low density, and the density guidance compensates for such error. We also compare different importance criteria under different drop rates $R_{drop}$ in Table. 6. The $R_{inbox}$ denotes the percentage of dropped features that are in the ground-truth bounding box. Solely using the predictor score (IP) or density (DG) results in high $R_{inbox}$ and performance degradation.

SP-BN preserves the sparsity without performance drop. Table. 4 shows that introducing the SP-BN increases the sparsity of 2D BEV features from 0% to 50% with no performance drop. Using the “Noraml BN” sacrifices the sparsity. Additionally, adopting the “Nonzero BN” for the entire network results in notable performance loss when finetuning from pretrained dense backbone. We hypothesize that it is because of the “Nonzero-BN” needs to learn the entire distribution shift, while the “SP-BN” only needs to learn the offset of zero elements.

Ada3D introduces negligible overhead. The extra cost that Ada3D introduces is the predictor inference. The predictor is conducted in a relatively low resolution and utilizes group convolution. The predictor’s computational cost is less than 1% of the 2D BEV backbone, which is negligible. The training cost of Ada3D includes a brief training of the predictor and model finetuning, which accounts for less than 30% of the original model’s training time.

Ada3D could improve the performance Adaptive inference removes redundant input features and saves computation and memory costs. However, adaptive inference does not necessarily have negative effects on performance. As shown in Table. 4 and Fig. 5, “Ada3D-A” improves the performance. We infer that the dropped redundant part is noisy and has negative effects on the training process.

Ada3D could be applied on the fully sparse 3D detectors. Fully sparse detectors (e.g., FSD [3], FSD++ [4], VoxelNeXt [1]) eliminate the dense BEV feature with novel architecture designs that directly process the sparse BEV feature to generate boxes. These models can still benefit from Ada3D’s spatial filtering, which further reduces redundant inputs. As shown in Tab. 2, when applying Ada3D for VoxelNeXT model, we further reduce 20% of redundant voxels with moderate performance degradation.

Ada3D could work on par with the model-level compression method and further improve efficiency. In comparison with existing model-level compression methods, Ada3D takes the perspective of compressing the spatial redundancy. Therefore, Ada3D could be combined with existing model-level to further improve efficiency. We adapt the SPVNAS [26] searched model to the 3D backbone of Centerpoint, and employ Ada3D to further compress it. As seen in Table. 8, Ada3D could further reduce the computaional and memory cost of SPVNAS optimized model.