Make Your ViT-based Multi-view 3D Detectors Faster via Token Compression

Multi-view 3D object detection has many advantages when deploying to the real world, given its low costs and simple sensor setups (i.e., it only needs cameras). Existing methods can be mainly categorized into two types: dense BEV-based paradigm [41, 16, 13, 18, 43] and sparse query-based paradigm [24, 25, 21, 22].

For dense BEV-based paradigm, many works [8, 15, 31] use the explicit view transformation (e.g., LSS [32]) to transform image features into dense BEV. BEVDet [8] is the pioneer work of this paradigm. BEVDepth [15] leverages explicit depth supervision to facilitate accurate depth estimation, and SOLOFusion [31] combines long-term and short-term temporal stereo for better depth estimation, both improve the performance significantly. Instead of explicit view transformation, BEVFormer [17] pre-defines grid-shaped BEV queries and aggregates dense BEV features through attention, which is implicit. PolarFormer [9] explores the polar coordinate system to replace the grid-shaped coordinate system. Since dense BEV-based methods need to extract dense BEV features, the computation and memory costs are relatively high.

For the sparse query-based paradigm, DETR3D [37] initializes a set of 3D queries and aggregates features by projecting 3D queries into the 2D image plane. PETR [24] encodes the position information of 3D coordinates into image features, eliminating the need for 3D query projection. CAPE [39] and 3DPPE [35] further improve the quality of 3D position information. SparseBEV [23] introduces adaptability to the detector in both BEV and image space. For temporal 3D detection, Sparse4D [20] proposes sparse 4D sampling to aggregate features from multi-view/scale/timestamp. StreamPETR [36] introduces a memory queue to store history object queries for long-term temporal information propagation. Because these methods pass image features directly to the 3D decoder for detection, high-quality image features are beneficial. With advancing pre-trained ViTs, the sparse query-based methods [36, 20, 23] have achieved state-of-the-art performance and nearly dominated the leaderboard. However, their inference speeds are mainly hindered by the backbone due to the computational burden of ViT, which motivates us to trim the ViT backbone.

Vision transformer (ViT) [3] has gone viral in various computer vision tasks [5, 38] due to its strong feature extraction ability. The visualization of trained ViT shows sparse attention maps, which means the final prediction only depends on a subset of salient tokens. Based on this observation, many works [33, 1, 40, 11, 26] attempt to speed up ViT by removing redundant tokens, dubbed token compression. Specifically, DynamicViT [33] introduces a lightweight prediction module to estimate the importance score of each token and then prunes redundant tokens progressively and dynamically. A-ViT [42] further proposes a dynamic halting mechanism. EViT [19] leverages class token attention to identify the importance of tokens, then reserves the attentive image tokens and fuses the inattentive tokens. AdaViT [29] further prunes at attention head, and block level. Si et al. [26] jointly considers the token importance and diversity. Evo-ViT [40] presents a self-motivated slow-fast token evolution approach, which maintains the spatial structure and information flow. All the methods are initially designed for 2D vision tasks and conduct token compression without 3D-aware priors.

In this paper, borrowing from the methods in [33, 11, 40], we extend the token compression from the 2D domain to the 3D domain by leveraging history object queries and modeling the 3D motion information, leading to 3D motion-aware token compression tailored for 3D object detection.

Sparse query-based methods [24, 20, 36] improve the efficiency of 3D detectors by mainly modeling sparse object-centric queries as the foreground proxy instead of the whole 3D scene. However, we argue that for existing sparse query-based methods, there still exists much room for efficiency improvements, as they treat foreground and background all the same in the backbone. When using ViT [14, 4] to achieve extraordinary performance, the backbone becomes the bottleneck of inference speed.

To tackle the above problem, we propose to leverage token compression for extending the design philosophy of sparse query-based methods to the ViT backbone, named TokenCompression3D (ToC3D). As Fig. 2(a) shows, ToC3D mainly comprises two designs: motion query-guided token selection strategy (MQTS) and dynamic router. The token compression in each block happens as follows: 1) First, MQTS takes image tokens and history object queries as inputs and calculates the importance score of each image token through the attention between image tokens and history queries, splitting image tokens into salient and redundant ones. 2) Then, the dynamic router is used to extract features from different groups of tokens efficiently. Salient tokens are passed to the regular path, which consists of many attention blocks. The free path with the identity layer is used for redundant tokens to save computational costs. To keep the interaction between salient and redundant tokens in attention blocks, we merge redundant tokens into one bridge token and append it with salient tokens before the regular path. 3) Finally, after obtaining features of salient and redundant tokens, we rearrange salient and redundant tokens to meet the compatibility with typical 3D object detectors.

By stacking token compression-empowered blocks, the computing resources are dynamically and more intensively assigned to the foreground proposals, which removes unnecessary consumption and accelerates the inference remarkably. Ultimately, we effectively trim the ViT backbone and develop a more efficient sparse query-based multi-view 3D detector with the 3D sparse decoder.

Motion query-guided token selection strategy (MQTS) is meant to measure the importance score of each image token and split tokens into salient/redundant tokens. Generally, salient and redundant tokens usually contain information about foreground objects and background things. Based on this, MQTS essentially segments the foreground tokens in the image, and the history object queries with 3D motion information can serve as the foreground prior of high quality, which leads to the motion query-guided token selection design shown in Fig. 2(b).

Specifically, MQTS first takes history query contents $Q_{h}^{c}\in\mathbb{R}^{N_{q}\times C_{q}}$, history query reference points $Q_{h}^{p}\in\mathbb{R}^{N_{q}\times 4}$ (in homogeneous coordinates), and image tokens of current frame $T\in\mathbb{R}^{N\times C}$ as inputs, where $N_{q},N$ are the number of history queries and image tokens, $C_{q},C$ are the channels of history queries and image tokens, respectively. Then, MQTS uses the following process to split tokens accurately and efficiently:

Motion query preparation. Since spatial transformation exists between current and history frames, we introduce the temporal alignment process to align $Q_{h}^{p}$ with the current ego-coordinate system. Following [36], we use the motion-aware layer normalization. First, we view all objects as static and align $Q_{h}^{p}$ to the current frame using the ego transformation matrix:

where $\hat{Q}^{p}$ is the aligned history query reference points and $E_{h}$ is the ego transformation matrix from history frames to current frame. Then, we model the motion of movable objects by a conditional layer normalization, with affine transformation coefficients calculated as follows:

where $V_{m}$ is the encoded motion vector, $\gamma,\beta$ are affine transformation coefficients used in layer normalization, $[\cdot]$ is the concatenate operator, $v$ is the velocity of queries, $\triangle t$ is the time difference between history queries and current frame. $\mathrm{PE}(\cdot)$ is the positional encoding function, and we adopt the sine-cosine encoding used in NeRF [30]. Finally, we encode the motion information through conditional layer normalization:

where $\widetilde{Q}^{c}\in\mathbb{R}^{N_{q}\times C_{q}},\widetilde{Q}^{p}\in\mathbb{R}^{N_{q}\times C_{q}}$ are temporally aligned history query contents and reference points embedding, and $\mathrm{MLP}(\cdot)$ is a multi-layer perceptron used to convert reference points into embedding.

Importance score calculation. After obtaining the temporally aligned history query, we segment salient foreground tokens. To better extract the foreground prior, we leverage the attention mechanism to calculate the importance score of each image token. We first add temporally aligned history query contents and reference points embedding to get the new query embedding and then use a linear layer to align the dimensions of image tokens and query embedding:

where $\widetilde{Q}\in\mathbb{R}^{N_{q}\times C_{q}},\widetilde{T}\in\mathbb{R}^{N\times C_{q}}$ are dimension-aligned query embedding and image tokens respectively, $N,N_{q}$ are the number of image tokens and history queries, respectively. $C_{q}$ is the channel number of history queries. Next, we obtain the attention map through efficient matrix multiplication:

Essentially, the attention map $A\in\mathbb{R}^{N\times N_{q}}$ models the correlations between image tokens and history queries and thus can represent the foreground information density of each token since history queries contain foreground priors. By aggregating the foreground information density via a simple linear transformation with sigmoid activation, we determine the importance score $S\in\mathbb{R}^{N\times 1}$ of each image token:

Finally, we select the top-k image tokens as salient tokens according to importance score, making batch processing easier:

where $T_{s}\in\mathbb{R}^{N_{s}\times C}$ is the salient tokens and $T_{r}\in\mathbb{R}^{N_{r}\times C}$ is the redundant tokens. $N_{s}=\rho N,N_{r}=\left(1-\rho\right)N$ are the number of salient and redundant tokens respectively. $\rho$ is the pre-set constant keeping ratio of each block. Increasing $\rho$ will scale up the number of salient tokens; otherwise, scale up the number of redundant tokens, and thus, we can control the inference speed by adjusting $\rho$.

Although the importance score calculation is light-weight, it still brings some overhead. We found that updating the importance score of every layer does not bring noticeable improvements, so we only update the importance score before specific transformer layers for efficiency, and the layers in between will reuse the newest importance score.

History query sampling. Although history object queries carry high-quality foreground priors, we empirically found that not all history object queries are valuable. This is because the number of object queries is larger than that of objects of interest in a typical DETR-style detector, and many object queries do not correspond to foreground objects but background things. If we directly use all these history queries in MQTS, the importance score calculation will be biased by background information. Thanks to the confidence measurement of object queries, we can simply solve this problem by sampling history queries according to their confidence scores, which the decoder has already predicted in the history frames. In detail, we sort the history queries according to their confidence scores and select the top-$N_{q}$ queries as inputs of MQTS.

After splitting tokens into salient and redundant ones, we introduce a dynamic router to accelerate the inference speed while keeping information losses as low as possible, shown in Fig. 2(b). Considering that not all redundant tokens correspond to the background, these tokens may contain some potential information for detection, and the information can be passed to foreground tokens via attention interactions. We merge redundant tokens into a single bridge token and append it with salient tokens, allowing interactions between salient and redundant tokens through the bridge token. Then, we use more neural layers (i.e., regular path) to extract rich semantic and geometric information in salient tokens, while using shallow layers (i.e., free path, we use the identity layer in this paper) to keep the information in redundant tokens.

Formally, we use the importance scores to conduct weighted sum with redundant tokens for getting bridge token:

where $T_{b}\in\mathbb{R}^{1\times C}$ is the bridge token, $S_{r:i}$ is the importance score of the $i$-th redundant token, and $T_{r:i}$ is the $i$-th redundant token.

Afterwards, we append the bridge token after salient tokens and pass them to the regular path for extracting rich semantic and geometric features:

where $T_{s}^{\prime},T_{b}^{\prime}$ are updated salient and bridge tokens, $\mathrm{Blocks}$ are transformer encoder blocks, typically consisting of several window attention layers and a global attention layer for multi-view 3D object detection.

For redundant tokens, we pass them to the fast path (i.e., identity layer) and add them with the updated bridge token:

where $T_{r}^{\prime}$ is the updated redundant tokens and $\mathrm{Repeat}(x,y)$ repeats $x$ by $y$ times.

Finally, we combine the updated salient and redundant tokens to obtain the updated image tokens. Thanks to this simple yet effective dynamic router, we refine image tokens more efficiently and meet the compatibility with typical multi-view 3D detectors.

We evaluate our method on the large-scale nuScenes [2] dataset, consisting of 700 scenes for training, 150 for validation, and 150 for testing. The data of each scene is captured by six cameras at 10Hz, with full 360^∘ field of view (FOV). We use annotations of 10 classes: car, truck, construction vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone. We use the official nuScenes metrics for comparison: the nuScenes detection score (NDS), the mean average precision (mAP), the average translation error (ATE), the average scale error (ASE), the average orientation error (AOE), the average velocity error (AVE), average attribute error (AAE).

We select recently representative StreamPETR [36] as our basic pipeline, considering its high performance. For the backbone, we adopt ViT-B, ViT-L [3] and conduct token compression on them. We use the Gaussian Focal Loss [12] to supervise the MQTS, with the ground truth coming from projected bounding boxes. The model is trained on 8 NVIDIA V100 with a total batch size of 16 for 24 epochs. The inference speed is tested on a single RTX3090. AdamW [27] is used as the optimizer. The augmentation follow the StreamPETR [36], and without CBGS [44]. The detailed configurations can be viewed in Tab. 1.

We compare our method with the basic pipeline StreamPETR [36] and other popular multi-view 3D detectors on nuScenes [2] val set.

The main results are illustrated in Tab. 2. When using ViT-B as the backbone, our method (ToC3D-Fast) can perfectly maintain NDS and mAP compared with the StreamPETR method, with nearly 20% speedup. If marginal 0.5% NDS and mAP drop are allowed, our method (ToC3D-Faster) can further accelerate the backbone by 30% and the whole pipeline by 26%. Notably, ToC3D-Faster performs at the same level as StreamPETR while only costs like SOLOFusion with R50 backbone, indicating the effectiveness of our method.

When using ViT-L as the backbone, our method (ToC3D-Fast) achieves nearly lossless performance compared to the basic StreamPETR while accelerating the whole pipeline by 36ms. Furthermore, with a performance loss of no more than 0.9%, our method (ToC3D-Faster) brings 25% inference speed gains and runs at the same speed with Sparse4D [20] while keeping the vast performance superiority (i.e., over 6.4% NDS and 7.7% mAP).

Furthermore, scaling up the input image resolution to 800$\times$1600, our method can tremendously reduce the inference time by 258ms and 431ms with ToC3D-Fast and ToC3D-Faster settings, saving a considerable amount of computing resources for detection deployed on the cloud. The results prove that our method can achieve better trade-offs and greatly improve the efficiency of 3D detectors.

We conduct experiments for analysis of our method using ViT-L as the backbone. All models are trained for only 12 epochs and evaluated on the val set.

Compared to 2D token compression methods. To prove the effectiveness of our motion query-guided token selection strategy (MQTS), we compare our method with typical 2D token compression methods DynamicViT [33] and SparseDETR [34]. For a fair comparison, we replace the MQTS with these two methods, keep the dynamic route unchanged, and tune these methods to their best performance. We also compare with the Random token compression.

As listed in Tab. 3, it clearly shows that the random token compression brings a significant performance drop, especially when keeping ratios are low. This is because the random compression cannot capture the importance of image tokens and thus drops much helpful information. When using the score-based 2D token compression methods, the performance drop is much smaller than random compression since they are better aware of important foreground tokens and thus suffer less information loss. However, because these 2D methods only take image tokens as input, they conduct token compression without any 3D-aware features or priors. The lack of 3D awareness leads to sub-optimal token compression and thus hurts the performance severely (i.e., about 2% mAP and 2% NDS).

When it comes to our method, because MQTS has history object queries as inputs, it can model the 3D motion information of objects and aggregate the rich 3D foreground priors of high quality, leading to remarkably better results than 2D competitors (more than 2% mAP and 1.2% NDS improvement). Notably, With efficient MQTS, our method is able to almost maintain the performance of the basic pipeline at the same speed level as 2D token compression methods, indicating the superiority of MQTS.

Effectiveness of components. After proving our key insight that the object queries from history predictions can serve as the foreground prior of high quality, we now study what makes this insight work. We take our method with the Faster setting as the baseline of this experiment, and we remove one component each time to measure its effectiveness, shown in Tab. 4. It is worth noting that removing any components only slightly reduces inference time, showing the high efficiency of each component.

For setting (a), we replace the attention in Eq. 5 with a lightweight module, which brings 0.8% mAP and 0.4% NDS drop. This is because the attention map naturally models the correlations between image tokens and history queries and thus more explicitly represents the foreground information density of each token, leading to better importance measurement.

For setting (b), we discard processes from Eq. 1$\sim$3. The degraded performance (1.1% mAP and 0.6% NDS) shows the importance of motion information, which adaptively handles movable objects and suppresses the noise brought by misalignment between history objects and the current coordinate system.

For setting (c), we remove the history query sampling and use all history queries instead. This choice reduces mAP by 0.8% and NDS by 0.4%, showing the necessity of history query sampling, as it removes object queries corresponding to background things and prevents importance score calculation from being biased.

For setting (d), we do not use the bridge token in the dynamic router. Because of the absence of interaction between salient and redundant tokens, potential information contained in redundant tokens can not be passed to foreground tokens, leading to information loss and is ultimately reflected in mAP and NDS.

Impact of history query num $N_{q}$. Since we sample $N_{q}$ history queries as inputs of MQTS, we study the impact of different $N_{q}$ in this section, shown in Tab. 4.4. When $N_{q}=256$, we use all history queries. Otherwise, we sample top-$N_{q}$ history queries according to their confidence scores from history predictions. The results show that sampling history queries is always beneficial, as many history queries correspond to background things. History query sampling helps prevent the importance score from being biased by background and thus improves the performance. However, if $N_{q}$ is too small, it will drop many foreground queries, suffer information loss, and hurt the performance. We empirically find that $N_{q}=64$ can achieve better performance.

Impact of keeping ratios $\rho$. Keeping ratios $\rho$ controls the number of salient tokens and decides the accuracy-speed trade-off. In this section, we conduct experiments to figure out how keeping ratios affects the efficiency of our method. From Tab. 4.4, we can get the following phenomenon: (1) We can speed up the pipeline by nearly 15% without noticeable performance loss, and 25% with marginal NDS loss (0.9%). (2) In a certain range, the performance drop is nearly linear with the inference time drop (about 0.2% NDS drop for 20ms inference time). (3) Too small keeping ratios (i.e., 0.3, 0.2, 0.1) will bring a significant performance drop. This is because foreground tokens usually account for more than 10% image tokens, and too small keeping ratios inevitably discard foreground tokens, bringing information loss. Considering the requirements of real applications, we set the model with keeping ratios of 0.7, 0.5, 0.5 as ToC3D-Fast, and 0.5, 0.4, 0.3 as ToC3D-Faster versions, respectively, as these models have relatively better trade-off.

Generalization. We select StreamPETR [36] as our basic pipeline, but this does NOT mean that the application of our method is limited. In fact, ToC3D can serve as a plug-and-play method, and we show the generalization ability of our method by applying it to another strong pipeline, Sparse4Dv2 [21]. The results are shown in Tab. 7. It indicates that the behavior of our method is consistent across different baseline methods. With the same speed-up ratio as on StreamPETR, our approach keeps the performance loss within 0.8% and surprisingly maintains the exact NDS compared to the strong Sparse4Dv2 [21], proving the feasibility and effectiveness of our method on other pipelines.

To better study the behavior of MQTS, we visualize the attention map and salient tokens in Fig. 3. The attention maps clearly show that our method focuses on foreground objects of interest precisely, no matter the large objects (e.g., cars and trucks in sample (a) or the small objects (e.g., pedestrians in sample (b) and (c)). This is an intuitive proof of our claim in Sec. 3.2, i.e., the attention map models the correlations between image tokens and history queries, and thus can represent the foreground information density of each token since history queries contain foreground priors. With the 3D foreground object-aware attention map, the whole model can be more concentrated on foreground tokens when keeping ratios $\rho$ getting lower, improving the efficiency.

Since our method leverages history object queries as high-quality foreground priors, we assume the inputs are temporal image sequences with contiguous information. Although this assumption narrows the application of our method, we argue that this assumption is not strong as the perception system runs at typical frequencies and perceives environments contiguously for real-world autonomous driving. The second limitation of our method is that we need to re-train the token compression model if keeping ratios are changed. Using dynamic keeping ratios with some technical tricks for stability when training would help to partially solve this limitation, which is left for our future work.