Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors

Camera-based 3D detectors aim to predict the 3D bounding boxes of objects in camera images. Some approaches [34, 39, 25, 3] focus on the monocular setting. FCOS3D [34] utilizes a fully convolutional single-stage network to regress 3D object information directly without using any 2D-3D correspondence priors. [25] identifies the localization error as a key factor that constrains the detection performances and proposes strategies to alleviate it. MonoPair [3] leverages spatial constraints between paired objects to deal with occlusion. MonoDETR [39] enhances the vanilla Transformer with the contextual depth cues to guide the 3D detection process.

Recently, 3D object detection under the surrounding multiple-camera setting has attracted a lot of research efforts. One line of approaches [29, 17, 16, 18, 12, 15] transforms the image features to the BEV space with the help of depth estimation before applying a detection head. LSS [29] utilizes an estimated depth distribution of each pixel to lift the features of each image individually into a frustum and converts the frustums of all images into a BEV grid. BEVDepth [17] uses the intrinsic camera parameters as one of the inputs of the depth estimation module with supervision from point cloud to predict depth of images for BEV feature construction. BEVStereo [16] designs an effective depth estimation method based on temporal stereo to build BEV features. BEVFormer [18] proposes to aggregate features from both the spatial and temporal spaces to the current BEV space with learnable queries. PolarFormer [12] builds the BEV features in the polar coordinate system to consider the wedge shape of the physical world under the ego car’s perspective. DFA3D [15] proposes to use 3D deformable attention to aggregate the lifted features in 3D space so that the depth ambiguity problem can be mitigated.

Another line of approaches [35, 24, 38, 33, 37] directly samples image features with queries and uses a decoder network to detect objects. DETR3D [35] proposes to use a set of sparse object queries to implicitly transform features from 2D to 3D without estimating dense 3D scene geometry. PETR [23] generates 3D position-aware image features and then uses a set of queries to interact with the features and predict 3D bounding boxes. PETRv2 [24] constructs the 3D position embedding in a data-dependent way, and temporal information is exploited by transforming the coordinates in the previous frame to the current coordinate system. StreamPETR [33] makes use of temporal information by propagating selected queries from a memory queue to the current frame, and a motion-aware layer normalization is designed as well. SparseBEV [19] lets the queries interact with image features in a sparse manner with promising performance by designing scale-adaptive self-attention and adaptive spatio-temporal sampling modules. MV2D [37] proposes to learn queries based on 2D detection results, which later interact with RoI image features. CAPE [38] adopts a local camera-view coordinate system instead of a global one to form 3D position embeddings so that variances caused by changes of camera extrinsic parameters can be eliminated.

DETR [27] is the first Transformer-based [6] object detection approach, and it uses a set of object queries to interact with images features and constructs loss via bipartite matching. Many subsequent works [41, 21, 26, 36, 14, 22] have proposed to improve DETR. Deformable-DETR [41] proposes deformable attention that only attends to a small number of key sampling points of a reference. To accelerate the convergence, Conditional-DETR [26] disentangles the content and spatial queries and predicts conditional spatial queries from the decoder embedding. Anchor-DETR [36] uses anchor points to build queries so that each query can focus on a specific region. DAB-DETR [21] proposes to construct queries with dynamic learnable anchor boxes which are updated layer-by-layer. DN-DETR [14] reduces the instability of bipartite graph matching by reconstructing ground truth boxes from noisy queries. [22] proposes to only use positional metrics to stabilize the matching process of the DETR loss.

Prompting is initially proposed to modify the input text string so that a pre-trained large language model can be adapted to new tasks with few or no labeled data [20]. CLIP [30] uses prompts to transfer visual models trained with natural language supervision to downstream tasks under the zero-shot setting. Recently, some works [10, 5, 1, 8] propose to adapt a pre-trained visual model to different domains by adding a small amount of prompt parameters instead of fine-tuning the entire model. VPT [10] utilizes a small amount of trainable parameters to adapt large-scale Transformer models to downstream tasks instead of full fine-tuning. LPT [5] introduces a shared prompt and group-specific prompts into a frozen pre-trained model to adapt to long-tailed data. [1] transforms the input image with prompts so that a frozen pre-trained model can perform new tasks. E²VPT [8] proposes to use learnable key-value prompts and visual prompts with a prompt pruning procedure for effective and efficient fine-tuning.

As shown in Fig. 2, at timestamp $t$, the captured multi-camera images $I_{t}=\{I_{c}^{t}\}_{c=1}^{N_{\mathrm{cam}}}$ are input into an feature extraction backbone (e.g. ResNet [9] or VovNet [13]) to extract image features $F_{t}=\{F_{c}^{t}\}_{c=1}^{N_{\mathrm{cam}}}$, where $N_{\mathrm{cam}}$ is the number of cameras. We first feed $F_{t}$ to the 2D detection branch to obtain 2D bounding boxes. The 3D anchor generator generates a set of 3D anchors for each 2D bounding box with its corresponding camera’s intrinsic and extrinsic parameters and its class information. Then, the query-based 3D detector takes $F_{t}$ and the generated 3D anchors as input to predict 3D bounding boxes. We train the network in two stages.

Given the 2D detection bounding boxes $B=\{(\mathbf{b}_{i},g_{i})\}_{i=1}^{N}$ of image $I$, where $\mathbf{b}_{i}=(x_{i},y_{i},w_{i},h_{i})$ represents a bounding box with its center coordinate, width, and height, and $g_{i}$ is its class information, we associate each sampled point in $\mathbf{b}_{i}$ with 3D size, depth and yaw angle candidates, and generate a set of 3D query anchors for $\mathbf{b}_{i}$ following the steps described below. The process is shown on the right side of Fig. 2.

3D anchor center candidates. Since the projection of the center of an object to an image plane is always within the minimum unrotated rectangle that contains the entire object, given a 2D bounding box $\mathbf{b}_{i}=(x_{i},y_{i},w_{i},h_{i})$, we first sample a set of 2D object centers in the following way:

	$\displaystyle C_{\mathrm{2D}}$	$\displaystyle=\{(x_{s},y_{s})|x_{s}=x_{\mathrm{min}}+s_{x}\times i_{x},$		(1)
		$\displaystyle y_{s}=y_{\mathrm{min}}+s_{y}\times i_{y};$
		$\displaystyle i_{x},i_{y}\in\mathbb{Z}_{\geq 0},x_{s}\leq x_{\mathrm{max}},y_{s}\leq y_{\mathrm{max}}\},$

where $s_{x}$ and $s_{y}$ are step sizes, $x_{\mathrm{min}}=\lfloor x_{i}-w_{i}/2\rfloor$, $x_{\mathrm{max}}=\lfloor x_{i}+w_{i}/2\rfloor$, $y_{\mathrm{min}}=\lfloor y_{i}-h_{i}/2\rfloor$, and $y_{\mathrm{max}}=\lfloor y_{i}+h_{i}/2\rfloor$. Then, we define a set of depth candidates $D=\{d_{0},...,d_{N_{D}-1}\}$ of size $N_{D}$ and associate each point in $C_{\mathrm{2D}}$ with the depth candidates to generate $C^{\prime}_{\mathrm{2D}}=\{(x_{s},y_{s},d_{s})|(x_{s},y_{s})\in C_{\mathrm{2D}},d_{s}\in D\}$. Next, we transform each point in $C^{\prime}_{\mathrm{2D}}$ to the 3D coordinate system and obtain a 3D object center set $C_{\mathrm{3D}}=\{(x_{e},y_{e},z_{e})\}$. The transformation is carried out with the intrinsic parameters $\mathbf{K}$ and extrinsic parameters $\mathbf{R}$ of the corresponding camera of $(x_{s},y_{s},z_{s})$ using the formula below:

$$\left[\begin{array}[]{c}x_{e}\\ y_{e}\\ z_{e}\end{array}\right]=\mathbf{R}^{-1}\mathbf{K}^{-1}\left[\begin{array}[]{c}x_{s}\cdot d_{s}\\ y_{s}\cdot d_{s}\\ d_{s}\end{array}\right]$$

(2)

3D anchor size candidates. As the 3D object sizes of different classes can vary considerably while the change of object size of the same class is usually small, we select a bounding box $\mathbf{b}_{i}$’s 3D size candidates based on its class $g_{i}$. For each class $g$, we decide its width range $(w_{\mathrm{g}}^{\mathrm{min}},w_{\mathrm{g}}^{\mathrm{max}})$, height range $(h_{\mathrm{g}}^{\mathrm{min}},h_{\mathrm{g}}^{\mathrm{max}})$, and length range $(l_{\mathrm{g}}^{\mathrm{min}},l_{\mathrm{g}}^{\mathrm{max}})$ by traversing all objects of class $g$ in the training data and choosing the maximum and minimum values. Based on the above ranges, we construct the sets of width, height, and length as follows:

	$\displaystyle W_{g}$	$\displaystyle=\{w|w=w_{\mathrm{g}}^{\mathrm{min}}+s_{w}\times i;i\in\mathbb{Z}_{\geq 0},w\leq w_{\mathrm{g}}^{\mathrm{max}}\},$		(3)
	$\displaystyle H_{g}$	$\displaystyle=\{h|h=h_{\mathrm{g}}^{\mathrm{min}}+s_{h}\times i;i\in\mathbb{Z}_{\geq 0},h\leq h_{\mathrm{g}}^{\mathrm{max}}\},$
	$\displaystyle L_{g}$	$\displaystyle=\{l|l=l_{\mathrm{g}}^{\mathrm{min}}+s_{l}\times i;i\in\mathbb{Z}_{\geq 0},l\leq l_{\mathrm{g}}^{\mathrm{max}}\}$

where $s_{w}$, $s_{h}$, $s_{l}$ are the step sizes. Then, the set of 3D object size candidates $S_{g}$ for class $g$ are generated by combining the above three sets:

$$S_{g}=\{(w,h,l)|w\in W_{g},h\in H_{g},l\in L_{g}\}$$

(4)

Yaw angle candidates. We construct the set of yaw angle candidates by uniformly sampling in $[0,2\pi)$ with interval $\pi/N_{\theta}$, and the set $\Theta$ is defined below:

$\displaystyle\Theta=\{\theta|\theta=\frac{n_{\theta}}{N_{\theta}}\pi,n_{\theta}=0,1,2,...,2N_{\theta}-1\}$

(5)

Generating 3D query anchors. Given the 3D center candidate set $C_{\mathrm{3D}}$, size candidate set $S_{\mathrm{g}}$, and yaw angle candidate set $\Theta$ of a 2D bounding box $\mathbf{b}$ of class $g$, we generate an initial set of 3D anchors via the Cartesian product of the three candidate sets, i.e. $P_{\mathrm{Init}}=\{\mathbf{p}_{i}|\mathbf{p}_{i}=(x_{i},y_{i},z_{i},w_{i},h_{i},l_{i},\theta_{i})\}=C_{\mathrm{3D}}\times S{\mathrm{g}}\times\Theta$. To remove the anchors that are not compatible with $\mathbf{b}$, we project each $\mathbf{p}_{i}$ to the image plane of the 2D bounding box $\mathbf{b}$ and get $\mathbf{p}_{i}^{\mathrm{2D}}=(x_{i}^{\mathrm{2D}},y_{i}^{\mathrm{2D}},w_{i}^{\mathrm{2D}},h_{i}^{\mathrm{2D}})$. Then, the IoU between $\mathbf{p}_{i}^{\mathrm{2D}}$ and $\mathbf{b}$ is calculated, and only $\mathbf{p}_{i}$ with IoU larger than the threshold $\mu$ is retained. The final set of query anchors is $P=\{\mathbf{p}_{i}|\mathrm{IoU}(\mathbf{p}_{i}^{\mathrm{2D}},\mathbf{b}))>\mu$}. We use $P$ as the decoder input of three selected query-based 3D object detectors to show its effectiveness.

To enable the 2D detection branch (DAB-DETR [21] is used as the default 2D detector) to share the image feature extraction backbone with the 3D detection branch without compromising the performance of the 3D detector, we train the network in two stages with visual prompts [1] added to the 2D detection branch.

Stage 1: Training 3D detection branch. For training, instead of using the output of the 2D detection branch, we project the ground truth 3D bounding boxes to the images to get 2D bounding boxes. Then, the proposed 3D query anchor generation method is used to generate 3D query anchors which are used as the input to the decoder of the query-based 3D object detector (based on StreamPETR [33], SparseBEV [19], or BEVFormer [18]). To take advantage of the class information of 2D bounding boxes, when calculating the set-to-set loss [27] of the query-based 3D detector, the predicted 3D bounding box of a query is only matched with a ground truth box that has the same class as the query’s corresponding 2D box. More specifically, we divide the set of predicted bounding boxes $\hat{B}$ and the set of ground truth bounding boxes $B$ into $\{\hat{B}_{g}\}_{g=1}^{G}$ and $\{B_{g}\}_{g=1}^{G}$, respectively, where $G$ is the number of classes. Given a set pair $(\hat{B}_{g},B_{g})$, boxes in $\hat{B}_{g}$ are matched with boxes in $B_{g}$ by Hungarian algorithm following [27], and the cost matrix is calculated based on the predicted probability of the ground truth class and the $L_{1}$ loss between the predicted and ground truth 3D bounding boxes.

Stage 2: Training 2D detection branch with visual prompts. After training the image feature backbone with the 3D detection task in Stage 1, the backbone is frozen, and the 2D detection branch uses the same image features as the 3D detector. To adapt the image features to the 2D detection task, we add visual prompts designed in [1] to the feature maps. It is worth noting that  [1] adds prompts to the input images instead of feature maps as do we. We adopt the padding prompt design of [1]. More specifically, given a feature map of size $C\times H\times W$, where $C$, $H$, and $W$ are the number of channels, height, and width, respectively. Two prompt patches of size $C\times(\tau\times H)\times W$ are added to the top and bottom of the feature map, respectively. Another two prompt patches of size $C\times(H-2\tau\times H)\times(\tau\times W)$ are added to the left and right sides of the feature map, respectively. The total number of prompt parameters varies with the change of $\tau$. For the training of the 2D detection branch, only the prompt parameters and the head of the 2D detector are updated.

In the test phase, the 2D bounding boxes predicted by the 2D detection branch are used to generate the 3D query anchors for the query-based 3D detector.

We integrate our approach into three selected query-based 3D object detectors (StreamPETR [33], SparseBEV [19], BEVFormer [18]) by replacing the randomly initialized anchors (or learnable queries) with our proposed 3D query anchors inferred from 2D bounding boxes.

StreamPETR. StreamPETR [33] is built upon the query-based 3D object detector PETR [23]. To make use of temporal information efficiently, it maintains a memory queue of historical object queries. The queries of the current frame consist of selected queries from the memory queue and newly added queries. The new queries depend on a set of learnable 3D anchor points initialized with uniform distribution between 0 and 1. We integrate the proposed 3D query anchors into StreamPETR by simply substituting a set of anchors $\{(x_{i},y_{i},z_{i},w_{i},l_{i},h_{i},\sin\theta_{i},\cos\theta_{i})\}$ inferred from 2D bounding boxes for the original learnable 3D anchor points.

SparseBEV. Besides adopting sparse queries, SparseBEV [19] removes the dense global attention between queries and image features of [23], and it proposes an adaptive spatio-temporal sampling method to aggregate image features. SparseBEV defines a set of learnable queries, and each query represents an object’s translation, dimension, rotation, and velocity. To integrate our 3D query anchors, we replace the learnble queries with $\{(x_{i},y_{i},z_{i},w_{i},l_{i},h_{i},\sin\theta_{i},\cos\theta_{i})\}$ generated from 2D detection results.

BEVFormer. In [18], BEV features are constructed by a spatial cross-attention between predefined BEV queries and multi-camera image features, which are used as input to a modified Deformable DETR [41] for 3D object detection. To apply the 3D query anchors, we design a 3D detection head based on DAB-Deformable-DETR [21], and the learnable dynamic anchors in the form of $(x,y,h,w)$ are replaced with 3D query anchors in the form of $(x,y,z,w,l,h,\sin\theta,\cos\theta)$, and the decoder predicts 3D bounding boxes and velocity rather than 2D bounding boxes.

We conduct experiments on the nuScene dataset [2]. It consists of 1000 multi-modal videos each of which is about 20s, and keyframes are annotated every 0.5s. The sensors include camera, LIDAR, and RADAR, and we use the images captured by the six surrounding cameras for our experiments. The videos are split into three subsets of 750, 150, and 150 for training, validation, and test, respectively. There are 1.4M annotated 3D bounding boxes of 10 classes in total.

We use the official evaluation metrics of nuScenes. Along with mean average precision (mAP), the following true positive errors are reported: average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE). In addition, a more comprehensive nuScenes detection score (NDS) is derived from the above metrics.

We implement our method with PyTorch [28]. After integrating our QAF2D into a base detector, we use the base detector’s data augmentation strategy and training setting (e.g. learning rate, batch size, number of epochs) for training. All models are trained with 8 NVIDIA GeForce RTX 3090 GPUs. For the parameters regarding 3D anchors generation in Section 3.2, $s_{x}$ and $s_{y}$ that control the sampling intervals of 2D object centers are set to 10. The depth candidates in $D$ are sampled between 3 meters and 103 meters with an interval of 1.5 meters. The width range, height range and length range of each class are given in the appendix. We set the sampling intervals $s_{w}$, $s_{h}$ and $s_{l}$ to 5, and $N_{\theta}$ that controls the sampling interval of yaw angle is set to 12. The IOU threshold $\mu$ for anchor validation is 0.99, i.e. only 3D anchors, the projections of which have high overlap with their corresponding 2D boxes, are kept. The parameter $\tau$ related to the number of prompt parameters in Section 3.3 is set to 0.2 based on the ablation study.

Validation on nuScenes val split. To verify the effectiveness of the 3D query anchors generated by the proposed QAF2D, we compare StreamPETR [33], SparseBEV [19], and BEVFormer [18] with their QAF2D-enhanced version on the nuScenes validation split. For fair comparison of StreamPETR, besides its original results using three dimensional random queries, we also report the results of StreamPETR with eight dimensional random queries (8DQuery), which have the same dimension as queries of QAF2D. For fair comparison of BEVFormer-small, along with its original result based on modified Deformable DETR [41], we report the result of BEVFormer-small with our modified DAB-Deformable-DETR [21] (BEVFormer-small-DAB3D) as well. BEVFormer-small-DAB3D uses eight dimensional randomly initialized queries. For SparseBEV, as its queries are nine dimensional containing velocity information, we do not change queries to eight dimensional ones. The results in Table 1 demonstrate that our QAF2D can bring about consistent improvement for all three base detectors. With regard to StreamPETR (V2-99 backbone [13]), after improving the performance of StreamPETR by $0.5\%$ NDS and $0.4\%$ mAP with eight dimensional queries, our proposed QAF2D can obtain additional improvement of $1.0\%$ NDS and $1.4\%$ mAP in comparison with StreamPETR-8DQuery, and the entire improvement is $1.5\%$ NDS and $1.8\%$ mAP. We also evaluate StreamPETR and its enhanced version with ResNet50 backbone [9], and our QAF2D can gain an improvement of $0.6\%$ NDS and $1.5\%$ mAP. In addition, we incorporate QAF2D into StreamPETR^∗‡ (benefits from perspective-view pre-training and uses different number of queries [33]), and QAF2D can improve its performance by $1.2\%$ NDS and $1.5\%$ mAP. For SparseBEV, our proposed QAF2D can improve its performance by $0.3\%$ NDS and $1.2\%$ mAP. As to BEVFormer, it can be first improved with our modified DAB-Deformable-DETR by $1.3\%$ NDS and $2.0\%$ mAP, and the proposed OAF2D can gain another improvement of $1.0\%$ NDS and $0.7\%$ mAP compared with BEVFormer-small-DAB3D, and the total improvement is $2.3\%$ NDS and $2.7\%$ mAP.

Comparison with state-of-the-art on nuScenes test split. We compare StreamPETR-QAF2D with the state-of-the-art approaches on the nuScenes test split. The results in Table 2 show that while StreamPETR-8DQuery has the same performance as StreamPETR in terms of NDS and mAP, our OAF2D still can improve StreamPETR by $0.6\%$ NDS and $1.6\%$ mAP, which further validates the effectiveness of our approach. Meanwhile, QAF2D-enhanced StreamPETR achieves the best performance on the nuScenes test split.

The performance improvement can be attributed to two aspects: (1) 3D query anchors inferred from 2D detection results can provide better initial 3D box positions and sizes than random anchors, and this can ease the optimization of the network and predict the results more accurately, (2) the state-of-the-art 2D detectors are more reliable than the 3D detectors, and some missed detections of the base 3D detectors can be recovered with the help of QAF2D.

Component time consumption and speed. We report the time consumption of each component and the speed of StreamPETR [33] and StreamPETR-QAF2D in Table 3. Note that the two approaches use the same backbone, the slight difference in time between two separate runs should be inevitable for the hardware. Though StreamPETR-QAF2D is somewhat slower than StreamPETR (15.4 FPS vs 18.9 FPS), we think that the overall efficiency of StreamPETR-QAF2D is acceptable. Meanwhile, the increase of the complexity mainly comes from the 2D detection head of DAB-DETR [21] and the 3D query anchor generation component is very fast (1ms). As QAF2D is not sensitive to the choice of the 2D detector (please refer to the ablation study in Section 4.4), the efficiency of QAF2D can be improved by using lighter 2D detectors.

We carry out ablation study on the nuScenes validation split and use BEVFormer [18] as the base detector.

Generalization to different 2D detectors. To study the generalization ability of QAF2D to different 2D detectors, besides the default 2D detector DAB-DETR [21] , we also combine QAF2D with the popular Faster-RCNN [31] and applies it to BEVFormer [18]. Based on the results in Table 4, we can see that the performances of DAB-DETR and Faster-RCNN are very close to each other, which can demonstrate that the proposed QAF2D is not sensitive to the selection of the 2D detector. As the result of DAB-DETR is slightly better, we use DAB-DETR as the default 2D detector to provide 2D detection boxes. The insensitivity should be because that QAF2D does not need very precise 2D boxes, as long as the 2D detector does not miss objects, QAF2D can use rough 2D boxes to generate meaningful 3D query anchors that are better than random ones.

Impact of additional random 3D query anchors. We also combine an additional set of randomly initialized 3D query anchors with anchors generated by QAF2D to explore its impact. The number of random 3D query anchors is 900, which is the number of queries used by BEVFormer-small. The results are shown in Table 5. The additional random anchors can bring about improvement. But as the improvement of $0.1\%$ NDS and $0.4\%$ mAP is small, we prefer not to use additional random anchors in our default setting.

Effect of visual prompts for feature sharing. To investigate the effects of adding a small number of prompt parameters to the feature maps of the backbone for feature sharing, we train a separate 2D detector that has its own image feature extraction backbone (denoted by “No sharing”). We also train another 2D detector that directly uses the image features of the backbone trained by the 3D detector, and only the 2D detection head is fine-tuned (denoted by “Sharing w/o prompt”). From the results in Table 6, we can see that “No sharing” and “Sharing w/o prompt” serve as the upper bound and lower bound of performance, respectively.

The parameter $\tau$ in “Stage 2” of Section  3.3 controls the number of prompt parameters. We vary $\tau$ from 0.1 to 0.5 to study how the choice of it affects the performance. The results in Table 6 show that all choices of $\tau$ can close the gap between “No sharing” and “Sharing w/o prompt”, and $\tau=0.2$ is the best value. With $\tau=0.2$, the NDS difference between “No sharing” and “Sharing w/o prompt” is reduced from $0.6\%$ to $0.1\%$, and the mAP difference is reduced from $1.0\%$ to $0.5\%$.

Visual comparison between StreamPETR-8DQuery and StreamPETR-QAF2D are shown in Fig. 3. We draw 3D detection boxes in multi-camera images and their projections in BEV space. Three typical cases where the proposed QAF2D helps are given. Case 1 shows that QAF2D can remove the false positive (blue box in the purple ellipse in the top-left image of Fig. 3 (a)) and make the true positive more accurate (see the alignment between the prediction and ground truth in BEV). Case 2 demonstrates that when the objects are faraway, the prediction of QAF2D is more accurate as well. Case 3 shows that when the object is small and difficult to distinguish from the background, QAF2D can help to alleviate miss detection. Please refer to the appendix for more visualization results.