You Only Look Bottom-Up for Monocular 3D Object Detection

There are plenty of works that conduct 3D object detection with LiDAR inputs. Many works [13, 14, 15, 16, 17] focus on the pure LiDAR-based 3D object detection with fully supervision. Some other methods [18, 19, 20] explore the semi and weakly supervised learning or even zero-shot learning to reduce the costs of labeling. To take advantage of the complementary between images and point clouds, some methods [21, 22, 23] propose different strategies for better fusion. Despite their remarkable performance, the dependence on expensive LiDAR sensors hinders the wide use of these methods. Our method only needs a single cheap camera, making it easier for practical usage.

Monocular 3D Object Detection methods can be roughly grouped into two categories: depth-guided methods and pure image-based methods.

Depth-guided methods need extra data sources such as point clouds and depth images in the training phase. These methods mainly use the extra sub-network to estimate pixel-level depth under the supervision of the sparse projected point cloud or even dense depth images, which introduces extra labeling efforts. In particular, Pseudo-LiDAR [24] converts pixels to pseudo point clouds and then feeds them into a LiDAR-based detector [16, 17, 25]. Besides, there are also some methods [26, 27, 28, 29] that fuse depth feature and image feature together to improve the performance while  [30, 31, 32] learn 3D prior information via knowledge distillation. Some methods [33, 34] conduct view transform with estimated depth distribution. DD3D [35] claims that pre-training paradigms could replace the pseudo-lidar paradigm. These methods benefit from precise depth information provided by point clouds, which could be considered as using another type of ground truth as extra supervision.

Pure image-based methods only need RGB images and calibrated information in the whole inference pipeline. These methods mainly learn depth information from objects’ apparent size and geometry constraints provided by eight keypoints or pin-hole model [5, 36, 37, 38, 7, 39, 40]. For example, M3D-RPN [5] proposes 2D-3D anchor and depth-wise convolution to detect 3D objects. RTM3D [37] predicts nine keypoints of 3D proposals and optimizes the proposals via a re-projection cost function. MonoDLE [41] proposes three strategies to alleviate problems caused by localization errors. GUPNet [42] proposes a geometry uncertainty projection method for better localization. MVM3Det [43] proposes feature orthogonal transformation to estimate the position. TIM [44] conducts monotonic attention to translate images into bird’s-eye-view maps. DID-M3D [45] decouples the object’s depth into visual depth and attribute depth. These methods show great promise with their simple structures and competitive performances.

Several methods have exploited the position clues provided by the ground plane. Mono3D [1] leverages the ground plane to filter redundant proposals. GAC [8] proposes ground-aware convolution that extracts depth priors from the ground plane hypothesis. DeepLine [46] extracts and encodes line segmentation features into image features via Hough transform. MonoGround [9] generates dense depth supervision from the ground plane of ground truth 3D bounding boxes. These methods could alleviate the size ambiguity problem to some extent. However, they do not fully leverage the positional and contextual information of pixels, which plays an important role in monocular 3D detection. Compared with the above methods, our YOLOBU only requires a weaker assumption that the depth of the ground plane is monotonically increasing from the bottom of the image to target instances, which could be satisfied more easily. Meanwhile, our method doesn’t rely on explicit geometry structures in the scene, such as lines or planes. We explicitly build the relationship of pixels in the bottom-up direction, which could serve as a position clue for reasoning objects’ location in 3D space.

The way to transform 2D image features into features representing 3D space is crucial for image-based perception. It can be roughly categorized into inverse perspective mapping (IPM)-based and depth estimation-based. IPM first assumes a flat plane, and then it maps all pixels from a given viewpoint onto this flat plane through homography projection, and many methods [47], [48], [49], [50] use IPM for obstacles detection or lane detection task. Depth estimation-based methods [34], [33], [24], [51], [45] mainly estimate pixel-level depth or depth distributions to transform the image features into 3D features. Since our method acts as a plug-and-play module and operates on the image feature level, it can be adopted for both types of approaches.

Structured layout (i.e., road) pixels with different semantics and locations devote differently to object localization. For example, in each column, the road pixels closer to the top of the image plane represent a larger depth interval, which may contribute more to the localization of far objects and less useful for near objects. This phenomenon indicates that it is infeasible to treat pixels in the same column equally. Furthermore, the depth interval of pixels in different columns varies, inspiring us to pay attention to the differences among different columns. To model such a relationship, we propose the coloumn-based cross attention mechanism termed as CCA, which leverages cross attention to assign different weights to pixels within each column, and uses a single distinct query for each column to represent the differences between columns.

Concretely, given the input feature map $F_{b}\in\mathbb{R}^{H\times W\times C}$ extracted from backbone, we define a set of learnable embedding queries $Q\in\mathbb{R}^{W\times C}$. Each query corresponds to a column of the feature map separately.

We illustrate the mechanism of CCA, taking the $j$-th column as an example. As shown in Fig. 2(b), the sine-cosine position encoding $P^{\cdot j}\in\mathbb{R}^{H\times C}$ is added to the image feature $F_{b}^{\cdot j}\in\mathbb{R}^{H\times C}$ to get position encoded feature $F_{k}^{\cdot j}\in\mathbb{R}^{H\times C}$, as below:

Next, we get the attention weight $W^{\cdot j}\in\mathbb{R}^{H\times 1}$ by conducting matrix multiplication attached with a $Softmax$ activation function:

where $Q^{j}\in\mathbb{R}^{1\times C}$ is the $j$-th embedding query, and $W^{\cdot j}\in\mathbb{R}^{H\times 1}$ represents the attention weights for the $j^{th}$ column.

We stack the weights $W^{\cdot j}$ and repeat through the channel dimension to get the complete attention weights $W\in\mathbb{R}^{H\times W\times C}$, and finally we get the output feature $F_{c}\in\mathbb{R}^{H\times W\times C}$ by multiplying $W$ with $F_{b}$:

where $\bigodot$ denotes Hadamard product.

Note that our neat CCA module builds the relationships within columns and across columns efficiently. Compared with the original transformer [52, 53] that costs $O((H*W)^{2}*C)$, the time complexity of our proposed CCA is only $O(H*W*C)$.

After re-weighting the pixels within each column, we need to build the connections for pixels in the one-way direction (i.e., bottom-up direction), since the bottom-up positional information is indispensable as extra information compensation and can serve as references for locating objects, as what we argued in Sec. I. For this purpose, we propose the RRCS, which models the bottom-up relationship between pixels in a row-by-row way.

Specifically, as illustrated in Fig. 2(c), we conduct vertical cumulative sum (VCS) on the output of CCA module $F_{c}\in\mathbb{R}^{H\times W\times C}$ in the bottom-up direction:

where the $F_{r}^{i\cdot}$ and $F_{c}^{k\cdot}$ represents the $i$-th row of feature map $F_{r}$ and $k$-th row of input $F_{c}$, respectively. The RRCS promises that each pixel could perceive all of the pixels below it, which could be treated as a dense unidirectional connection at the pixel level. Thus, encoded features of structured hierarchical layout pixels (i.e., road pixels) could be propagated to the target objects, which acts as a position-related clue for 3D pose reasoning.

Note that the cumulative sum would change the value magnitude of features, which might be hard for the network to converge. Thus, normalization is needed here to keep values stable, which is conducted as the Eq. 5 shows:

The feature map $\hat{F_{r}}$ is then encoded by a $1\times 1$ convolution layer $\varphi$ and added with the feature $F_{b}$ to formulate the position aware feature $F_{p}\in\mathbb{R}^{H\times W\times C}$:

Our method predicts 3D proposals based on the feature map $F_{p}$, which is attached to the detection head. Our detection heads are composed of three branches: classification branch, 2D regression branch and 3D regression branch. Optimizing 2D parameters $b_{2D}$ and 3D parameters $b_{3D}$ simultaneously helps the network learn geometrical relationships between the image and world space.

We follow [41] as for the training loss. Suppose that $y=(c,b_{2D},b_{3D})$ and $\hat{y}=(\hat{c},\hat{b_{2D}},\hat{b_{3D}})$ denote the set of ground truths and predictions respectively, where $c$ denotes the classification score. The training objective is summarized as Eq. 7.

Here $L$ is the overall loss, while $L_{cls}$, $L_{2D}$ and $L_{3D}$ represents classification loss, 2D detection loss and 3D detection loss respectively.

We first introduce the dataset and evaluation metrics.

KITTI [60], containing 7,481 training samples and 7,518 testing samples (test set), is one of the most widely used datasets for autonomous driving-related tasks. We split the training samples into two parts: 3,712 samples for model training (train set) and 3,769 for validation (val set), following [41]. We test our method on the official server for fair comparisons with other methods.

For evaluation metrics, we report $AP_{3D|R40}$ and $AP_{BEV|R40}$ on car, pedestrian and cyclist categories with detection IoU thresholds of 0.7, 0.5 and 0.5, respectively. For better analysis, all experiments are done under all three difficulties (Easy, Moderate, and Hard) according to 2D bounding box height, occlusion, and truncation degrees.

We then explain the details of model hyperparameters and training settings in this section.

Model settings. Similar to [41], the input image is first resized to (384, 1280) after random crop and random flip as data augmentation. DLA-34 [61] without deformable convolutions is adopted as our backbone network. The feature maps are downsampled by 4$\times$, and the channel of feature maps is 64. The number of query embeddings in Cross Attention is equivalent to the feature map width, i.e., 96. We encode the image features and the fixed position encoding with 1$\times$1 convolution, ReLU, and 1$\times$1 convolution as the input keys in attention. Each head branch comprises 3$\times$3 convolution, ReLU, and 1$\times$1 convolution.

Training and Inference. We implement our method using PyTorch [62]. Our model is trained on two NVIDIA 3090 Ti GPUs in an end-to-end manner for 140 epochs, and the batch size is set to 16. Following [41], we employ the common Adam optimizer with an initial learning rate of 1.25e-4 and decay it by ten times at 90 and 120 epochs. We also applied the warm-up strategy (5 epochs) to stabilize the training process. In the inference stage, NMS with an IoU threshold of 0.2 is used to filter proposals.

In this part, we compare our method with some well-known and effective methods on KITTI [60] dataset.

The results for Car on test set are shown in Tab. I. We can see that our method outperforms all listed pure image-based methods on the $AP_{3D|R_{40}}$ metric at Moderate and Hard difficulties. Specifically, our method outperforms DCD [38] by 0.31%, 0.52% and HomoLoss [59] by 1.27%, 0.66% at Moderate, Hard difficulties respectively. Besides, our method outperforms MonoGround [9] by 1.06%, 1.85%, 1.11% at Easy, Moderate and Hard settings separately. Although our method is inferior to DeepLine [46] under the Easy setting, it performs better under the Moderate and Hard settings, exceeding 1.88% and 3.43%, respectively. Even when compared with listed methods using extra data [64], our method is still competitive. Specifically, our method outperforms the knowledge distillation methods SGM3D [32] by 1.56% and 0.76% and MonoDistill [31] by 0.18% and 0.13% at Moderate and Hard settings. Furthermore, we only get a 0.13% performance drop at the Moderate setting on $AP_{3D|R_{40}}$ metric when compared with DD3D [35] that uses a large amount of data in pretraining. We also get the competitive result to DID-M3D [45] that leverages Lidar supervision at Moderate and Hard difficulties. This is mainly because our proposed YOLOBU can leverage position clues as supplementary information to locate objects.

The results for Pedestrian and Cyclist on test set are shown in Tab. II. Although the performance of our method on Pedestrian and Cyclist is not as impressive as the performance on Car, YOLOBU still ranks third and second on Pedestrian and Cyclist, lagging behind the best methods by 0.31% and 0.67% at Moderate setting, respectively.

Our method is significantly better in performance when compared with other position-aware methods [63, 8]. We conduct comparative experiments between CoordConv [63], GAC [8] and our YOLOBU on val set. For CoordConv [63] setting, we simply concatenate the pixels’ $u-v$ coordinates and features and then use a 1$\times$1 convolution to extract information. For GAC [8] setting, we use the ground-aware convolution in regression heads following the implementation of its official codebase. The results are listed in Tab. III, which shows that our YOLOBU outperforms the other two methods by a large margin on $AP_{3d}$. Specifically, YOLOBU beats CoordConv  [63] by 2.73%, 1.05%, 1.48% on $AP_{3d}$ at Easy, Moderate, Hard difficulty respectively. YOLOBU also exceeds GAC [8] by 1.03%, 0.61%, 1.38% on $AP_{3D}$ at Easy, Moderate, Hard difficulty respectively. This is mainly because our YOLOBU models the bottom-up relationships between pixels and thus encodes more representative position information for monocular 3D detection.

We further conduct additional experiments to analyze the effectiveness of each component. We use MonoDLE [41] as our baseline and keep all experimental settings the same.

Effectiveness of the CCA. To figure out how much contribution the Column-based Cross Attention (CCA) brings, we attach the CCA to the backbone of our baseline. As Tab. IV(a) shows, the CCA brings $AP_{3D}$ (0.77%, 0.38% and 0.55% improvements on Easy, Moderate and Hard difficulties, respectively) while slightly reducing in the $AP_{BEV}$ metric. The results indicate that the contribution of the CCA only is limited since the positional relation between pixels is not modeled.

Effectiveness of the RRCS. To validate the effectiveness of Row-based Reverse Cumulative Sum (RRCS) only, we insert the proposed RRCS after the backbone of our baseline. The results are listed in Tab. IV(b). We can observe that only adding RRCS is detrimental to the performance. Note that the semantic and positional information of each pixel is variant and keeping all the edge weights the same would bring much noise to the model.

We emphasize that the CCA and RRCS are an integral whole to establish the association of pixels. The former generates the edge values, and the latter connects nodes with the edge values. As shown in Tab. IV(c), when we use both the CCA and RRCS, we can obtain a significant improvement over the baseline. Our method brings 2.37%, 1.32%, 1.95% improvement of $AP_{3D}$ on Easy, Moderate, and Hard difficulties respectively, demonstrating the significance of the positional relation modeling.

Analysis of attention types. Tab. V quantifies the influence of different attention types in the proposed CCA. For the model without column-based attention, we use a single query for all pixels and conduct global cross attention instead. We can observe that column-based attention is necessary since the semantics of structured layout pixels (e.g., road pixels) vary across columns, and one single query is not representative enough.

Analysis of the Cumulative Sum direction. In order to examine the impact of different cumulative sum directions (i.e., Bottom-Up and Up-Bottom), we conduct comparative experiments and the results are listed in Tab. VI. We can observe that building the graph either in the Bottom-Up or Up-Bottom direction improves the performance over baseline, which proves the effectiveness of pixels’ association in columns for the monocular 3D detection task. Meanwhile, we can see that building relations from the Bottom-Up direction is much better than Up-Bottom. This is consistent with our prior that the pixels which help locate objects mainly occur below them.

To further evaluate the effectiveness of our method, we conduct a comparison experiment on the large-scale nuScenes dataset. We train models on the official train split and evaluate on the val split. For evaluation metrics, we report the official metrics include: Nuscenes Detection Score (NDS), mean Average Precision (mAP), Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE) and Average Attribute Error (AAE).

We use the FCOS3D [51] as the baseline, and we insert the proposed modules (CCA and RRCS) after the neck to refine the features of each scale. Because of the significant increase in the scale of training data, we use 8 NVIDIA 3090 GPUs with a total batch of 32 to train models. The results are listed in Tab. VII. Our method outperforms the FCOS3D by 1.6% NDS and mAP, showing its effectiveness, and the conclusions are consistent with that of the KITTI dataset.

We visualize the model predictions to analyze our method more intuitively. The results are shown in Fig.3.

Qualitative results. As we can see, our proposed YOLOBU predicts remarkably precise 3D Bounding Boxes in scenario (b). Specifically, compared with our baseline MonoDLE [41] in scenario (a), our method can correctly detect the car in the top right corner of the image while the baseline predicts an inaccurate bounding box. This phenomenon demonstrates that image-based 3D detection methods benefit from the bottom-up positional information.

Generalization for other applications. Since our method effectively leverages the position clues from images and serves as a plug-and-play module, one can easily adapt our method for other applications by simply plugging it into the original solution, as long as the task (e.g., stereo and multi-view 3D object detection for research, and autonomous driving, underwater robots and intelligent logistics for the industry) meets the assumptions introduced by our method.

Limitations. Our method has some limitations: 1) Our method fails to detect the truncated and large objects, as shown in Fig. 3 (c). The main reason is that these samples cannot benefit from the bottom-up positional information. 2) The inference time of the proposed method will be slightly slower than the baseline [41] (22.56 FPS vs. 23.64 FPS). We will explore real-time solutions in the future. 3) Our method does not model the calibration parameters, and the performance may be affected when the calibration parameter change. However, we can easily modify the detection head to model them explicitly for better results.