Every Dataset Counts: Scaling up Monocular 3D Object Detection with Joint Datasets Training

Bird-Eye-View Methods: These techniques focus on performing monocular 3D detection directly in 3D spaces, simplifying the output representation design. However, the main difficulties lie in converting perspective-view images to 3D coordinates features. In [5, 6, 7, 8], researchers first predict depth information from monocular images, and then perform 3D object detection based on the predicted depth information. Another method directly conducts differentiable feature transformation, creating 3D features from image features, enabling end-to-end training of 3D detection in 3D spaces [9]. These methods often address the scale-ambiguity issue through depth prediction sub-networks or attention modules. However, inference speed in the BEV space heavily depends on 3D labels from LiDAR or direct supervision from LiDAR data, making it difficult to leverage existing 2D datasets or cost-effective 2D labeling tools. As a result, researchers lacking access to 3D labeled data may face challenges when deploying or fine-tuning networks in new environments.

Perspective-View Methods: These methods perform monocular 3D detection directly in the original perspective view, which is more intuitive. Many single-stage monocular 3D object detection methods are built upon existing 2D object detection frameworks. The main challenge here lies in designing robust and accurate encoding/decoding methods to bridge 3D predictions and dense perspective view features. Various techniques have been proposed, such as SS3D [10], which adds additional 3D regression parameters; ShiftRCNN [11], which introduces an optimization scheme. Other works like M3DRPN [12], D4LCN [13], and YoloMono3D [14] use statistical priors in anchors to improve the accuracy of 3D regression. Additionally, SMOKE [15], RTM3D [16], Monopair [17], and KM3D [18] leverage heatmap-based keypoint predictions, combined with anchor-free object detection frameworks like CenterNet [19].

Despite impressive progress in monocular 3D detection, most research still focuses on training within a single homogeneous dataset, leading to overfitting and poor generalization of models in specific camera settings. In this study, we propose a more robust output representation based on MonoFlex [4], enabling the network to be trained on different datasets. We also introduce a training strategy for performing 3D object detection on 2D datasets, enhancing the model’s generalization and reducing the annotation cost for 3D object detection.

The weakly-supervised approach is also one of the methods aimed at reducing dependency on annotated data. Autolabels [20] present an automatic annotation pipeline to recover 9D cuboids and 3D shapes from pre-trained off-the-shelf 2D detectors and sparse LIDAR data. WS3D [21] introduced a weakly supervised approach for 3D LiDAR object detection in two stages. Initially, cylindrical object proposals are generated by manual annotations in the bird’s eye-view. Subsequently, the network refines these proposals using a small set of precisely labeled object instances to produce the final 3D object bounding boxes. WeakM3D [22] first detected objects in images and combined them with 3D point cloud data to obtain object-LiDAR-points. Furthermore, it proposed a method to estimate object orientation $\theta$ by determining the orientation of each point pair in the object-LiDAR-points. WeakMono3D [23] introduce a new labeling method called 2D direction label, replacing the 3D rotation label in point clouds data and a direction consistency loss based on the new labels. Compared to the aforementioned weakly supervised methods, our approach does not require point clouds assistance, which reduces the complexity of sensors. It also does not rely on multi-view images, avoiding errors in the spatial position of objects caused by imprecise poses between multi-view images.

Monocular 3D object detection involves estimating the 3D location center $(x,y,z)$, dimensions $(w,h,l)$ and planar orientation $\theta$ of objects of interest with a single image. Since most SOTA detectors perform prediction in the camera’s front-view, we generally predict the projection of the object center on the image plane $(c_{x},c_{y})$ instead of the 3D position $(x,y)$. The orientation $\theta$ is further replaced with the observation angle

which better conforms with the visual appearance of the object [4, 12].

MonoFlex [4] is an anchor-free method. It extracts feature maps from the input images with a DLA [24] backbone, similar to CenterNet[19] and KM3D [18]. As an anchor-free algorithm, MonoFlex predicts the center positions of target objects with a heat map. Monoflex primarily consists of the following components: 2D detection, dimension estimation, orientation estimation, keypoint estimation and depth estimation ensemble. The depth prediction $z$ is simultaneously estimated from both geometry and direct prediction, and these predictions are adaptively ensembled to obtain the final result.

Since we aim to perform joint training on diverse datasets, where data collection involves different cameras with distinct camera settings, our method needs to overcome the challenges posed by varying camera parameters in order to facilitate effective knowledge transfer across these datasets. Given that MonoFlex exhibits insensitivity to camera parameters, we build upon MonoFlex as the foundation of our approach.

Overall, our approach is similar to MonoFlex, with the main difference being that our method has modifications in the depth prediction component, which takes camera parameters into account. Specifically, in the original MonoFlex paper, the direct regression part of depth prediction assumes an absolute depth of

where $z_{o}$ is the unlimited network output following [17, 19] and

While in our method, we have improved it by taking camera’s parameters into account as follows:

where $f_{x}$ is epresents the focal length of the camera used in the training dataset, while $f_{x0}$ is a hyperparameter, we set its value to 500 in our experiments.

During dataset training, some parts of the dataset are incompletely annotated. For instance, in the case of KITTI, certain categories like “Tram” are not labeled. However, it is not appropriate to treat the data from KITTI as negative samples for the “Tram” category. Therefore, each data point, in addition to its own annotations, is associated with the categories labeled in the respective dataset. This association provides supervision for the network’s classification output and is applied only to the categories with annotations. Specifically, for different 3D datasets with varying labeled categories, each data frame stores the categories currently labeled in the dataset. When calculating the loss, we do not penalize (suppress) the detection predictions for unannotated categories. In summary, this approach effectively handles the issue of incomplete annotations during dataset training. It ensures that only annotated categories influence the model’s training, and unannotated categories do not lead to erroneous model behavior. The training procedure is shown in Fig. 3.

In the Monoflex method, the center of the heatmap is determined by projecting the 3D center. For data with only 2D annotations, we have no direct means of generating supervision signals for the object’s 3D center. Therefore, we need to find a way to generate 3D detection supervision information from 2D labels. We propose a novel method to train 3D detection algorithms solely relying on 2D detection labels. Specifically, we start by feeding data with only 2D annotations into a pre-trained 3D detection model and set a very low score threshold to enable the model to produce multiple detection results (including both 2D and 3D detection results). This step may include some erroneous or inaccurate detections. Next, we use the 2D training labels from the new dataset to match them with the 2D detection results obtained in the previous step, filtering out erroneous or inaccurate detections to obtain pseudo 3D training labels. Finally, we reconstruct the ground truth heatmap and 2D detection map (as shown in Fig. 4), and ultimately, we calculate the loss between the model predictions and the pseudo 3D labels only on the heatmap and 2D detection map. The method of training a monocular 3D model using 2D annotated data is illustrated in Algorithm 1, we refer to it as Pseudo 3D Training with 2D Labels.

We evaluate the proposed networks on the KITTI 3D object detection benchmark [25] and Cityscapes dataset [26]. The KITTI dataset consists of 7,481 training frames and 7,518 test frames. Chen et al. [27] split the training set into 3,712 training frames and 3,769 validation frames. The Cityscapes dataset contains 5000 images split into 2975 images for training, 500 images for validation, and 1525 images for testing.

To demonstrate the effectiveness of our method, we conducted extensive experiments on the KITTI and Cityscapes datasets. When testing on the KITTI dataset, we designed our experiments as follows:

• •

  We initially pre-trained our model on four datasets: BDD100K, nuScenes, ONCE, and Cityscapes.

• •

  With the pre-trained model, we evaluated its zero-shot detection performance on the KITTI dataset.

• •

  Subsequently, we fine-tuned the model using our method, which involves training a 3D detection model using 2D training labels from KITTI.

• •

  Finally, we obtained detection results on the KITTI dataset based on our method.

When testing on the Cityscapes dataset, we followed the same experimental setup on the KITTI dataset.

The quantitative results of the “Car”, “Pedestrain” and “Cyclist” category on KITTI dataset are reported in Table I, while quantitative results of “Car”, “Truck” and “Bicycle” category on Cityscapes dataset are shown in Table II, Table III, and Table IV respectively.

From Table I, we can observe that our method has achieved significant improvements in both 3D and 2D detection tasks compared to zero-shot learning. Specifically, for the 3D detection task, our method has shown an improvement in $AP_{3D}$/$AP_{BEV}$ for the “Car” category in the Easy, Moderate, and Hard difficulty levels by $18.55/18.67$, $10.10/10.32$ and $8.20/8.37$, respectively. In the “Pedestrian” category, we achieved a progress in $AP_{3D}$/$AP_{BEV}$ in the Easy, Moderate, and Hard difficulty levels by $7.29/7.90$, $7.13/7.28$ and $5.72/5.94$, respectively. And for the “Cyclist” category, we improved the $AP_{3D}$/$AP_{BEV}$ by $0.93/0.57$, $0.47/0.24$ and $0.46/0.48$ in Easy, Moderate and Hard level, respectively. For the “Cyclist” class, although there is not a significant increase in the number of 3D detection points, the improvement in AP3D/APBEV across three different difficulty levels ranges from 29.27% to 145.32%. Therefore, the improvement from our method is significant. In summary, the experimental results demonstrate the effectiveness of our method on the KITTI dataset.

Table II, Table III and Table IV present the detection results for the “Car”, “Truck” and “Bicycle” categories on the Cityscapes dataset. Quantitative results indicate that our method, when compared to the zero-shot approach, has shown significant improvements in various metrics, except for the PRSim metric (as we only focused on the yaw angle, considering pitch and roll angles to be 0). Specifically, in the “Car” category, we observed an increase of $25.05$ in AP, a $0.69$ improvement in BEVCD, a $2.15$ improvement in YamSim, a $6.18$ improvement in SizeSim, and a remarkable $24.02$ enhancement in DS. In the “Truck” category, AP, BEVCD, YawSim, SizeSim, and DS have improved by $13.71$, $0.85$, $0.06$, $12.50$, and $13.12$, respectively. And for the “Bicycle” class,AP, BEVCD, YawSim, SizeSim, and DS have improved by $2.77$, $3.50$, $5.21$, $11.74$, and $2.35$, respectively. From the above experimental results, it can be seen that our method has also achieved significant performance improvement on the Cityscapes dataset. Fig. 5 provides a visual comparison between our method and the zero-shot approach. It can be observed that our method exhibits significantly improved detection capability compared to the zero-shot approach.

Table V illustrates the comparison between our method and several weakly-supervised methods on the “Car” class in the KITTI dataset. It is apparent that overall, our method falls short of the latest state-of-the-art algorithm, WeakMono3D [23]. However, it outperforms other weakly-supervised algorithms. It is worth mentioning that our approach relies solely on easily accessible 2D annotations, without requiring any 3D annotation information or additional sensor data such as point clouds or multi-view images for assistance.