Lidar Point Cloud Guided Monocular 3D Object Detection

The LiDAR device can provide accurate depth measurement of the scene, thus is employed by most state-of-the-art 3D object detection methods [31, 12, 32, 43, 44, 29]. These methods can be roughly divided into voxel-based methods and point-based methods. Voxel-based methods [48] first divide the point cloud into a voxel grid and then feed grouped points into fully connected layers, constructing unified feature representations. They then employ 2D CNNs to extract high-level voxel features to predict 3D boxes. By contrast, point-based methods [30, 24] directly extract features on the raw point cloud via fully connected networks, such as PointNet [25] and PointNet++ [26]. SOTA 3D detection methods predominantly employ LiDAR point clouds both in training and inference, while we only use LiDAR point clouds in the training stage.

As a commonly available and cheap sensor, the camera endows 3D object detection with the potential of being adopted everywhere. Thus monocular 3D object detection has become a very popular area of research and has developed quickly in recent years. Monocular works can be categorized into image-only-based methods [1, 13] and depth-map-based methods [40, 19, 18] according to input representations. M3D-RPN [1] employs different convolution kernels in row-spaces that can explore different features in specific depth ranges and improve 3D estimates with the 2D-3D box consistency. Furthermore, RTM3D [13] predicts perspective key points and initial guesses of objects’ dimensions/orientations/locations, where the key points are further utilized to refine the initial guesses by solving a constrained optimization problem. More recently, many image-only-based works utilize depth estimation embedding [46], differentiable NMS [11], and geometry properties [49, 15, 17], obtaining great success. There is also a related work [45] that introduces a novel autolabeling strategy of suggesting a differentiable template matching model with curriculum learning, using differentiable rendering of SDFs, while the pipeline is rather complicated.

Although monocular methods are developing quickly, a large performance gap still exists compared to LiDAR-based methods. Some prior works [40, 41] argue that the improper choice of data representation is one of the main reasons, and propose to use transformed image-based depth maps. They first project LiDAR point clouds onto the image plane, to form depth map labels to train a depth estimator. Pseudo-LiDAR [40] converts the image into a point cloud by using an estimated depth map and then conducts 3D detection on it. They show promising results compared to previous image-only-based methods. Inspired by this, many later methods [41, 19, 6, 18] also utilize off-the-shelf depth estimates to aim 3D detection and gain performance improvements. More recently, CaDDN [28] integrates the dense depth estimation into monocular 3D detection, by using a predicted categorical depth distribution to project contextual features to the 3D space. Compared to previous depth-map-based methods, we aim to explore the potential of using LiDAR point clouds to generate pseudo labels for monocular 3D detectors.

To take advantage of available 3D box annotations, as shown in Figure 2, we first train a LiDAR-based 3D detector from scratch with LiDAR point clouds and associated 3D box annotations. The pre-trained LiDAR-based 3D detector is then utilized to infer 3D boxes on other unlabeled LiDAR point clouds. Such results are treated as pseudo labels to train monocular 3D detectors. We compare the pseudo labels with manually annotated perfect labels in Section 5.5. Due to precise 3D location measurements, pseudo labels predicted from the LiDAR-based 3D detector are rather accurate and qualified to be used directly in the training of monocular 3D detectors. We summarize the outline in Algorithm 1.

Interestingly, with different training settings for the LiDAR-based 3D detector, we empirically find that monocular 3D detectors trained by resulting pseudo labels show close performances. It indicates that monocular methods can indeed be beneficial from the guidance of the LiDAR point clouds and only a small number of 3D box annotations are sufficient to push the monocular method to achieve high performance. Thus the manual annotation cost of high accuracy mode is much lower than the one of the previous manner. Detailed experiments can be found in Section 5.6. Please note, the observations on label requirements and 3D locations are the core motivation of LPCG. The premise that LPCG can work well is that LiDAR points provide rich and precise 3D measurements, which offer accurate 3D locations.

In this section, we describe the method of using LiDAR point clouds to eliminate the reliance on manual 3D box labels. First, an off-the-shelf 2D instance segmentation model [9] is adopted to perform segmentation on the RGB image, obtaining 2D box and mask estimates. These estimates are then used for building camera frustums in order to select associated LiDAR RoI points for every object, where those boxes without any LiDAR point inside are ignored. However, LiDAR points located in the same frustum consist of object points and mixed background or occluded points. To eliminate irrelevant points, we take advantage of DBSCAN [7] to divide the RoI point cloud into different groups according to the density. Points that are close in 3D spatial space will be aggregated into a cluster. We then regard the cluster containing most points as a target corresponding to the object. Finally, we seek the minimum 3D bounding box that covers all target points.

To simplify the problem of solving the 3D bounding box, we project points onto the bird’s-eye-view map, reducing parameters since the height ($h$) and $y$ coordinate (under camera coordinate system) of the object can be easily obtained. Therefore, we have:

where $B_{bev}$ refers to a bird’s-eye-view (BEV) box. We solve this problem by using the convex hull of object points followed by obtaining the box by using rotating calipers [37]. Furthermore, the height $h$ can be represented by the max spatial offset along the y-axis of points, and the center coordinate $y$ is calculated by averaging $y$ coordinates of points. We use a simple rule of restricting object dimensions to remove outliers. The overall training pipeline for monocular methods is summarized in Algorithm 2.

We use the image-only-based monocular 3D detector M3D-RPN [1], and adopt PV-RCNN [29] as the LiDAR 3D detector for the high accuracy mode. We filter out 3D boxes generated from LiDAR point clouds with the confidence of 0.7. Experiments on other methods are conducted by the official code that is publicly available, and all settings keep the same as the original paper. During the process of using LiDAR point clouds to train monocular 3D detectors, the learning iteration is scaled according to the number of training data. The high accuracy mode is employed by default. For the low cost mode, we use Mask-RCNN [9] pre-trained in the COCO dataset [14], and filter the final 3D bounding box by the width range of 1.2-1.8 meters and the length range of 3.2-4.2 meters. We filter out 2D boxes predicted from Mask-RCNN [9] with the confidence of 0.9. More details and ablations are provided in the supplementary material as the space limitation.

Additionally, to further validate the effectiveness of LPCG, we conduct experiments on the Waymo Open Dataset [36], which is a modern large dataset. It contains 798 training sequences and 202 validation sequences, and we adopt the same data processing strategy proposed in CaDDN [28]. The sampled training dataset includes approximately 50K training samples with manual annotations.

We evaluate LPCG on KITT test set using two base monocular detectors[1, 46] with the high accuracy mode. Table 2 shows quantitative results in test set. Due to the space limitation, qualitative results are included in the supplementary material. We can observe that our method increases the current SOTA BEV/3D AP from 21.19/15.02 to 24.81/17.80 under the moderate setting, which is rather significant. Even using a monocular detector [1] proposed in 2019, our method still allows it to achieve new state-of-the-art compared to prior works. Note that our method still performs better, while DD3D [22] employs both the large private DDAD15M dataset (containing approximately 15M frames) and the KITTI depth dataset (containing approximately 20K frames). Also, we boost the performance on pedestrian and cyclist categories of the original method, and provide the results in Table 5. Such results prove the effectiveness of LPCG.

For the low cost mode, we note that there are few works exploring this area, namely, few works have explored monocular 3D detection without any 3D box annotation. The most related work is SDFLabel [45], which also does not require 3D box annotation. Thus we compare LPCG with the low cost mode with SDFLabel [45] in Table 3. Please note, in this experiment we use the same number of training samples, namely, the 3, 769 samples in KITTI3D training set. Our method outperforms it by a large margin, and our method is more generally usable as our pipeline is much simpler than SDFLabel [45].

To further prove the effectiveness of our method, we conduct experiments on the Waymo open dataset. Concerning its large scale, when enough perfect labels are available, in this dataset we aim to investigate the performance gap between the generated pseudo labels and the manual 3D box annotations. More specifically, we use the baseline detector M3D-RPN [1], training it with pseudo labels and manual annotations, respectively. We report the results in Table 4. Pseudo labels on unlabeled data are generated by the high accuracy mode in LPCG. Interestingly, we can see that the detector using 10% labeled data and 90% unlabeled data achieves comparable accuracy to the one using 100% labeled data (e.g., 6.42 vs. 6.23 and 6.19 vs. 6.09). This result demonstrates the generalization ability of LPCG, indicating that LPCG can also reduce the annotation costs for the large scale dataset with slight accuracy degradation.

As expected, pseudo labels are not as accurate as manually annotated labels. It is interesting to quantitatively evaluate pseudo labels using manually annotated labels. We report the results in Table 6. TP, FP, FN are calculated by matching pseudo labels and annotated labels. Regarding matched objects, we average the relative error (MRE) on each group of 3D box parameters (location, dimension, and orientation). We can see that pseudo labels from the high accuracy mode can match most real objects (91.39%), and the mean relative errors are 1%-6%. Therefore pseudo labels from the high accuracy mode are good enough for monocular 3D detectors. Actually, experiments in Table 2 and 4 also verify the effectiveness. On the other hand, for the low cost mode, we can see that many real objects are missed (11834). We note that missed objects are often truncated, occluded, or faraway. The attached LiDAR points cannot indicate the full 3D outline of objects, thus they are hard to recover by geometry-based methods.

We conduct the ablation studies on KITTI val set. Because of the space limitation, we provide extra ablation studies in the supplementary material.