RangeRCNN: Towards Fast and Accurate 3D Object Detection
with Range Image Representation

3D Object Detection with Grid-based Methods. Most state-of-the-art methods in 3D object detection project the point clouds to the regular grids. [6, 7, 8, 9] directly projects the original point clouds to the 2D bird’s eye view to utilize the efficient 2D convolution for feature extraction. They also combine RGB images and other views for deep feature fusion. [10] is a pioneering work in 3D voxel-based object detection. Based on [10], [1] increases the efficiency of the 3D convolution using sparse optimization. Several following methods [11, 12, 13] utilize the sparse operation [1, 2] to develop more accurate detectors. For the real-time 3D object detector, [14] proposes the pillar-based voxel to significantly improve the efficiency. However, the grid-based methods suffer from information loss in the stage of initial feature extraction. The sparsity issue of the point clouds also limits the effective receptive field of 3D convolution. These problems will become more serious when processing the large outdoor scene.

3D Object Detection with Point-based Methods. Compared to the grid-based methods, the point-based methods are limited by the high computation cost in early research works. [15, 16] project 2D bounding boxes to the 3D space to obtain 3D frustums and conduct the 3D object detection in each frustum. [17, 18] directly process the whole point cloud using [19] and generate proposals in a bottom-up manner. [20] introduces a vote-based 3D detector that is more suitable to process indoor scenes. [21] uses a graph neural network for point cloud detection. [22] proposes a fusion sampling strategy to speed up the point-based method.

3D Object Detection with Range Image. Compared to the grid-based and point-based methods, fewer researchers utilize the range image in 3D object detection. [6] takes the range image as one of its inputs. [3, 4] directly process the range image for 3D object detection. However, the range image based methods have not matched the performance of the grid-based or point-based methods on the popular KITTI dataset until now. [4] mentions that better results are achieved on the larger self-collected dataset. Recently, [23] has proposed the range conditioned dilated block for learning scale invariant features from the range image and achieves the top performance on the large-scale Waymo Open dataset. We think that the scale of the dataset is a secondary reason for low accuracy. The main reason is that the range image is a good choice for extracting initial features, but not a good choice for generating anchors. In this paper, we design a better framework utilizing the range image representation for 3D object detection.

Recently, feature learning on point clouds has been studied for different tasks, such as classification, detection and segmentation. Among these tasks, the 3D semantic segmentation task is chosen by many methods [24, 19, 25, 2, 26] as the touchstone for evaluating the ability to extract features from point clouds. Early research was mainly focused on indoor scenes due to the lack of outdoor datasets. SemanticKITTI [27] is a recent semantic segmentation benchmark for autonomous driving scenes. In the benchmark, [28] proves the effectiveness of learning features from the range image for 3D semantic segmentation and simultaneously runs at a high speed. However, the value of the range image in the field of 3D object detection is far from being fully explored. We believe that the range image can also provide rich and useful information for the 3D object detection task.

Our proposed network is illustrated in Fig. 2. The 3D point cloud is represented as the native range image, which is fed into an encoder-decoder 2D backbone to extract features efficiently and effectively. We upsample the deep feature map to the original resolution of the range image using a decoder in order to retain more spatial information. We then transfer features from the range image to each point. We do not extract features based on the point view using the point-based convolution [24, 19]. Actually, the point view has two functions. First, it serves as the bridge from the range image to the bird’s eye image. Second, it provides the pointwise feature to the 3D RoI pooling module for refining the proposals generated by the region proposal network (RPN). After obtaining the pointwise feature, we can easily obtain the BEV feature by projecting the 3D point to the x-y plane. Since we have effectively extracted high-level features from the range image, BEV mainly plays the role of proposal generation. This projection is therefore different from projecting the 3D point to BEV at the beginning, which is extremely dependent on the feature extraction from BEV. We use a simple RPN to generate proposals from BEV and refine the proposals using the 3D RoI pooling module. We name the one-stage network without RCNN pooling RangeDet, and name the two-stage framework RangeRCNN.

Most existing datasets such as KITTI provide the point cloud as the LIDAR data format, so we need to convert the points to the range image. As described in [28], the formula is as follows:

$$\begin{pmatrix}~{}u~{}\\ ~{}v~{}\end{pmatrix}=\begin{pmatrix}~{}\frac{1}{2}[1-arctan(y,x)\pi^{-1}]\times w\\ [1-(arcsin(z,r)+f_{down})f^{-1}]\times h\end{pmatrix}$$

(1)

where $(x,y,z)$ represents the point coordinate in the 3D space. $(u,v)$ is the pixel coordinate in the range image. $r=\sqrt{x^{2}+y^{2}+z^{2}}$ is the range of each point. $w$ and $h$ are the predefined width and height of the range image. $f=f_{up}+f_{down}$ is the vertical field-of-view of the LIDAR sensor. For each pixel position, we encode its range, coordinate, and intensity as the input channel. As a result, the size of the input range image is $5\times h\times w$. In [28], the categories of semantic segmentation are labeled for all points, so the range image contains the 360-degree information. The LIDAR used by the KITTI dataset is the Velodyne 64E LIDAR with 64 vertical channels. Each channel generates approximately 2000 points. In their task, therefore, $h$ and $w$ are respectively set as 64 and 2048. In the KITTI 3D detection task, the LIDAR and the camera are jointly calibrated. Only the objects in the front view of the camera are labeled; this view represents approximately 90 degrees of the whole scene. Additionally, some vertical channels are filtered by the FOV of the camera. We thus set $h=48$ and $w=512$, which is sufficient to contain the front view scene. The size of the range images used for the KITTI dataset is $5\times 48\times 512$. The Waymo Open dataset provides range images with the size of $64\times 2650$ as the LIDAR data format. Therefore, we can directly use it without any extra conversion.

The backbone for the range image is illustrated in Fig. 3. The 2D backbone for extracting features from the range image is an encoder-decoder structure. The size of the feature map is given in the figure. The input is downsampled in the encoder by six dilated downsample blocks (shown in Fig. 4(b)), then gradually upsampled in the decoder by four corresponding dilated upsample blocks (shown in Fig. 4(c)). The range image provides dense and compact representation for utilizing the 2D CNN but simultaneously presents the scale variation issue. The scales of objects with different distances exhibit significant differences. To better adapt different scales and obtain a more flexible receptive field, we design the dilated residual block (DRB), which inserts the dilated convolution into the normal residual block. In this block (Fig. 4(a)), three $3\times 3$ convolutions with different dilated rates {1, 2, 3} are applied to extract features with different receptive fields. The outputs of the three dilated convolutions are concatenated followed by a $1\times 1$ convolution to fuse the features with different receptive fields. A residual connection is used to add the fused feature and the input feature. In the dilated downsample block (Fig. 4(b)), we first use a $1\times 1$ convolution to extract features across channels. Then the feature is input to DRB to extract features with a flexible receptive field. The dropout operation and the pooling operation are used for better generalization performance and downsampling the feature map, respectively. In the first two blocks of the encoder, we do not use the pooling operation. In the dilated upsample block (Fig. 4(c)), we first use the bilinear interpolation to upsample the feature map. A skip connection is introduced to concatenate the feature from the encoder. The concantenated feature is then input to DRB. The size of the final output features (visualized in Fig. 3 by t-SNE) extracted from the range image is the same size as the input with 64 dimensions.

The range image representation is suitable for feature extraction by utilizing the 2D convolution. However, it is difficult to assign anchors in the range image plane due to the large scale variation. The severe occlusion also makes it difficult to remove redundant bounding boxes in the Non-Maximum Suppression (NMS) module. Fig. 5 shows a typical example. The size of the bounding boxes varies greatly in the range image. Some bounding boxes have a large overlap. In contrast, these bounding boxes have a similar shape in the BEV plane because most cars are of similar size. It is also impossible for different cars to overlap with each other in the BEV place even though they are very close. We therefore think that it is more suitable to generate anchors in the BEV plane. Thus, we transfer the feature extracted from the range image to the bird’s eye image.

For each point, we record its corresponding pixel coordinates in the range image plane, so we can obtain the pointwise feature by indexing the output feature of the range image backbone. Then, we project the pointwise feature to the BEV plane. For points corresponding with the same pixel in the BEV image, we use the average pooling operation to generate the representative feature for the pixel. Here the point view only serves as the bridge to transfer features from the range image to the BEV image. We do not use the point-based convolution to extract features from points.

Discussion. Different from projecting point clouds to the BEV plane at the beginning of the network [6, 7], we perform the projection after extracting high-level features. If projecting at the beginning, the BEV serves as the main feature extractor. The information loss caused by the discretization leads to inaccurate features. In our framework, the BEV primarily plays the role of anchor generation. As we have extracted features from the lossless range image, the quantization error caused by the discretization has a minor influence. Experiments show the superiority of our methods compared with those methods projecting at the beginning.

Based on the bird’s eye image, we generate 3D proposals using the region proposal network (RPN). However, neither the range image nor the bird’s eye image explicitly learns features along the height direction of the 3D bounding box, which causes our predictions to be relatively accurate in the BEV plane, but not in the 3D space. As a result, we want to explicitly utilize the 3D space information. We conduct a 3D RoI pooling based on the 3D proposals generated by RPN. The proposal is divided into a fixed number of grids. Different grids contain different parts of the object. As these grids have a clear spatial relationship, the height information is encoded among their relative positions. We directly vectorize these grids from three dimensions to one dimension sorted by their 3D positions (illustrated in Fig. 6). We apply several fully connected layers to the vectorized features and predict the refined bounding boxes and the corresponding confidences.

The whole network is trained in end-to-end fashion. The loss contains two main parts: the region proposal network loss $L_{rpn}$ and the region convolutional neural network loss $L_{rcnn}$, which is similar to [1, 11, 13]. The RPN loss $L_{rpn}$ includes the focal loss $L_{cls}$ for anchor classification, the smooth-L1 loss $L_{reg}$ for anchor regression and the direction classification loss $L_{dir}$ for orientation estimation [1]:

$$L_{rpn}=L_{cls}+\alpha L_{reg}+\beta L_{dir}$$

(2)

where $\alpha$ is set to 2 and $\beta$ is set to 0.2 empirically. We use the default parameters for focal loss [29]. The smooth-L1 loss regresses the residual value relative to the anchor [1].

The RCNN loss includes the confidence prediction loss $L_{score}$ guided by the IoU [11], the smooth-L1 loss $L_{reg}$ for refining proposals and the corner loss $L_{corner}$ [15]:

$$L_{rcnn}=L_{score}+L_{reg}+L_{corner}$$

(3)

The total training loss is the sum of the above losses:

$$L_{total}=L_{rpn}+L_{rcnn}$$

(4)

Network Details. For the KITTI dataset, the size of the input range image is $5\times 48\times 512$ as mentioned above. For the Waymo Open dataset, the size of the range image is $6\times 64\times 2650$, where 64 indicates 64 lasers and 2650 means that each laser has 2650 points. The LIDAR of the Waymo Open dataset has a 360 degree field of view instead of 90 degrees as the KITTI dataset. The range image backbone for the two datasets is described in Sec. III-B.

For the KITTI dataset, the range of BEV is within $[0,69.12]m$ for the x-axis and $[-39.68,39.68]m$ for the y-axis. The resolution of BEV is $0.16^{2}~{}m^{2}$, so the initial spatial size of BEV is $496\times 432$. We use three convolution blocks to downsample the BEV to $248\times 216$, $124\times 108$ and $62\times 54$, and upsample the three sizes to $248\times 216$. The three features with the same size are then concatenated along the channel axis. For the Waymo Open dataset, the range of BEV is within $[-75.52,75.52]m$ for the x-axis and $[-75.52,75.52]m$ for the y-axis. The resolution of BEV is $0.32^{2}~{}m^{2}$, so the initial spatial size of BEV is $472\times 472$, which is slightly different from the size of $496\times 432$ for the KITTI dataset. The aforementioned concatenated features are input to the RPN network for generating 3D proposals.

In the two-stage RCNN, the 3D proposal generated by the RPN is divided into a fixed number of grids along with the local coordinate system of the 3D proposal. The spatial shape is set as $12\times 12\times 12$. We reshape the $12\times 12\times 12$ grids to a vectorized format with dimension of $12^{3}\times C$, where $C$ is the feature dimension of each grid. The feature of each grid is obtained from the point features. $C$ is equal to 64. If multiple points fall within the same grid, the max pooling is used. Three fully connected layers are then applied to the vectorized feature. Finally, the confidence branch and the refinement branch are used to output the final result.

Training and Inference Details. We implement the proposed RangeRCNN with PyTorch1.3. The network can be trained in an end-to-end fashion with the ADAM optimizer. For the KITTI dataset, we train the entire network with the batch size of 32 and learning rate of 0.01 for 80 epochs on 8 NVIDIA Tesla V100 GPUs, which takes approximately 1.5 hours. For the large-scale Waymo Open dataset, we train the entire network with the batch size of 32 and learning rate of 0.01 for 30 epochs on 8 NVIDIA Tesla V100 GPUs, which takes approximately 70 hours. We adopt the cosine annealing learning rate strategy for the learning rate decay.

For the KITTI dataset, we use the data augmentation strategy similar to [1, 11], including random flipping along the x-axis, random global scaling with a scaling factor sampled from $[0.95,1.05]$, random global rotation around the vertical axis with a sampled angle from $[-\frac{\pi}{4},\frac{\pi}{4}]$, and ground-truth sampling augmentation to randomly ”paste” some new ground-truth objects to current training scenes. For the Waymo Open dataset, we discuss the data augmentation strategy in Sec. IV-D.

For RCNN, we choose 128 proposals with a 1:1 ratio for positive and negative samples during training. During inference, we retain the top 100 proposals according to the confidence with the NMS threshold 0.7. We apply the 3D NMS to the refined bounding boxes with a threshold of 0.1 to generate the final result.

Metrics. The detection result is evaluated using the mean average precision (mAP) with the IoU threshold 0.7. For the official test benchmark, the mAP with 40 recall positions is used. We report the performance of 3D detection and BEV detection in Table I.

Performance. We submit our results to the online benchmark for comparison with other state-of-the-art methods. For evaluating the test set, we use all provided labeled samples to train our model. Table I reports the results evaluated on the KITTI online test server. Fig. 7 shows the qualitative results. Our RangeRCNN outperforms almost all previous approaches except PV-RCNN [13] on the commonly used moderate level for 3D car detection. We surprisingly observe that our method achieves the highest accuracy for the hard level. We think that the performance is beneficial to two aspects. First, some hard examples are very sparse in the 3D space, but they have more obvious features in the range image thanks to the compact representation. These objects can thus be detected using the range image representation. Second, RCNN further refines the 3D position of the bounding box, which boosts the 3D performance. The ablation study also proves the value of RCNN.

Runtime Analysis. To provide a fair runtime comparison, we run several representative two-stage methods with the same device (NVIDIA V100 GPU). The comparison is shown in Table. II. The proposed RangeRCNN runs at 22 fps, which is much faster than the point-based PointRCNN and the voxel-based PartA2 and PV-RCNN. The high computation performance is beneficial from the compact representation of the range image and the efficiency of 2D convolution. Research on the range image can promote real-time 3D object detection.

Metrics. The mean average precision (mAP) and the mean average precision weighted by heading (mAPH) are adopted as the metrics of this dataset. The IoU threshold for vehicles is set as 0.7. The detection performances with different object distances are reported. The dataset divides the objects into two difficulty levels based on the number of points within the 3D boxes. Level 1 includes at least five points with the 3D box and Level 2 includes at least one point. As few previous methods report their Level 2 performance, we compare RangeRCNN with state-of-the-art methods with respect to Level 1.

Performance. Table III shows that RangeRCNN outperforms previous methods by a large margin. The Waymo dataset directly provides the range image without the requirement of additional conversion, so the quality of the range image is better than that of the KITTI dataset. RangeRCNN has the ability to extract rich information for distant objects. For objects within $[0,30]m$, the performance of RangeRCNN is similar to that of PV-RCNN. For objects within $[30,50]m$ and $[50,75]m$, RangeRCNN performs much better than PV-RCNN. The farther away the object is, the greater the advantage of our method over other methods. RangeRCNN outperforms RCD which also utilizes the range image representation. RCD only operates on the range image which does not utilize the ability of BEV to generate anchors. The performance of RangeRCNN on this large-scale dataset shows the large potential of the range image based method.

For the KITTI dataset, it does not directly provide the range image, so we need to convert the points to the range image. In this dataset, we first perform data augmentation on the points similar with [1, 11]. Then we convert the augmented points to the range image. Although the augmented range image no longer strictly adheres to the scanning characteristic of the rotating LIDAR, the augmentation strategy brings an improvement in accuracy.

For the Waymo Open dataset, the data augmentation pipeline is a little bit different from that of the KITTI dataset. Since the Waymo dataset provides the range image representation, we directly flip and rotate the range image, and scale the point coordinates stored in the channel of the range image. One interesting operation is to apply the cut-and-paste strategy on the range image (illustrated in Fig. 8). We first traverse the training data and read all annotated objects. For each point in the object, we store its 3D coordinate, intensity, elongation and corresponding pixel coordinate on the range image plane. When training the model, we randomly paste the object according to the pixel coordinates to augment the original range image. In addition to augment the range image, we also need to simultaneously consider the corresponding changes and transformation for the 3D point coordinates. The improvement brought by the cut-and-paste strategy depends on several factors. If we use a small part of the whole Waymo Open dataset for training, the improvement will be more obvious, which means that this strategy is very effective with limited training data. For the category with much fewer labels such as cyclists, the improvement will be much greater, which means that this strategy is helpful for some scarce categories. As the Waymo Open dataset is large, if training with all data, this strategy hardly improves the accuracy of vehicles with large amount of annotations. We believe that the augmentation strategy is very useful for future reserach based on range image. Through the range image based data augmentation, the preprocessing pipeline of the range image based method is almost equivalent to the voxel based method and the point based method. Detailed experiments and analysis will be provided in the future version.

In this section, we conduct the ablation study on the validation set of the KITTI dataset.

Effects of 3D RoI Pooling. As the entire framework is two-stage, we compare the results of the one-stage model RangeDet and the two-stage model RangeRCNN to analyze the value of RCNN. From Table IV, we can find that RangeDet and RangeRCNN exhibit similar performances for BEV detection. For 3D detection, RangeRCNN outperforms RangeDet by a large margin. The better 3D performance comes from the 3D information encoded by 3D RoI pooling.

Effects of Grid Size in 3D RoI Pooling. We further evaluate the influence of the grid size in 3D RoI pooling. We compare a set of grid sizes in {6, 8, 10, 12, 14}. Table V shows the results. It can be found that the grid size exerts no great influence on the metric. We choose 12 as the grid size, which is a relatively better size.