PillarGrid: Deep Learning-based Cooperative Perception for 3D Object Detection from Onboard-Roadside LiDAR

With the tremendous progress achieved by a convolutional neural network (CNN) in image-based computer vision tasks, CNN is also widely applied to object detection based on point clouds [13]. Since LiDAR-based object detection is an intrinsically three-dimensional task, early work [14, 15] tends to deploy 3D convolutional layers. Although 3D convolutional layers work well for extracting hidden features from 3D point clouds data, they require much higher computational power compared with 2D convolution, resulting in longer processing time. For instance, as a fast object detection model, Vote3Deep proposed by Engelcke et al. [14] utilized sparse convolutional layers to deal with 3D point clouds and required about $500ms$ to process a single frame of point clouds data, i.e., $2Hz$ inference frequency. Comparatively, the fast 2D object detection models can run at a frequency over $50Hz$ [16, 17].

To speed up the 3D object detection, one popular paradigm is to encode the 3D point clouds into a 2D birds-eye-view (BEV) pseudo-image which can be processed by a standard image detection architecture, such as ComplexYolo [18] and PointPillars [5]. By discretizing point clouds into 3D voxels and then encoding them to a pseudo-image format, these object detectors can run the inference pipeline at a similar frequency to camera-based 2D detectors [18, 5, 16, 17].

Although general object detection has gone through a rapid development era, research focusing on roadside sensors is still an emerging topic and has the potential to break the ground for cooperative automated driving, especially in a mixed traffic environment [19]. Due to the lack of labeled datasets, most of the roadside-LiDAR-based object detectors are still following a traditional paradigm to deal with point clouds, which consists of 1) Background Filtering to remove the laser points reflected from the road surface or buildings by applying filtering methods, such as statistics-based background filtering [20]; 2) Clustering to generate clusters for the laser points by implementing clustering methods, such as Density-Based Spatial Clustering Applications with Noise (DBSCAN)  [21]; and 3) Classification to generate different labels for different traffic objects, such as vehicles and pedestrians, based on neural networks [22]. This paradigm is straightforward and popular for implementation in the real world [23], but has a large performance gap compared with SOTA learning-based detectors in terms of accuracy and real-time performance [13]. Although some of the learning-based object detectors trained with on-board datasets can be implemented directly in roadside sensors without further training [24, 25], the system performance is still inevitably limited by the lack of training data.

Considering the existence of the physical occlusion, it is extremely difficult to further unlock the bottleneck for the detection performance of a single-sensor-based perception system. Perceiving the environment from spatially separated sensors is increasingly becoming a promising solution to address the issues mentioned above [10].

Sensor fusion plays the most significant role in a CP system. In terms of the real-world communication bandwidth, the early-fusion scheme requires high-volume data transmission which is more likely to be satisfied in an infrastructure-based CP system, while late fusion suffers from the detection accuracy [11]. Recently, the deep-fusion scheme is an emerging way to support cooperative object perception by sharing the internal hidden features, which requires less communication power but contains more information than the detection results [12].

In this paper, the sensors are assumed to be equipped onboard and at the roadside, respectively. Since the two LiDAR sensors are spatially separated, preprocessing has to be done cooperatively in terms of the calibration matrix of these sensors.

Grid-wise Feature Fusion (GFF) is one of the most important stages in PillarGrid, which fuses multi-sensor data based on a mechanism with high expandability. Although only two sensors are deployed in this paper to illustrate our method, GFF can fuse features generated from more than two sensors.

Fig. 3 depicts the core process in GFF. The whole process of GFF can be divided into three stages. First, map pillar features with the size of $(C,P)$ into a grid format with the size of $(C,H,W)$ according to the pillar index generated in the voxelization stage. Specifically, $H$ and $W$ are two dimensions of the grid, which are related to the size of voxels and PCD geo-fencing range $\Omega$.

After transforming the feature data into the grid format, each grid contains the hidden features representing the PCD at the specific spatial location. The grid feature from each sensor is then augmented by an additional dimension to enable grid-on-grid fusion by unsqueezing from the size of $(C,H,W)$ to $(C,H,W,1)$. Then concatenation process is implemented to all the features in terms of the last dimension, i.e., generating the output size of $(C,H,W,F)$ where $F$ is the number of fused sensors, i.e., $F=2$ in this paper.

As demonstrated in Fig. 3, a max operation layer is applied to fuse features over different channels. A 3D max-pooling is implemented to generate the output with the size of $(C,H,W,1)$, followed by a squeeze operation to transform the size to $(C,H,W)$ which is compatible with 2D CNN backbones.

A Region Proposal Network (RPN) [6] is applied as the backbone whose structure is shown as Fig. 4.

The backbone consists of two sub-networks: 1) one 2D convolutional (Conv2D) layer-based network that generates the extracted features with increasingly small spatial resolution; and 2) one deconvolutional (DeConv2D) layer-based network that generates output features by performing upsampling and concatenation. Each Conv2D block consists of one Conv2D layer with the kernel of $(3,2,1)$, followed by several Conv2D layers with kernels of $(3,1,1)$. Specifically, the numbers of Conv2D layers in each block are $4$, $6$, and $6$, respectively.

An anchor-based 3D detection head [5] is applied to generate 3D bounding boxes. Intersection over Union (IoU) is implemented to match the prior boxes to the ground truth. In this paper, since only vehicles and pedestrians are detected, the number of classes is set to be $2$.

Ground truth boxes and anchors are defined by a 7-dimensional vector $(x,y,z,w,l,h,\theta)$ where $w$, $l$, $h$, and $\theta$ represent the width, length, height, and yaw angle, respectively. The loss function is defined as:

where the superscript $gt$ and $a$ represent the ground truth and anchor, respectively; and $d^{a}$ is defined by:

The total localization loss is:

The object classification loss is defined as:

where $p^{a}$ is the class probability of an anchor; and $\alpha$ and $\gamma$ are set to be $0.25$ and $2$, respectively. Hence, the total loss is:

where $N_{pos}$ is the number of positive anchors; and $\beta_{loc}$, $\beta_{cls}$ and $\beta_{dir}$ are set as $2$, $1$, and $0.2$, respectively.

Since so far, no dataset is available for supporting cooperative perception (CP) between one onboard LiDAR and one roadside LiDAR, a CP platform is developed to collect sensor data and ground truth labels based on the CARLA simulator [26]. In addition, to ease the re-usability and accessibility, we organize the new dataset to be compatible with existing open-source training platforms (e.g., OpenMMLab [27]), by following the KITTI’s format [8]. However, the KITTI’s settings and CARLA have different coordinate systems, as shown in Fig. 5. Therefore, a transformation is designed to generate the ground truth labels in the KITTI coordinate.

This dataset (named “CARTI”) consists of data from two LiDAR sensors, one is equipped at the roadside (i.e., on a traffic signal pole of an intersection) with the height of $3.74m$, while the other LiDAR is mounted on the top a vehicle with the height of $1.74m$. The simulation is running in a synchronization mode at $10$Hz. Data is collected at $2$Hz, and totally $11,000$ frames of 3D point clouds are available, including $8,000$ for training, $1,000$ for evaluation, and $2,000$ for testing. Fig. 6 shows the visualized 3D point clouds and ground truth labels in one data frame.

Two benchmarks are applied in terms of object detection performance: bird’s-eye view (BEV) and 3D perspective. Average precision (AP) and average recall (AR) are measured for each class under the IoU of 0.50 for cars and 0.25 for pedestrians. Mean average precision (mAP) is calculated across all the classes to illustrate the overall performance of the model.

As shown in Table I and Table II, PillarGrid outperforms all baselines in terms of mAP for both BEV and 3D benchmarks. Specifically, for BEV detection, PillarGrid outperforms 44.96% for onboard detection mAP and 23.72% for roadside detection mAP. For 3D detection, PillarGrid outperforms 90.52% for onboard detection mAP and 25.39% for roadside detection mAP.

For vehicle detection, in both benchmarks, PillarGrid surpasses all the baselines with a large margin in terms of both precision and recall. The results also validate the feasibility of the core idea of the cooperative perception shown in Fig. 1, which represents that PillarGrid can take advantage of both onboard and roadside point clouds. For pedestrian detection, PillarGrid outperforms the baselines in terms of the AP but has similar AR performance to roadside detection. According to the Pedestrian AR from onboard detection (lower than 1%), one hypothesis is that current voxel settings are too large ($0.2m\times 0.2m$) to generate effective pillar features for pedestrians (with the size of $0.38m\times 0.38m$), which may cause the slight degradation of the fused features.

To further illustrate the performance of PillarGrid, we provide qualitative analysis results, as shown in Fig. 7. The left, middle and right subfigures depict the detection results for roadside baseline, onboard baseline, and PillarGrid, respectively, where wrongly detected results are highlighted.

From the perspective of precision, both roadside detection and onboard detection have one wrong detection result marked as False Positive (FP). In the right subfigure of Fig. 7, i.e., results from PillarGrid, the Grid-wise Feature Fusion process can successfully merge the feature-level information from two spatially separated LiDARs to improve the detection precision (e.g., detection marked as fused feature).

To depict the performance of PillarGrid in terms of recall, we focus on the roadside LiDAR point cloud data in the right subfigure of Fig. 7. Due to the physical occlusion and sensing range, two vehicles located at the lower-left corner cannot be detected by the roadside LiDAR (due to the building blockage) and another two vehicles on the top of the subfigure cannot be detected by the onboard LiDAR. However, PillarGrid can detect all of these occluded or out-of-range vehicles by fusing the features from both onboard and roadside sensors, which further demonstrates the capability of our deep fusion-based cooperative perception method.