MAFF-Net: Filter False Positive for 3D Vehicle Detection with Multi-modal Adaptive Feature Fusion

3D vehicle detection using only point clouds can be roughly divided into 3 types: Bird’s Eye View(BEV)-based, Voxel-based, and Point-based. BEV-based methods[17, 18] first use rules[19] to convert the point cloud into the BEV form, and then uses the 2D CNN network for learning and prediction[20]. Voxel-based methods divides the point cloud space into regularly arranged voxel blocks, use rules[21, 22, 23] or neural networks[24, 25] to encode the voxels, and then adopt 3D or 2D convolution[2, 26, 4, 27, 28] for 3D detection. PointPillars[1] further simplified voxels into pillars, realizing a real-time one-stage 3D detection method. The voxel-based methods are conducive to accurate 3D proposal generation but are also limited by the receptive field of 2D/3D convolution. The point-based method mainly uses PointNet[29, 30] technology to encode 3D points, and then uses more strategies[3, 5, 6] to process 3D points to achieve the purpose of accurately predicting 3D vehicles. Most of these point-based methods are based on the PointNet series, which makes the receptive fields of point cloud feature learning more flexible. Recently, there has been a trend of fusing point-based and voxel-based methods[15, 13] to utilize the best of two worlds, thereby improving 3D detection performance.

In order to take advantages of the camera and lidar sensors, various fusion methods have also been proposed. MV3D[31] is a pioneering work in the fusion method. It combines the lidar BEV representation, front view and image together, and proposes a two-stage network. AVOD[32] fuses the features of the BEV and the image in the middle layer of the convolution to predict 3D objects. ContFuse[33] uses continuous convolution to fuse images and lidar features on different resolutions. MMF adds ground estimation and depth estimation to the fusion framework, and learns better fusion feature representations while jointly learning multi-tasks. [34, 35, 36] first use camera images to generate proposals and then exploit some methods to process the lidar points in these regions to generate 3D objects. [10, 37] use 2D tasks (detection or segmentation) to obtain the feature representations of the image, then fuse these feature representations with the point cloud, and finally use the architecture of VoxelNet to process the fused features.

Although various multi-modal fusion methods have been proposed, they have seldom paid attention to the specific gains brought by the image for 3D detection and the detection speeds are often very slow. In this study, we focus on using images to eliminate the FP of 3D vehicle detection and hope to obtain a fast multi-modal fusion method.

Compared with lidar point clouds, images have rich color and texture information, which is very effective for overcoming the false detection of point clouds, especially the background false detection. In this paper, we only use the raw RGB features of the image as the input of MAFF-Net. MAFF-Net does not rely on any 2D annotation information and the high-level features of any 2D tasks.

The PointPillars structure is used as the base 3D detection network for two main reasons: (i) it achieves a good balance between speed and performance: with excellent performance, it has a very fast detection speed. A lot of work in academia and industry is also based on this algorithm[27, 28, 38, 39, 40, 41]. (ii) It provides a natural and effective interface for fusing image features at different granularities in 3D space such as points and pillars. The network used in this study is described in [1]. For completeness, this section briefly reviews PointPillars. This algorithm consists of three modules: (i) a Pillar Feature Net(PFN); (ii) 2D Convolutional Neural Networks; (iii) a Detection Head.

PFN is a feature learning network designed to encode raw point clouds for individual pillars. All non-empty pillars are encoded by PFN and share the same network parameters. The encoded features are scattered back to the original pillar positions to construct a pseudo-image. The pseudo-image features are forwarded through a series of 2D convolutional blocks to extract high-level features. The features are then used by a detection head to generate the targets.

In this study, two concise technologies that can fuse raw RGB data with the point cloud data are proposed to filter false positive in 3D detection.

PointAttentionFusion(PAF): This is an early and simple fusion approach in which each 3D point is aggregated through an image feature and two attention features to achieve the adaptive fusion features. Fig. 3 presents the procedure of this technology.

The method first uses the calibration matrix to project each 3D point onto the image, and the projected image features can be obtained according to the corresponding projection location index. Note that these features are the raw RGB features. In order to make the projected image features and point cloud features compatible, a simple fully connected network called $\mathrm{MLP_{PD}}$ is applied to map the image features to appropriate dimensions. The $\mathrm{MLP_{PD}}$ is composed of a set of blocks and each block consists of a linear layer, a BN layer, and a ReLU layer. Next, the point cloud features and the mapped image features are concatenated channel-wise to obtain point-wise extended features. However, since the image has many noise factors, such as occlusion, truncation, etc., the extended features will introduce interference information.

To address this issue, we adopt the point-wise channel attention module, which uses expanded features to adaptively estimate the importance of each type of features in a channel-wise manner. First, feed the extended features into a fully connected layer that includes a linear layer, a ReLU layer, and a linear layer. Then output the feature weights through a sigmoid, and finally multiply the weights with the corresponding features in a channel-wise manner to get the attention features. The above process is used to process the point cloud features and image features respectively. The specific forms of channel attention are as follows:

	$\displaystyle\mathbf{F_{a.P}}$	$\displaystyle=\mathbf{F_{P}}\otimes\underbrace{\sigma(\mathrm{MLP_{P}}(\mathbf{F_{E}}))}_{\text{Point cloud attention}}$		(1)
	$\displaystyle\mathbf{F_{a.I}}$	$\displaystyle=\mathbf{F_{I}}\otimes\underbrace{\sigma(\mathrm{MLP_{I}}(\mathbf{F_{E}}))}_{\text{Image attention}}$

where $\mathbf{F_{P}}$ and $\mathbf{F_{I}}$ represent point cloud and image features, respectively, $\mathbf{F_{a.P}}$ and $\mathbf{F_{a.I}}$ are the corresponding point-wise attention features, $\mathbf{F_{E}}$ is the extended point image features, $\sigma$ is the sigmoid activation function and $\otimes$ is the element-wise product operator. After obtaining the attention features, concatenate $\mathbf{F_{P}}$, $\mathbf{F_{I}}$, $\mathbf{F_{a.P}}$ and $\mathbf{F_{a.I}}$ channel-wise to get the point-wise fusion feature. Then divide the 3D point cloud space into pillars, followed by grouping the points to pillars. Finally, the simplified version of PointNet is adopted to generate the pillar-wise fusion features.

The advantage of this method is that since the fusion method is simple and the added networks are all shallow networks, the approach can achieve fast detection speed. Moreover, the approach can learn to summarize useful information from both modalities through PFN layer because of the early stage fusion strategy.

DenseAttentionFusion(DAF): In contrast to PointAttentionFusion that fuses features in a concise manner, DenseAttentionFusion employs a relatively complex fusion strategy, where features are divided into three forms and fused together. As shown in Fig. 4, the three features are the point cloud features, the image features, and the extended point image features $\mathbf{F_{E}}$ described in PointAttentionFusion.

In PointAttentionFusion, only extended point image features $\mathbf{F_{E}}$ is used for subsequent work. However, after PFN learning, the feature is equivalent to the global features of the point cloud and the image. This feature will lose some characteristics of the original two modalities. Thus, a naive approach would be to put the original point cloud features and image features back into the global features. However, due to the noise information of the image features and the redundancy of the three features, blindly fusing the three features will introduce lots of interference information, resulting in the degradation of detection performance. In the following, we describe a novel dense attention fusion method that effectively combines the three types of features.

First, after projecting 3D points onto the image, the corresponding RGB features can be obtained. The RGB features have two uses, one is to expand the point cloud features in the same way as PointAttentionFusion to obtain the point cloud image features, and the other can regard this point-wise image feature as another feature form of 3D points. In this way, we can obtain point cloud features, point cloud image mixed features, and projected image features. Because the above three features are all in point-wise form, they can use the $x,y$ and $z$ coordinates of the 3D point to perform pillar operations, respectively. Here, we can get the pillar representations of three different features, and then three PFNs with the same structure are adopted to encode the pillars to generate three type of pillar-wise features.

Next, we use the pillar-wise channel attention module to adaptively estimate the importance of each pillar feature. The above three features are concatenated together as the input of this module. The attention features of the three features are estimated by the following attention maps:

	$\displaystyle\mathbf{F_{a.P}}$	$\displaystyle=\mathbf{F_{P}}\otimes\underbrace{\sigma(\mathrm{MLP_{P}}(\mathbf{F_{C}}))}_{\text{Point cloud attention}}$		(2)
	$\displaystyle\mathbf{F_{a.PI}}$	$\displaystyle=\mathbf{F_{PI}}\otimes\underbrace{\sigma(\mathrm{MLP_{PI}}(\mathbf{F_{C}}))}_{\text{Point Image attention}}$
	$\displaystyle\mathbf{F_{a.I}}$	$\displaystyle=\mathbf{F_{I}}\otimes\underbrace{\sigma(\mathrm{MLP_{I}}(\mathbf{F_{C}}))}_{\text{Image attention}}$
	$\displaystyle\mathbf{F_{A}}$	$\displaystyle=\mathbf{F_{a.P}}+\mathbf{F_{a.PI}}+\mathbf{F_{a.I}}$

where $\mathbf{F_{P}}$, $\mathbf{F_{PI}}$ and $\mathbf{F_{I}}$ represent the pillar-wise point cloud, point image and image features, respectively, $\mathbf{F_{C}}$ is the concatenated feature of the above three features, $\mathbf{F_{a.P}}$, $\mathbf{F_{a.PI}}$ and $\mathbf{F_{a.I}}$ are the corresponding pillar-wise attention features, $\mathbf{F_{A}}$ is the pillar-wise attention features that need to be obtained, $\sigma$ is the sigmoid activation function and $\otimes$ represents the element-wise product operator. After obtaining the attention features, concatenate $\mathbf{F_{P}}$, $\mathbf{F_{PI}}$, $\mathbf{F_{I}}$, and $\mathbf{F_{A}}$ channel-wise to get the pillar-wise fusion features.

Although DenseAttentionFusion is a relatively complex fusion strategy, it has the following advantages. First, it fully retains the original characteristics of the three features of point cloud, image, and point cloud image, while minimizing the noise impact of each feature. This makes its detection performance better than PointAttentionFusion. Second, the fusion is based on Pillars features, which can reduce the dependence on the availability of high-resolution 3D points.

Network Architecture: For the fairness of comparison, we keep most of the settings of PointPillars as described in [1] except for some newly added module structures. According to the type of fusion, the input and output dimensions of the PFN layers and the MLP layers are different. Note that, except for the input dimensions, the structure of 2D convolutional network in the proposed MAFF-Net is the same as PointPillars.

For PointAttentionFusion, the configuration of image dimension prediction $\mathrm{MLP_{PD}}$ is (3,96,16). The $\mathrm{MLP_{P}}$ and $\mathrm{MLP_{I}}$ in the attention module have (9+16,25,9) and (9+16,25,16) configurations respectively, where 9 is the dimension of the point cloud feature and 16 is the dimension of the image feature. Next, the configuration of PFN layer is (9+16+9+16,64), which leads to the input dimension of the subsequent 2D convolutional network is 64.

For DenseAttentionFusion, the configuration of $\mathrm{MLP_{PD}}$ is the same as the $\mathrm{MLP_{PD}}$ in PointAttentionFusion. $\mathrm{PFN_{P}}$, $\mathrm{PFN_{PI}}$ and $\mathrm{PFN_{I}}$ have configurations of (16, 64), (9+16, 64) and (3, 64) respectively. The configuration of the three MLPs in the attention module are all (64*3, 64*3, 64), but their weights are not shared. The final fusion feature is combined by four types of features, so its dimension is 64*4, which leads to the input dimension of the 2D convolutional network is 64*4.

Loss: We use the same loss functions described in PointPillars. The loss function is divided into 3 types, namely the localization regression loss $\mathcal{L}_{loc}$, the classification loss $\mathcal{L}_{cls}$, and the orientation loss $\mathcal{L}_{dir}$. $\mathcal{L}_{loc}$ uses SmoothL1 function to define the loss for $(x,y,z,w,l,h,\theta)$, $\mathcal{L}_{cls}$ uses focal loss, and $\mathcal{L}_{dir}$ uses a softmax classification loss. The overall loss function can be defined as:

$$\mathcal{L}_{total}=\frac{1}{{N_{pos}}}\left({\beta_{loc}\mathcal{L}_{loc}{\rm{+}}\beta_{cls}\mathcal{L}_{cls}{\rm{+}}\beta_{dir}\mathcal{L}_{dir}}\right)$$

(3)

where $\beta_{loc}=2.0,\ \beta_{cls}=1.0,\ \beta_{dir}=0.2$, and $N_{pos}$ is the number of positive anchors.

Data Augmentation: Our two fusion methods both project the 3D points to the image in a point-wise manner, so all the data augmentation methods of PointPillars can be used, which is similar to PointPainting[9]. Data augmentation adopts sample objects from database, augment ground truths independently, and perform global augmentations for the whole point cloud and all boxes.

Dataset and Implementation: This study uses the KITTI benchmark dataset [12] to evaluate the proposed fusion method, which includes 7481 training samples and 7518 testing samples. There are three levels of difficulty: easy, moderate and hard, which are assessed based on the object height in image, occlusion and truncation. The rank of leaderboard is based on the moderate result. We follow the general approach proposed by [31] to split the training samples into a training set of 3712 samples and a validation set of 3769 samples. The proposed MAFF-Net is compared with previously published method on the car category. Inference of the entire network is carried out on Tesla V100 GPU. The code of PointPillars used in this study comes from (httP://github.com/nutonomy/second.pytorch).

Metric: The metric of KITTI is defined by average precision(AP) on 40 recall positions of the Precision/Recall curve [42] with IoU=0.7. This metric is a good indicator of the overall performance, but it cannot effectively reflect the details of the performance, such as the FP that we are concerned about. To show the performance of filtering FP, a simple way can be used: for each recall position, the number of TP is the same. According to the formula of precision $precision=\frac{TP}{TP+FP}$, when TP remains unchanged, the less FP, the higher precision. Thus, the precision at a single recall position can reflect the performance of the algorithm processing FP. In this study, we use precision of five recall positions ($0.725,0.75,0.775,0.8$) to verify the proposed method performance for filtering FP. The reason for choosing these positions is that the scores of detection at these positions are generally at a medium level and it is difficult to filter vehicles only using the score threshold, which will lead to many FP.

Evaluation on filtering FP. Table I shows the comparison of the 3D detection performance of PointPillars and MAFF-Net at different recall positions. The performance of each position has been significantly improved. As recall increases (in this case, the detection score decreases and FP is easier to increase), the improvement of mAP gradually increases. For the moderate category that is concerned on the KITTI benchmark, when the recall is 0.8, the proposed fusion method improves the performance by 4.22%. For a single recall position, only reducing FP can improve performance because of the same number of TP. Therefore, the performance improvement on each single recall position shows that MAFF-Net is effective for filtering FP.

In order to reflect the performance of filtering FP more intuitively, Table II lists the number of TP and FP for the 3D vehicle detection under different score thresholds. When the score are 0.4 and 0.1, the FP are reduced by 11.18% and 11.08%, respectively, and the FP caused by the background are reduced by 18.75% and 12.58%, respectively. It shows that the proposed two fusion methods can effectively reduce FP while slightly improving TP, especially for reducing FP caused by background.

Analysis of the attention mechanisms. We visualize the point cloud features in DenseAttentionFusion before ($\mathbf{F_{P}}$) and after ($\mathbf{F_{a.P}}$) using attention, as shown in Figure 6. After using the attention module, the background area is fully suppressed, so that the geometric shape of the vehicles can be highlighted. Moreover, for some objects that are similar to the vehicle in 3D structure, the attention module can eliminate some of their shapes, so that their 3D structures are no longer similar to the vehicle.

In order to further show the role of the attention module, Table III shows the performance comparison of the KITTI validation set with and without the attention module. When the attention module is not used, compared to PointPillars, the two fusion methods have improved performance in the easy category, but the performance in the moderate and hard categories decreases. In the moderate and hard categories, interference information is very serious. In these cases, when the point cloud is projected to the image, it will get the wrong image features, which will lead to the degradation of detection performance. After adding the attention module, the MAFF-Net is able to adapt to the weight of each modal feature, thereby improving the detection performance.

The visualization of Detection Results. Figure 5 provides a qualitative analysis of some 3D detection results. It can be observed from Figure 5 that the 3D structures of many false detection objects are similar to that of vehicles, such as the hierarchical tree clusters on the left side of the second column image, the object clusters on the right side of the third column image, and wall and fence on the left side of the fourth column image. These conditions will make it difficult for the lidar point cloud to distinguish the objects. However, MAFF-Net can effectively reduce the above false detection objects by using image information, which demonstrates the effectiveness of the proposed algorithm.

We evaluate the proposed MAFF-Net with DenseAttentionFusion on the KITTI testing set by submitting the test results to the official server. The results are summarized in Table IV. It can be observed that MAFF-Net with DenseAttentionFusion can have competitive results compared with other state-of-the-art multimodal fusion algorithms. The proposed method has the fastest speed, achieves top rank in one category and 2nd rank in three categories within the fusion methods. It is worth pointing out that the results of the test set we submitted(PointPillars(baseline)) using the code given in PointPillars[1] are not ideal, for example, it is lower than the original PointPillars 1.1% in the 3D moderate category. However, even if only the low-performance PointPillars are used, the detection performance of MAFF-Net is still better than the original PointPillars in most categories. In 3D moderate, MAFF-Net is 1.9% higher than PointPillars (baseline) and 0.7% higher than PointPillars. Therefore, if a more powerful point cloud backbone network can be used, MAFF-Net will achieve better performance.