BiFNet: Bidirectional Fusion Network for Road Segmentation ^††thanks: This work is supported by the Beijing Science and Technology Plan under Grants Z191100007419002, and the National Natural Science Foundation of China (NSFC) under Grants No. 61803371.

Current image-based methods can be divided into two categories: model-based and learning-based. Model-based methods[18][19][20] build the road model based on unique shapes and textures. Mohamed et al.[21] detected the lanes according to the shape color and figured out the road area in the image. Ankit et al.[22] used the appearance features of the road surface and map prior to implement the automatic labeling road area. The model built by expert domain knowledge needed to tune according to the different datasets or scenes. Model-based methods rely on the robust road model which are applicable to scenes with the similar road. For the complex illumination and textures, these methods have bad adaptive ability.

Learning-based methods are popular at road segmentation task. These methods learn the robust classifier to identify the road pixel in the image fed by large labeled datasets. Profiting by the development of deep learning and convolutional neural networks, hand-designed features[23] used for classifier are replaced with automatic learning features[12][24]. At the same time, the development of semantic segmentation methods has greatly advanced road detection methods. Since the road has a special appearance, Zhe et al.[25] added the road boundary constraints to the deep segmentation networks and improved the performance of methods.

Although image-based segmentation methods have developed rapidly in recent years, the image noise caused by the illumination leads to the limitation of the image-based methods. As a distance sensor, LiDAR is not affected by light, and many researchers use the point cloud to detect road. Fernandes et al.[26] introduced the similarity of the histogram of the point cloud as the discriminator to classify the points and segmented the road from the point cloud. Caltagirone et al.[13] employed the fully convolutional neural networks over the BEV of the point cloud and achieved the desired result. Zhang et al.[27] proposed a sliding-beam method to segment the road by using the off-road data and applied a curb-detection method to obtain the position of curbs for each road segments.

Depth information is the key to solve the problem of RGB image. How to combine depth information with color image effectively has always been a research hotspot[28][29]. Since the data of the camera and LiDAR can complement each other, fusing image and point cloud to segment road has gradually become the mainstream method. Shinzato et al.[30] proposed a fusion method which combined the distribution of the point cloud and image pixels as the discriminator. Conditional Random Field(CRF) was used for constructing the fusion model. There are many methods[31][32][33] which constructed Gibbs energy by using Euclidean distance between the points and RGB values of pixels. The above methods needed the expert knowledge to build the data features and combined the features to find robust classifier for the pixels. Gu et al.[15][34] used the CRF to fuse the results from the point cloud[35] with image features extracted by deep convolutional neural networks. Compared to the hand-selected features, this method achieved better performance. Caltagirone et al.[13] projected the point cloud into camera space and designed a cross fusion module to fuse image features with front view features of the point cloud. Since the coordinate of the point can not directly reflect the road characteristics, Chen et al.[17] proposed an altitude difference transformation which transformed the point cloud to a special front view image. And they introduced the feature space adaptation module to fuse the features from different sensors.

Image-based 2D object detection and tracking have made great progress[36][37][38][39]. Since the point cloud of LiDAR has accurate distance information[40], fusing the camera image with the point cloud also plays an important role in 3D object detection. The categories of the fusion for object detection can be divided into a region-based method and point-based method. The region-based method combined the sensors data or features from a unique 3D region and got a better feature representation[41][42]. Compared with the region-based methods, point-based methods fused the features in terms of the point and the pixel. Continuous fusion layer[43] belongs to this category. It built the relationship of pixels in the camera image and the BEV-based on 3D points and designed the continuous fusion which constructed a mapping between sparse points and BEV pixels. You et al.[44] proposed an attention fusion block which cascaded the global context feature of camera image and feature of each point. PointFusion[45] used such a method to implement 3D object detection. Wang et al.[46] proposed a sparse non-homogeneous pooling and used projection matrix to transform features between the RGB image and the BEV.

The image and point cloud can be obtained by observing the objects in 3D space with different sensors. Image is the 2D data of the camera through the principle of the hole imaging, compressing the spatial structure and retaining the texture. Far objects will scale proportionally on the imaging plane. The far road will be compressed and the proportion is much smaller than the near road proportion, which results in the perspective image based methods paying more attention to the accuracy of the near road. Point cloud is the spatial data obtained by LiDAR through scanning a beam of laser to image an object. It retains the spatial structure but loses texture. Chen et al.[41] proposed two projection methods which converted the point cloud into ordered data which we use to upsample the height. BEV is one of the common preprocessing methods used in road segmentation. It compresses only the height of the point cloud and preserves the geometry of the road. In this section, we build the relationship between the camera space and BEV space as shown in Fig. 2.

Since the imaging principle and measuring range of each sensor are different, the abilities of sensors to express various environments in diverse scenarios are different. The purpose of the multi-sensor fusion is obtaining the consistent and effective representation of the environment for various scenarios. In order to achieve this effect, the direct methods are projecting the LiDAR features into camera space and adding the features into image features. The problem of this direct linear addition method is that it does not consider the different expressive abilities of sensors for various scenarios, which may result in the confusion of features when the difference between the two features is large. Inspired by SENet[47], we design a context-based adaptive feature fusion module, as shown in Fig. 3. According to the scene expression ability of different sensor features, the module can stimulate the good features and suppress the bad features, so as to get a more effective feature expression.

For the feature maps from convolutional neural networks, we argue that different channel features represent different context of the environment. We can get the feature representation ability $\kappa$ of the current feature map as follows

Here $H$ and $W$ are the height and width of the feature maps, respectively. $f_{h,w,*}$ is the corresponding vector at $(w,h)$ of the feature map. which is insensitive to noise compared with max pooling.

After obtaining the expressive power of different feature channels, we need to determine which channels are conducive to the segmentation task and which ones confuse features. SENet pointed out that suppressing the poor features and encouraging the better feature is conducive to more effective feature expression. Therefore, we need to consider the relationship between channels, to evaluate the favorable feature channels and the disadvantageous feature channels. Here we use a three-layer perceptron(MLP) to predict the gain of each channel for the current segmentation task in the current scenario.

where $\hat{\mu}$ represents context quality for the features. And the $\sigma(\cdot)$ sigmoid function at the output layer is used to scale the weights into $[0,1]$. $\kappa$ is the feature representation ability vector. $\psi$ is the fully connected layer, and $\Theta$ means the weight of the layer. At the middle layer of MLP, we use ReLU as the activation function. And the final feature in each channel is

Here $f_{c}$, $\hat{\mu}_{c}$ and $\hat{f}_{c}$ are the original features, context quality and reweighted features of the channel $c$, respectively. For the feature of image and point cloud, we fuse the weighted features based on context to get the final consistent feature.

Based on the dense space transformation module and the context-based fusion module, we design the bidirectional fusion module to fuse the features from different deep features. Considering the different imaging modes of images and point clouds, their data sources and data expressions are also different. There may be differences in scale and dimension arrangement between the two group features. Therefore, we add a domain transformation(DT) module between the DST and CBF. And it is realized by $1\times 1$ convolution.

Here $f_{i,j,c}$ is $i$-th row, $j$-th column and $c$-th channel element of the feature map. $f^{D}_{i,j,k}$ is the corresponding domain transformed feature with $k$ channels. $\Theta^{DT}$ is the parameter of the convolution.

Based on this module, we construct the bidirectional fusion networks. Since ResNet[48] achieves the excellent performance in segmentation tasks, we also use ResNet as the backbone network to extract features of image and BEV of point cloud respectively. After the output of two parallel ResNet, we insert a bidirectional fusion module to realize feature fusion. The whole network architecture is shown in Fig. 4.

In order to make the network converge faster and better, we adopt the multi-task learning to train the network. We introduce the road segmentation loss in camera space and BEV. Note that the loss of camera space can make the network see objects farther away, while the loss of BEV can make the distant and near objects distribute equally. For each pixel in camera space, we use the focal loss[49] for the imbalance between road and background

Here $\mathcal{L}^{I}_{p}$ means the loss of the pixel $p$ in image $I$. Where $\alpha$ reduces the impact of the imbalance between the classes. $\gamma$ controls the weight of the hard examples. $g$ is the pixel label, and $\hat{y}$ is the estimated probability belonging to the road. Multi-task learning are widely used for objects detection and segmentation[50]. And we use the multi-task objective function to train the whole networks

Here $N$ is the number of the pixels in one batch. $\beta$ is the weight for balancing the different loss from different space. The superscripts $I,L$ is corresponding to the camera space and LiDAR BEV space.

KITTI is a well-know dataset which provides 289 training images and 290 testing images for road segmentation tasks. The training set and the test set contain three different road scenarios including Urban Marked road(UM), Urban Multiple Marked road(UMM), and Urban Unmarked road(UU).

In the KITTI dataset, the LiDAR data format is 3D point cloud. Because of the disorder of 3D point cloud, convolutional neural networks can not be directly applied to point cloud. As mentioned earlier, the LiDAR input part of our algorithm is the top view projection of 3D point cloud. The resolution of the BEV is 0.5 meter. Each pixel in the overhead projection contains a vector, which has 6 attributes, namely, the maximum height value, the minimum height value, the average height value, the maximum intensity value, the minimum intensity value and the average intensity value of the point set falling into the pixel area. The default value is 0.

We use the evaluation metrics in literature[52] to analyze our algorithm. The evaluation measures include precision(PRE), recall(REC) and pixelwise maximum F-measure (MaxF). PRE measures the proportion of correctly segmented pixels, and REC reflects the missing detection of the algorithm. The calculations are as follows.

Here TP, FP and FN are true positive, false positive and false negitive, respectively. There is a bit of trade-off between PRE and REC. MaxF is designed as a comprehensive metric considering PRE and REC. Another comprehensive metric is the average precision(AP), which provides insights into the performance over the full recall range. The calculations are as follow.

Here $\zeta$ represents recall threshold. All metrics are calculated in BEV space.

In order to evaluate the algorithm in the training process, we select 58 images from 289 training sets as validation sets. In the process of algorithm analysis, in order to reduce computing resources and accelerate network training, ResNet-18 is used as the basic network to verify the modules designed in this paper. We use Adam algorithm to train the network with parameters shown in Table II, and the learning rate is decaying linearly with training epoch. In the training process, the learning rate decreases linearly with time. In the KITTI road experiment, we use ResNet-101[48] and ResNet-50[48] as the basic networks of the image and BEV branch respectively. The two networks are trained separately first. During the training, symmetric transformation, translation transformation and scaling transformation are employed for the data augmentation.

To compare with state-of-the-art road detection methods, we estimate the BiFNet on the KITTI road benchmark validation set. In this experiment, we use the ResNet-101 and ResNet-50 as the backbones for the camera image and the BEV of the point cloud. We use the 5-fold cross validation on the training set and obtain the mean of the results. The overall results of BiFNet and other state-of-the-art road detection methods are shown in Table III. LC-CRF[15], LidCamNet[16] and PLARD[17] are the multi-sensor fusion-based methods. LC-CRF fused the features from camera and perspective views of LiDAR while LidCamNet used the learnable weights to add two different features. Since the proposed method has better scene adaptability, it achieves the better performance than the above two methods. According to the results, our proposed method achieves state-of-the-art in UMM scenario and competitive results in other scenarios.

Compared with PLARD, BiFNet achieves slightly better results according to Table III. However, the results of PLARD shown above are integrated from 3 different models, while BiFNet only needs single model.

In order to verify the validity of different components of the proposed BiFNet, we conduct a set of ablation experiments on KITTI dataset. The training set of KITTI is split into two parts. One is used to train the networks while the other is for validation.

Wange et al.[46] proposed sparse non-homogeneous pooling(SHPL) module to realize the fusion of image and LiDAR BEV. Unlike DST in this paper, SHPL is based on the points of the point cloud to establish the relationship between image and BEV. Because the points are sparse, this transformation is sparse. Only a small proportion of pixels in feature maps have corresponding feature vectors. In this paper, DST is a dense transformation, which realizes the transformation relationship of each pixel of feature maps in different spaces.

In order to verify the validity of this dense transformation, we set up a comparison experiment with SHPL. During the comparison, element-wise addition is used for features fusion. The experimental results are shown in Table V. From the results, we see that the two methods have the same effect on BEV branch. For the perspective branch, DST outperforms SHPL significantly. The reason is that DST is a dense spatial transformation method, which can establish the mapping relationship between each pixel and the target feature at different scales, and thus obtain dense transformation features. In the experimental results, an interesting phenomenon is that with the sparse mapping, SHPL achieves the good performance on BEV space. One possible reason is that since the input LiDAR BEV is sparse, the neural network has adapted to the sparsity in BEV space, and has the ability to learn from the sparse input to segment the road, but the perspective image is dense, and the network can not adapt to the sparsity from the point cloud. This also proves the advantages of DST.

In order to verify the validity and universality of our proposed CBF module, we reproduce LidCamNet[16] and PLARD[17] methods, and migrate the module to these methods. The experimental results are shown in Table VI. In the reproduction process, we use ResNet-18 as feature extraction network, and the output features are used for fusion. Unlike the experimental results in PLARD, our reproduced LidCamNet is slightly better than PLARD. LidCamNet and PLARD both use the learnable weights for feature fusion. The difference is that LidCamNet only learns the weights of LiDAR, and each fusion connection has only one value. The difference of PLARD is that it sets a learnable weight for each pixel position of the feature, it is more adaptive than LidCamNet. However, the fusion weights of these methods remain unchanged for all sample during the inference, while the proposed method can adjust adaptively according to the content of samples. Therefore, applying CBF module to the above two methods will gain performance to a certain extent.