Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

In general, a few tasks can be accounted for as Driving Perception Tasks, containing Object Detection, Semantic segmentation, Depth Completion & Prediction, and so on[26, 71]. Here, we mainly focus on the first two tasks as one of the most concentrated research areas. Also, they cover tasks such as the detection of obstacles, traffic lights, traffic signs, and the segmentation of lanes or free space. We also briefly introduce some remaining tasks. An overview of perception tasks in the autonoumous driving is shown in Figure 1.

Object Detection

It is crucial for autonomous driving cars to understand the surrounding environment. Unmanned vehicles need to detect both stationary and moving obstacles on the road for safe driving. Object Detection, a traditional computer vision task, is widely-used in autonomous driving systems[108, 61]. Researchers build such framework for Obstacle Detection (Cars, Pedestrians, Cyclist, etc.), Traffic Light Detection, Traffic Sign Detection, and so on.

Generally speaking, object detection uses rectangles or cuboids represented by parameters to tightly bound instances of predefined categories, as cars or pedestrians, which needs to both excel in localization and classification. Because of the lack of depth channel, 2D object detection is often expressed as $(x,y,h,w,c)$ simply while 3D object detection bounding box is often represented as $(x,y,z,h,w,l,\theta,c)$.

Semantic Segmentation

Except for object detection, many autonomous driving perception tasks can be formulated as semantic segmentation. For example, free-space detection [35, 57, 107] is a basic module for many autonomous driving systems which classify the ground pixel into the drivable and non-drivable parts. Some Lane Detection [24, 84] methods also use the multi-class semantic segmentation mask to represent the different lanes on the road.

The essence of semantic segmentation is to cluster the basic components of input data, such as pixels and 3D points, into multiple regions containing specific semantic information. Specifically, semantic segmentation is that given a set of data, such as image pixels $D_{I}=\{d_{1},d_{2},...,d_{n}\}$ or LiDAR 3D point clouds $D_{L}=\{d_{1},d_{2},...,d_{n}\}$, and a predefined set of candidate labels $Y=\{y_{1},y_{2},y_{3},...,y_{k}\}$, we use a model to assign each pixel or point $d_{i}$ with selected one of the k semantic labels or the probabilities for all.

Other Perception Tasks

Besides the object detection and semantic segmentation mentioned above, the perception task in autonomous driving includes object classification [44], depth completion and prediction [26]. Object classification mainly solves the problem of determining the category given point clouds and images through a model. The depth completion and prediction tasks focus on predicting the distance for every pixel in the image from the viewer given LiDAR point cloud and image data. Although these tasks may benefit from multi-modal information, the fusion module is not widely discussed in these fields. As a result, we choose to omit such two tasks in this paper.

Although many other perception tasks are not covered in this paper, most can be regarded as object detection or semantic segmentation variants. Therefore, we focus on these two research works in this paper.

Over ten datasets [50, 71, 30, 58, 88, 47, 52, 80, 10, 94, 97, 7, 27, 101, 93, 56, 64] are related to autonomous driving percception. However, only three datasets are commonly used, including KITTI[26], Waymo[71], and nuScenes[6]. Here we summarize the detailed characteristics of these datasets in Table 1.

KITTI [26] open benchmark dataset, as one of the most used dataset for object detection in autonomous driving, contains 2D, 3D, and bird’s eye view detection tasks. Equipped with four high-resolution video cameras, a Velodyne laser scanner, and a state-of-the-art localization system, KITTI collected 7481 training images and 7518 test images as well as the corresponding point clouds. Only three objects are labeled as cars, pedestrians, and cyclists with more than 200k 3D object annotations divided into three categories: easy, medium, and hard by detection difficulty. For the KITTI object detection task, Average Precision is frequently used for comparison. Besides, Average Orientation Similarity is also used to assess the performance of jointly detecting objects and estimating their 3D orientation.

As one of the largest open datasets commonly used for autonomous driving benchmarks, the Waymo [71] open dataset was collected by five LiDAR sensors and five high-resolution pinhole cameras. Specifically, there are 798 scenes for training, 202 for validation, and 150 scenes for the test. Each scene spans 20 seconds with annotations in vehicles, cyclists, and pedestrians. For evaluating 3D object detection tasks, Waymo consists of four metrics: AP/L1, APH/L1, AP/L2, APH/L2. More specifically, AP and APH represent two different performance measurements, while L1 and L2 contain objects with different detection difficulties. As for APH, it is computed similar to AP but weighted by heading accuracy.

NuScenes [6] open dataset contains 1000 driving scenes with 700 for training, 150 for validation, and 150 for the test. Equipped with cameras, LiDAR, and radar sensors, nuScenes annotates 23 object classes in every key-frame, including different types of vehicles, pedestrians, and others. NuScenes uses AP, TP for detection performance evaluation. Besides, it proposed an innovative scalar score as the nuScenes detection score (NDS) calculated by AP, TP isolating different error types.

As the most commonly used sensor for data acquisition in 2D or 3D object detection and semantic segmentation tasks, the monocular camera provides RGB images rich in texture information [2, 36, 86]. Specifically, for every image pixel as $(u,v)$, it has a multiple channel feature vector as $F_{(u,v)}=\{R,G,B,...\}$ which usually contains the camera capture color decomposing in the red, blue, green channel or other manually designed feature as the gray-scale channel.

However, direct detecting objects in 3D space is relatively challenging because of the limited depth information, which is hard to be extracted by a monocular camera. Therefore, many works[43, 103, 11] use binocular or stereo camera systems through the spatial and temporal space to exploit additional information for 3D object detection, such as depth estimation, optical flow, and so on. For the extreme driving environment such as night or fog, some work also uses gated or infra-red cameras to improve the robustness[4].

As for 3D perception sensors, LiDARs use a laser system to scan the environment and generate a point cloud. It samples the points in the world coordinate system that denotes the intersection of the laser ray and the opacity surface. Generally speaking, the raw data of the most LiDARs is quaternion like $(x,y,z,r)$, which $r$ stands for the reflectance of each point. The different texture leads to different reflectance, which offers additional information in several tasks [31].

In order to incorporate the LiDAR data, some methods use points directly by a point-based feature extraction backbone[61, 62]. However, the quaternion representation of points suffers redundancy or speed drawbacks. Therefore, many researchers [108, 41, 66, 18] try to transform the point cloud into voxel or 2D projection before feeding it to the downstream modules.

Some work utilizes 3D CNN by discretizing the 3D space into 3D voxels, representing as $X_{v}=\{x_{1},x_{2},x_{3}...x_{n}\}$, where each $x_{i}$ stands for a feature vector as $x_{i}=\{s_{i},v_{i}\}$. $s_{i}$ stands for the centorid of the voxelized cuboid while $v_{i}$ stands for some statistical-based local information.

The local density a commonly used feature defined by the quantity of 3D points in the local voxel [8, 75]. Local offset is commonly defined as the offset between the point real-word coordinates and the local voxel centroid. Others may contain local linearity and local curvature[65, 74].

Recent work may consider a more reasonable discretizing way as cylinder-based voxelization[91], but Voxel-based Point Cloud Representation, unlike mentioned Point-based Point Cloud Representation above, dramatically reduces the redundancy of the unstructured point cloud [41]. Besides, being able to utilize 3D sparse convolutional techniques, the perception tasks achieve not only faster-training speed but also higher accuracy [41, 18].

Instead of proposing a new network structure, some works utilize sophisticated 2D CNN backbones to encode the point cloud. Specifically, they tried to project the LiDAR data into image space as two common types, including camera plane map (CPM) and bird’s eye view (BEV) [96, 41].

A CPM can be obtained with the extrinsic calibration by projecting every 3D point as $(x,y,z)$ into the camera coordinate system as $(u,v)$. Since the CPM has the same format as the camera image, they can be naturally fused by using the CPM as an additional channel. However, due to the lower resolution of LiDAR after the projection, the feature of many pixels in CPM is corrupted. Hence, some methods have been proposed to up-sample the feature map while others left them blank [39, 49].

Unlike CPM that directly projects LiDAR information into front-view image space, the BEV mapping provides an elevated view of a scene from above. It is utilized by the detection and localization tasks for two reasons. Firstly, unlike the camera installed behind the windscreen, most LiDARs are on the vehicle’s top with fewer occlusions [26]. Secondly, all objects are placed on the ground plane in the BEV, and models can generate predictions without distortion in length and width [26]. BEV components may vary. Some are directly converted from height, density, or intensity as point-based or voxel-based features [12], while others learn features of the LiDAR information in the pillars through feature extractor modules [41].

We divide the strong-fusion into four categories, as early-fusion, deep-fusion, late-fusion, and asymmetry-fusion, by the different combination stages of LiDAR and camera data representations. As the most studied fusion method, strong fusion shows a lot of outstanding achievements in recent years [55, 76, 77]. From the overview in Figure 3, it is easy to notice that each minor class in strong-fusion is highly dependent on the LiDAR point cloud instead of the camera data. We will then discuss each of them in particular.

Early-fusion. Unlike the traditional definition of data-level-fusion, which is a method that fuses data in each modality directly by a spatial alignment and projection at the raw data level, early-fusion fuses LiDAR data at the data level and camera data at data-level or feature-level. One example of early-fusion can be the model in Figure 4.

As for the LiDAR branch mentioned above, the point cloud can be used in the form of 3D points with reflectance, voxelized tensor, front-view/ range-view/ bird’s eye view, as well as pseudo-point clouds. Though all these data have different intrinsic characteristics, which are highly associated with the latter LiDAR backbone, most of these data are produced with a rule-based procession except pseudo-point clouds[79]. Besides, all these data representations of LiDAR can be visualized straightforward because the data in this stage still have interpretability compared to the embedding in feature space.

As for the image path, the strict data-level definition should only contain data as RGB or Gray, which lacks generality and rationality. Compared to the traditional definition of early-fusion, we here loosen camera data to data-level and feature-level data. Especially, here we treat semantic segmentation task results in image branch benefiting the 3D object detection as feature-level representation since these kinds of ”object-level” features are different from the final object-level proposals for the whole task.

[76] and [90] fused semantic-feature in image branch and raw lidar point clouds together which results in a better performance in object detection tasks. [68], and [20] also utilize semantic-feature, but different from the above methods, it pre-processes raw lidar point cloud into voxelized tensor to further leverage a more advanced LiDAR backbone. [54] transforms 3D lidar point clouds into the 2D image and fuses feature-level representation in the image branch leveraging mature CNN techniques to achieve better performance. [87] fused raw RGB pixel with the voxelized tensor while [79] directly combines pseudo-point clouds generated from the image branch and raw point clouds from the LiDAR branch together to accomplish object detection tasks.

Based on the VoxelNet[108], [69] proposes one of the fusion methods as point-fusion, which directly attached the image feature vector of the corresponding pixel to the voxelized vector. [92] proposes the dense-fusion that attaches every original point with the global features from the image branch. [53] focuses on 2D pedestrian detection using CNN. As early-fusion, it directly fuses different branches before inputting into CNN. [105] proposes one fusion method named point attention fusion which fused image feature to a voxelized tensor in LiDAR point clouds.

Deep-fusion. Deep-fusion methods fuse cross-modal data at the feature level for the LiDAR branch but data-level and feature-level for the image branch. For example, some methods use feature extractor to acquire the embedding representation of LiDAR point cloud and camera image respectively and fuse the feature in two modalities by a series of downstream modules [102, 32]. However, unlike other strong-fusion methods, deep-fusion sometimes fuses features in a cascading way [32, 4, 46], which both leverage raw and high-level semantic information. One example of deep-fusion can be the model in Figure 5.

[92] proposes the global-fusion attached global LiDAR feature with the global features from the image branch. [69] proposes the other fusion methods as voxel-fusion, which attached the ROI pooling image feature vector to a dense feature vector for each voxel in LiDAR point clouds. [105] proposes another method named dense attention fusion that fuses pseudo images from multiple branches. [49, 45], propose two deep-fusion method each. EPNet [32] a deep LIDAR-Image fusion estimating the importance of corresponding image features to reduce the noisy influence. [4] presents a multi-modal data set in extreme weather and fused each branch in a deep-fusion way, which greatly improves the robustness of the autonomous driving model. Other deep-fusion work includes [102, 46, 16, 9, 78, 85, 37, 14, 73, 22] which have the seemingly same fusion module.

Late-fusion. Late-fusion, also known as object-level fusion, denotes the methods that fuse the result of pipelines in each modality. For example, some late-fusion methods leverage output from both the LiDAR point cloud branch and camera image branch and make the final prediction based on the result in two modalities[55]. Notice that both branch proposals should have the same data format as the final results but vary in quality, quantity, and precision. Late-fusion can be regarded as a kind of ensemble method that utilizes multi-modal information to optimize the final proposal. One example of late-fusion can be the model in Figure 6.

As mentioned above, [55] leverages late-fusion for secondly refining every 3D region proposal’s score combining 2D proposal in image branch with the 3D proposal in LiDAR branch. Besides, for every overlapping region, it utilized statistical features like confidence score, distance and IoU. [1] focuses on 2D object detection, it combines the proposals from two branches along with features like confidence score, and the model outputs the final IoU score. [29],[28] solves the road detection by combing the segmentation results together. As late-fusion in [53], it summarizes the scores from different branches for the same 3D detection proposal into one final score.

Asymmetry-fusion. Besides the early-fusion, deep-fusion, and late-fusion, some methods treat cross-modal branches with different privileges, so we define methods that fuse object-level information from one branch while data-level or feature-level from other branches as asymmetry-fusion. Unlike other methods in strong-fusion, which treat both branches in seemingly equal status, asymmetry-fusion has at least one branch dominating while other branches provide auxiliary information to conduct the final task. One example of late-fusion can be the model in Figure 7. Especially compared to late-fusion, though they might have the same extract feature using proposal [55], asymmetry-fusion only have one proposal from one branch while late-fusion have proposals from all the branches.

Such fusing methods are reasonable due to the excellent performance using convolutional neural network on camera data which filters semantically useless points in the point cloud and promotes the performance on 3D LiDAR backbone in a frustum perspective, such as [106]. It extracts frustum in raw point clouds along with corresponding pixel’s RGB information to output the parameters of the 3D bounding box. However, some works think outside the box and use the LiDAR backbone to guide the 2D backbone in a multi-view style and achieve higher accuracy. [40] focuses on pedestrian detection by the extracted multi-view image based on 3D detection proposals, which further utilizes CNN to refine the previous proposal. [12] and [17] refines the 3D proposal predicted solely by the LiDAR branch with ROI feature in other branches. [5] focuses on 2D detection, utilizing 3D region proposal from LiDAR branch and re-projecting to 2D proposal along with the image features for further refinement. [11] proposes a 3D potential bounding box by statistical and rule-based information. Combining with the image feature, it outputs the final 3D proposal.[70] is focusing on small object detection accomplished by a specially collected data set, which is essentially a 2D semantic segmentation task combining the proposal from LiDAR with the raw RGB image to output the final results.

Unlike strong-fusion, weak-fusion methods do not fuse data/ feature/ object directly from branches in multi-modalities but operate data in other ways. The weak-fusion-based methods commonly use rule-based methods to utilize data in one modality as a supervision signal to guide the interaction of another modality. Figure 8 demonstrates the basic framework of the weak-fusion schema. For example, the 2D proposal from the CNN in the image branch might result in a frustum in the raw lidar point cloud. However, unlike combining image features as asymmetry-fusion mentioned above, weak fusion directly input those raw LiDAR point cloud selected into the LiDAR backbone to output the final proposal [60].

[83] advance the techniques by dividing each frustum into several parts by fixed chosen stride which, further increases the 3D detection accuracy. [104] focuses on remote sparse-point-cloud object detection. [99] filters out all the background points of LiDAR point cloud in frustum from semantic segmentation results in an image. [72] focuses on semi-supervised and transfer learning where frustum is proposed according to 2D image proposal.

Other weak-fusion like [19] highlights the real-time detection performance of 2D objects by selecting only one model of the two branches at each time to predict the final proposal using a reinforcement learning strategy. In [21], multiple 3D box proposals are generated by 2D detection proposals in the image branch, and then the model outputs the final 3D detection box with its detection score. [67] uses an image to predict 2D bounding box and 3D pose parameters and further refines it utilizing the LiDAR point clouds in the corresponding area.

Some work could not be simply defined as any kind of fusion mentioned above because they possess more than one fusion method in the whole model framework, such as a combination of deep-fusion and late-fusion [39] while [77] combines early-fusion and deep-fusion together. These methods suffer from redundancy at the model design view, which is not the mainstream for fusion modules.

Current fusion models suffer from the problem of misalignment and information loss [67, 98, 13]. Besides, the flat fusion operations [76, 20] also prevent the further improvement of perception task performance. We summarize them as two aspects: Misalignment and Information Loss, More Reasonable Fusion Operations.

Misalignment and Information Loss

The intrinsic and extrinsic of the camera and LiDAR are vastly different from each other. Data in both modalities need to be re-organized under a new coordinate system. Traditional early and deep fusion methods utilize an extrinsic calibration matrix to project all the LiDAR points directly to the corresponding pixels or vice versa [54, 76, 69]. However, this point-by-pixel alignment is not accurate enough because of the sensory noise. Therefore, we can see that in addition to such strict correspondence, some work [90] utilizing the surrounding information as a supplement results in better performance.

Besides, there exists some other information loss during the transformation of input and feature space. Generally, the projection of dimension reduction operation will inevitably lead to massive information loss, e.g., mapping the 3D LiDAR point cloud into the 2D BEV image. Therefore, by mapping the two modal data into another high-dimensional representation specially designed for fusion, future works can efficiently utilize the raw data with less information loss.

More Reasonable Fusion Operations

The current research works use intuitive methods to fuse cross-modal data, such as concatenation and element-wise multiply [69, 77]. These simple operations may not be able to fuse the data with a large distribution discrepancy, and as a result, hard to close the semantic gap between two modalities. Some work tries to use a more elaborated cascading structure to fuse the data and improve performance [12, 46]. In future research, mechanisms such as Bi-linear mapping[25, 38, 3] can fuse the feature with different characteristics.

Single frame at the front view is the typical scenario for autonomous driving perception tasks[26]. However, most frameworks utilize limited information without elaborated designed auxiliary tasks to further understand the driving scenes. We summarize them as With More Potential Useful Information and Self-Supervision for Representation Learning.

With More Potential Useful Information

Existing methods [81] lack the effective use of the information from multiple dimensions and sources. Most of them focus on a single frame of multi-modal data at the front view. As a result, other meaningful information is under-utilized such as semantic, spatial, and scene contextual information.

Some models [76, 90, 20] try to use the results obtained from the image semantic segmentation task as additional features, while others may exploit the features in intermediate layers of a neural network backbone whether trained by specific downstream tasks or not [46]. In autonomous driving scenarios, many downstream tasks with explicit semantic information may greatly benefit the performance of object detection tasks. For example, lane detection can intuitively provide additional help for detecting vehicles between the lanes, and the semantic segmentation results can improve object detection performance [76, 90, 20]. Therefore, future research can jointly build a complete semantic understanding framework of the cityscape scenarios through various downstream tasks like detecting the lane, traffic light, and signs to assist perception tasks’ performance.

In addition, current perception tasks mainly rely on a single frame that overlooks the temporal information. Recent LiDAR-based method [63] combines a series of frames to improve the performance. The time-series information contains serialized supervision signals, which can provide more robust results than methods that use a single frame.

Therefore, future work may dig deeper into utilizing temporal, contextual, and spatial information for continuous frames with innovative model designs.

Self-Supervision for Representation Learning

The mutual-supervised signals naturally exist among the cross-modal data sampled from the same real-world scenarios but different perspectives. However, current methods cannot mine the co-relationship among each modality, suffering from lacking a deep understanding of data. In the future, the studies can be focused on how to use the multi-modal data for self-supervised learning, including pre-training, fine-tuning or contrastive learning. By implementing these state-of-the-art mechanisms, the fusion model will result in a deeper understanding of the data and achieve better results, which has already shown some promising signs in other areas while leaving a blank space for autonomous driving perception [48].

Domain bias and resolution are highly related to the real-world scenes and the sensors [26]. These unexpected flaws prevent large-scale training and implementation for autonomous driving deep learning models, which need to be solved in future work.

Data Domain Bias

In autonomous driving perception scenes, the raw data extracted by different sensors is accompanied by severe domain-related characteristics. Different camera systems have their optical properties, while LiDAR may vary from mechanical LiDAR to solid-state LiDAR. What is more, data itself may be domain-biased such as weather, season, or location [71, 6], even if it is captured by the same sensors. As a result, the detection model can not be adapted to new scenarios smoothly. Such defects prevent the collection of a large-scale dataset and the re-usability of the original training data due to the failures of generalization. Therefore, it is crucial to find a way to eliminate domain bias and integrate different data sources adaptively in future work.

Conflicts with Data Resolution

Sensors from different modalities often have different resolutions [42, 100]. For example, the spatial density of LiDAR is notably lower than that of the image. No matter what projection method is adopted, some information is eliminated because the corresponding relationship cannot be found. This may lead the model to be dominated by the data of one specific modality, whether it is due to the different resolution of the feature vector or the imbalance of original information. Therefore, future works can explore a new data representation system compatible with sensors in different spatial resolutions.