Deep Learning for LiDAR Point Clouds
in Autonomous Driving: A Review

In the perception module of autonomous vehicles, semantic segmentation, object detection, object localization, and classification/recognition constitute the foundation for reliable navigation and accurate decision [38]. These tasks are described as follows respectively:

• •

  3D point cloud semantic segmentation: Point cloud segmentation is the process to cluster the input data into several homogeneous regions, where points in the same region have the identical attributes [39]. Each input point is predicted with a semantic label, such as ground, tree, building. The task can be concluded as: given a set of ordered 3D points $X=\left\{x_{1},x_{2},x_{i},\cdots,x_{n}\right\}$ with ${x_{i}}\in{R^{3}}$ and a candidate label set $Y=\left\{y_{1},y_{2},\cdots,{y_{k}}\right\}$, assign each input point $x_{i}$ with one of the k semantic labels [40]. Segmentation results can further support object detection and classification, as shown in Fig.2(a).

• •

  3D object detection/localization: Given an arbitrary point cloud data, the goal of 3D object detection is to detect and locate the instances of predefined categories (e.g., cars, pedestrians, and cyclists, as shown in Fig.2(b)), and return their geometric 3D location, orientation and semantic instance label [41]. Such information can be represented coarsely using a 3D bounding box which is tightly bounding the detected object [42, 42, 13]. This box is commonly represented as $(x,y,z,h,w,l,\theta,c)$, where $(x,y,z)$ denotes the object (bounding box) center position, $(h,w,l)$ represents the bounding box size with width, length and height, and $\theta$ is the object orientation. The orientation refers to the rigid transformation that aligns the detected object to its instance in the scene, which are the translations in each of the of x, y, and z directions as well as a rotation about each of these three axes [43, 44]. $c$ represents the semantic label of this bounding box (object).

• •

  3D object classification/recognition: Given several groups of point clouds, the objectiveness of classification /recognition is to determine the category (e.g., mug, table, or car, as shown in Fig.2(c)) the group points belong to. The problem of 3D object classification can be defined as: given a set of 3D ordered points $X=\left\{x_{1},x_{2},x_{i},\cdots,x_{n}\right\}$ with $x_{i}\in{R^{3}}$ and a candidate label set $Y=\left\{y_{1},y_{2},\cdots,{y_{k}}\right\}$, assign the whole point set $X$ with one of the $k$ labels [45].

In order to segment, detect, and classify the general objects using DL for AVs with robust and discriminative performance, several challenges and problems that must be addressed, as shown in Fig.2. The variation of sensing conditions and unconstrained environments results in the challenges on data. The irregular data format and requirements for both accuracy and efficiency pose the problems that DL models need to solve.

Datasets pave the way towards the rapid development of 3D data application and exploitation using DL networks. There are two roles of reliable datasets: one for providing a comparison for competing algorithms, another for pushing the fields towards more complex and challenging tasks [23]. With the increasing application of LiDAR in multiple fields, such as autonomous driving, remote sensing, photogrammetry, there is a rise of large scale datasets with more than millions of points. These datasets accelerate the crucial breakthroughs and unpredicted performance in point cloud segmentation, 3D object detection, and classification. Apart from the mobile LiDAR data, some discriminative datasets [56] acquired by terrestrial laser scanning (TLS) by static LiDAR are also employed due to they provide high-quality point cloud data.

As shown in Table I, we classify those existing datasets related to our topic into three types: segmentation-based datasets, detection-based datasets, classification-based datasets. Besides, long-term autonomy dataset is also summarized.

• •

  Segmentation-based datasets

Semantic3D [56]. Semantic3D is the existing largest LiDAR dataset for outdoor scene segmentation tasks with more than 4 billion points and around 110,000$m^{2}$ covering area. This dataset is labeled with 8 classes and split into training and test sets with nearly equal size. These data are acquired by a static LiDAR with high measurement resolution and covered long measurement distance. The challenges for this dataset mainly stems from the massive point clouds, unevenly distributed point density, and severe occlusions. In order to fit the high computation algorithms, a reduced-8 dataset is introduced for training and testing, which share the same training data but fewer test data compared with Semantic3D.

Oakland 3-D Point Cloud Dataset [57]. This dataset is acquired in an early year compared with the above two datasets. A mobile platform equipped with LiDAR is used to scan the urban environment and generated around 1.3 million points, while 100,000 points are split into a validation set. The whole dataset is labeled with 5 classes such as wire, vegetation, ground, pole/tree-trunk, and facade. This dataset is small and thus suitable for lightweight networks. Besides, this dataset can be used to test and tune the network architectures without a lot of training time before final training on other datasets.

IQmulus & TerraMobilita Contest [58]. This dataset is also acquired by a mobile LiDAR system in the urban environment in Paris. There are more than 300 million points in this dataset, which covered 10km street. The data is split into 10 separate zones and labeled with more than 20 fine classes. However, this dataset also has severe occlusion.

Paris-Lille-3D [59]. Compared with Semantic3D [56], Paris-Lille-3D contains fewer points (140 million points) and covering area (55,000$m^{2}$). The main difference of this dataset is that its data are acquired by a Mobile LiDAR system in two cities: Paris and Lille. Thus, the points in this dataset are sparse and comparatively low measurement resolution compared with Semantic3D [56]. But this dataset is more similar to the LiDAR data acquired by AVs. The whole dataset is fully annotated into 50 classes unequally distributed in three scenes:Lille1, Lille2, and Paris. For simplicity, these 50 classes are combined into 10 coarse classes for challenging.

• •

  Detection-based datasets

KITTI Object Detection/Bird’s Eye View Benchmark [60]. Different from the above LiDAR datasets which are specific for segmentation task, KITTI dataset is acquired from an autonomous driving platform and records six hours driving using digital cameras, LiDAR, GPS/IMU inertial navigation system. Thus, apart from the LiDAR data, the corresponding imagery data are also provided. Both the Object Detection and Bird’s Eye View Benchmark contains 7481 training images and 7518 test images as well as the corresponding point clouds. Due to the moving scanning mode, the LiDAR data in this benchmark is highly sparse. Thus, only three objects are labeled with bounding box: cars, pedestrians, and cyclists.

• •

  Classification-based datasets

Sydney Urban Objects Dataset [61]. This dataset contains a set of general urban road objects scanned with a LiDAR in the CBD of Sydney, Australia. There are 588 labeled objects and classified in 14 categories, such as vehicles, pedestrians, signs, and trees. The whole dataset is split into four folds for training and testing. Similar to other LiDAR datasets, the collected objects in this dataset are sparse with incomplete shape. Although it is small and not ideal for the classification task, it the most commonly used benchmark due to the limitation of the tedious labeling process.

ModelNet [30]. This dataset is the existing largest 3D benchmark for 3D object recognition. Different from Sydney Urban Objects Dataset [61], which contains road objects collected by LiDAR sensors, this dataset is composed of general objects in CAD models with evenly distributed point density and complete shape. There are approximately 130K labeled models in a total of 660 categories (e.g., car, chair, clock). The most commonly used benchmarks are ModelNet40 that contains 40 general objects and ModelNet10 with 10 general objects. The milestone 3D deep architectures are commonly trained and tested on these two datasets due to the affordable computation burden and time.

Long-Term Autonomy: To address challenges of long-term autonomy, a novel dataset for autonomous driving has been presented by Maddern et al. [64]. They collected images, LiDAR, and GPS data while traversing 1,000 km in central Oxford in the UK for one year. This allowed them to capture different scene appearances under various illumination, weather, and season with dynamic objects and constructions. Such long-term datasets allow for in-depth investigation of problems that detain the realization of autonomous vehicles such as localization at different times of the year.

To evaluate those proposed methods performance, several metrics, as summarized in Table II, are proposed for those tasks: segmentation, detection, and classification. The detail of these metrics is given as follows.

For the segmentation task, the most commonly used evaluation metrics are the Intersection over Union (IoU) metric, $\overline{IoU}$, and overall accuracy (OA) [62]. IoU defines the quantify the percent overlap between the target mask and the prediction output [56].

For detection and classification tasks, the results are commonly analyzed region-wise. Precision, recall, $F_{1}$-score and Matthews correlation coefficient (MCC) [65] are commonly used to evaluate the performance. The precision represents the ratio of correctly detected objects in the whole detection results, while the recall means the percentage of the correctly detected objects in the ground truth, the $F_{1}$-score conveys the balance between the precision and the recall, the MCC is the combined ratio of detected and undetected objects and non-objects.

For 3D object localization and detection task, the most frequently used metrics are: Average Precision ($AP_{3D}$) [66], and Average Orientation Similarity (AOS) [36]. The average precision is used to evaluate the localization and detection performance by calculating the averaged valid bounding box overlaps, which exceed predefined values. For orientation estimation, the orientation similarities with different threshold-ed valid bounding box overlaps are averaged to report the performance.

Conventionally, CNNs are mainly applied to data with regular structures, such as the 2D pixel array [79]. Thus, in order to apply CNNs to unordered 3D point cloud data, such data are divided into regular grids with a certain size to describe the distribution of data in 3D space. Typically, the size of the grid is related to the resolution of data [80]. The advantage of voxel-based representation is that it can encode the 3D shape and viewpoint information by classifying the occupied voxels into several types such as visible, occluded, or self-occluded. Besides, 3D convolution (Conv) and pooling operations can be directly applied in voxel grids [69].

3D ShapeNet [30], proposed by Wu et al. and shown in Fig.3, is the pioneer in exploiting 3D volumetric data using a convolutional deep belief network. The probability distribution of binary variables is used to represent the geometric shape of a 3D voxel grid. Then these distributions are input to the network which is mainly composed of three Conv layers. This network is initially pre-trained in a layer-wise fashion and then trained with a generative fine-tuning procedure. The input and Conv layers are modeled based on the Contrastive Divergence, where the output layer was trained based on the Fast-Persistent Contrastive Divergence. After training, the input test data is output with a single depth map and then transformed to represent the voxel grid. ShapeNet has notable results in low-resolution voxels. However, the computation cost increases cubically with the increment of input data size or resolution, which limit the model’s performance in large-scale or dense point clouds data. Besides, multi-scale and multi-view information from the data is not fully exploited, which hinder the output performance.

VoxNet [67] is proposed by Maturana et al. to conduct 3D object recognition using 3D convolution filters based on volumetric data representation, as shown in Fig.3. Occupancy grids represented by a 3D lattice of random variables are employed to show the state of the environment. Then a probabilistic estimate is used to estimate the occupancy of these grids which is maintained as the prior knowledge. Three different occupancy grid models, such as binary occupancy grid, density grid, and hit grid are experimented to select the best model. This network framework is mainly composed of Conv, pooling layer, and fully connected (FC) layers. Both ShapeNet [30] and VoxNet employ rotation augmentation for training. Compared with ShapeNet [30], VoxNet has a smaller architecture that has less than 1 million parameters. However, not all occupancy grids contain useful information but only increase the computation cost.

3D-GAN [68] combines the merits of both general-adversarial network (GAN) [81] and volumetric convolutional networks [67] to learn the features of 3D objects. This network is composed of a generator and a discriminator as shown in Fig.3. The adversarial discriminator is conducted to classify objects into synthesized and real categories due to the generative-adversarial criterion has the advantage in capturing the structural variation between two 3D objects. And the employment of generative-adversarial loss is helpful to avoid possible criterion-dependent over-fitting. The generator attempts to confuse the discriminator. Both generator and discriminator consist of five volumetric fully Conv layers. This network provides a powerful 3D shape descriptor with unsupervised training in 3D object recognition. But the density of data affects the performance of adversarial discriminator for finest feature capturing. Consequently, this adaptive method is suitable for evenly distributed point cloud data.

In conclusion, there are some limitations of this general volumetric 3D data representation:

• •

  Firstly, not all voxel representations are useful because they contain occupied and non-occupied parts of the scanning environment. Thus, the high demand for computer storage is actually unnecessary within this ineffective data representation [69].

• •

  Secondly, the size of the grid is hard to set, which affects the scale of input data and may disrupt the spatial relationship between points.

• •

  Thirdly, computational and memory requirements grow cubically with the resolution [69]. Thus, existing voxel-based models are maintained at low 3D resolutions, and the most commonly used size is $30^{3}$ for each grid.[69].

A more advanced voxel-based data representation is the octree-based grids [69, 82], which use adaptive size to divides the 3D point cloud into cubes. It is a hierarchical data structure that recursively decomposes the root voxels into multiple leaf voxels.

OctNet [69] is proposed by Riegler et al., which exploits the sparsity of the input data. Motivated by the observation that the object boundaries have the highest probability in producing the maximum responses across all feature maps generated by the network at different layers, they partitioned the 3D space hierarchically into a set of unbalanced octrees [83] based on the density of the input data. Specifically, the octree nodes that have point clouds are split recursively in its domain, ending at the finest resolution of the tree. Thus, the size of leaf nodes varies. For each leaf node, those features that activate their comprised voxel is pooled and stored. Then the convolution filters are conducted in these trees. In [82], the deep model is constructed by learning the structure of the octree and the represented occupancy value for each grid. This octree-based data representation largely reduces the computation and memory resources for DL architectures, which achieves better performance in high-resolution 3D data compared with voxel-based models. However, the disadvantage of octree data is similar to voxels, both of them fail to exploit the geometry feature of 3D objects, especially the intrinsic characteristics of patterns and surfaces [29].

Different from volumetric 3D data representation, point cloud data can preserve the 3D geospatial information and internal local structure. Besides, the voxel-based models that scan the space with fixed strides are constrained by the local receptive fields. But for point clouds, the input data and the metric decide the range of receptive fields, which has high efficiency and accuracy.

PointNet [10], as a pioneer in consuming 3D point clouds directly for deep models, learns the spatial feature of each point independently via MLP layers and then accumulates their features by max-pooling. The point cloud data are input directly to the PointNet, which predicts per-point label or per-object label, its framework is illustrated in Fig.4. In PointNet, spatial transform network and a symmetric function are designed to improve the invariance to permutation. The spatial feature of each input point was learned through the networks. Then, the learned features are assembled across the whole region of point clouds. The outstanding performance of PointNet has achieved in 3D objects classification and segmentation tasks. However, the individual point features are grouped and pooled by max-pooling, which fails to preserve the local structure. As a result, PointNet is not robust to fine-grained patterns and complex scenes.

PointNet++ was proposed later by Qi et al. [12], which compensate the local feature extraction problems in PointNet. Within the raw unordered point clouds as input, these points are initially divided into overlapping local regions using the Euclidean distance metric. These partitions are defined as a neighborhood ball in this metric space and labeled with the centroid location and scale. In order to sample the points evenly over the whole point set, the farthest point sampling (FPS) algorithm is applied. Local features are extracted from the small neighborhoods around the selected points using K-nearest-neighbor (KNN) or query-ball searching methods. These neighborhoods are gathered into larger clusters and leveraged to extract high-level features via PointNet [10] network. The sampling and grouping module are repeated until the local and global features of the whole points are learned, as shown in Fig.4. This network, which outperforms the PointNet [10] network in classification and segmentation tasks, extracts the local feature for points in different scales. However, features from the local neighborhood points in different sampling layers are learned in an isolated fashion. Besides, max-pooling operation based on PointNet [10] for high-level feature extraction in PointNet++ fails to preserve the spatial information between the local neighborhood points.

Kd-networks [70] uses the kd-tree to create the order of the input points, which is different from PointNet [10] and PointNet++ [12] as both of them use the symmetric function to solve the permutation problem. Klokov et al. used the maximum range of point coordinates along the coordinate axis to recursively split the certain size point clouds $N=2^{D}$ into subsets with a top-down fashion to construct a kd-tree. As shown in Fig.5, this kd-tree is ending with a fixed depth. Within this balanced tree structure, vectorial representations in each node, which represents a subdivision along certain axis, is computed using kd-networks. These representations are then exploited to train a linear classifier. This network has better performance than PointNet [10] and PointNet++ [12] in small objects classification. However, it is not robust to rotations and noise, since these variations can lead to the change of tree structure. Besides, it lacks the overlapped receptive field which reduces the spatial-correlation between leaf nodes.

PointCNN, proposed by Li et al. [71], solves the input points permutation and transformation problems based on an $\chi$-Conv operation, as shown in Fig.5. They proposed the $\chi$-transformation which is learned from the input points by weighting the input point features and permutating the points into a latent and potentially canonical order. Then the traditional convolution operators are applied in the learned $\chi$-transformation features. These spatially-local correlation features in each local range are aggregated to construct a hierarchical CNN network architecture. However, this model still has not exploited the correlations of different geometric features and their discriminate information toward results, which limits the performance.

Point cloud based deep models are mostly focused on solving permutation problems. Although they treat points independently at local scales to maintain permutation invariance. This independence, however, neglects the geometric relationships among points and their neighbors, presenting a fundamental limitation that leads to local features’ missing.

Graphs are a type of non-Euclidean data structure that can be used to represent point cloud data. Their node corresponds to each input point and the edges represent the relationship between each point neighbors. Graph neural networks propagate the node states until equilibrium in an iterative manner [75]. With the advancement of CNNs, there is an increment graph convolutional networks applied to 3D data. Those graph CNNs define convolutions directly on the graph in the spectral and non-spectral (spatial) domain, operating on groups of spatially close neighbors [84]. The advantage of graph-based models is that the geometric relationships among points and their neighbors are exploited. Thus, more spatially-local correlation features are extracted from the grouped edge relationships on each node. But there are two challenges for constructing graph-based deep models:

• •

  Firstly, defining an operator that is suitable for dynamically sized neighborhoods and maintaining the weight sharing scheme of CNNs [75].

• •

  Secondly, exploiting the spatial and geometric relationships among each node’s neighbors.

SyncSpecCNN [72] exploited the spectral eigen-decomposition of the graph Laplacian to generate a convolution filter applied in point clouds. Yi et al. constructed SyncSpecCNN based on that two considerations: the first is the coefficients sharing and multi-scale graph analyzing; the second is information sharing across related but different graphs. They solved these two problems by constructing the convolution operation in the spectral domain: the signal of point sets in the Euclidean domain is defined by the metrics on the graph nodes, and the convolution operation in the Euclidean domain is related to the scaling signals based on eigenvalues. Actually, such operation is linear and only applicable to the graph weights generated from eigenvectors of the graph Laplacian. Despite SyncSpecCNN achieved excellent performance in 3D shape part segmentation, it has several limitations:

• •

  Basis-dependent. The learned spectral filter’s coefficients are not suitable for another domain with a different basis.

• •

  Computationally expensive. The spectral filtering is calculated based on the whole input data, which requires high computation capability.

• •

  Missing local edge features. The local graph neighborhood contains useful and distinctive local structural information, which is not exploited.

Edge-conditioned convolution (ECC) [73] considers the edge information in constructing the convolution filters based on the graph signal in the spatial domain. The edge labels in a vertex neighborhood are conditioned to generate the Conv filter weights. Besides, in order to solve the basis-dependent problem, they dynamic generalized the convolution operator for arbitrary graphs with varying size and connectivity. The whole network follows the common structure of feedforward network with interlaced convolutions and pooling followed by global pooling and FC layers. Thus, features from local neighborhoods are extracted continually from these stacked layers, which increase the receptive field. Although the edge labels are fixed for a specific graph, the learned interpretation networks may vary in different layers. ECC learns the dynamic pattern of local neighborhoods, which is scalable and effective. However, the computation cost remains high, and it is not applicable for large-scale graphs with continuous edge labels.

DGCNN [74] also constructed a local neighborhood graph to extract the local geometric features and applied Conv-like operations, named EdgeConv which is shown in Fig.6, on the edges connecting neighboring pairs of each point. Different from ECC [73], EdgeConv dynamically updates the given fixed graph with Conv-like operations for each layer output. Thus, DGCNN can learn how to extract local geometric structures and group point clouds. This model takes $n$ points as input, and then find the K neighborhoods of each point to calculate the edge feature between the point and its K neighborhoods in each EdgeConv layer. Similar to PointNet[34] architecture, the features convolved in the last EdgeConv layer are aggregated globally to construct a global feature, while all the EdgeConv outputs are treated as local features. Local and global features are concatenated to generate results’ score. This model extracts distinctive edge features from point neighborhoods, which can be applied in different point clouds related tasks. However, the fixed size of edge features limits the performance of the model when facing different scales and resolution point clouds.

ECC [73] and DGCNN [74] propose general convolutions on graph nodes and their edge information, which is isotropy about input features. However, not all the input features contribute equally to its nodes. Thus, attention mechanisms are introduced to deal with variable sized inputs and focus on the most relevant parts of the nodes’ neighbors to make decisions [75].

Graph Attention Networks (GAT) [75]. The core insight behind GAT is to calculate the hidden representations of each node in the graph, by assigning different attentional weights to different neighbors, following a self-attention strategy. Within a set of node features as input, a shared linear transformation, parametrized by a weight matrix is applied to each node. Then a self-attention, a shared attentional mechanism which is shown in Fig.6, is applied on the nodes to computes attention coefficients. These coefficients indicate the importance of corresponding nodes’ neighbor features, respectively, and are further normalized to make them comparable across different nodes. These local features are combined according to the attentional weights to form the output features for each node. In order to improve the stability of the self-attention mechanism, multi-head attention is employed to conduct k independent attention schemes, which are then concatenated together to form the final output features for each node. This attention architecture is efficient and can extract fine-grained representations for each graph node by assigning different weights to the neighbors. However, local spatial relationship between neighbors are not considered in calculating the attentional weights. To further improve its performance, Wang et al. [85] proposed graph attention convolution (GAC) to generate attentional weights by considering different neighboring points and feature channels.

The last type of MLS data representation is 2D views obtained from 3D point clouds from different directions. With the projected 2D views, traditional well-established convolutional neural networks (CNN) and pre-trained networks on image datasets, such as AlexNet [86], VGG [87], GoogLeNet [88], ResNet [89] can be exploited. Compared with voxel-based models, these methods can improve the performance for different 3D tasks by taking multi-view of the interest object or scenes and then fusing or voting the outputs for final prediction. Compared with the above three different 3D data representations, view-based models can achieve near-optimal results, as shown in Table III. Su et al. [90] experimented that multiview methods have the optimal generalization ability even without using pre-trained models compared with point cloud and voxel data representation models. The advantages of view-based models compared with 3D models can be concluded as:

• •

  Efficiency. Compared with 3D data representations such as point clouds or voxel grids, the reduced one dimension information can greatly reduce the computation cost but with increased resolution [76].

• •

  Exploiting established 2D deep architectures and datasets. The well-developed 2D DL architectures can better exploit the local and global information from projected 2D view images [91]. Besides, existing 2D image databases (such as ImageNet [92]) can be used to train 2D DL architectures.

Multi-View CNN (MVCNN) [76] is the pioneer in exploiting 2D DL models to learn 3D representation. Multiple views of 3D objects are extracted without specific order using a view pooling layer. Two different CNNs models are proposed and tested in this paper. The first CNN model takes 12 views rendered from the object via placing 12 virtual cameras with equal distance around the objects as the input, while the second CNN model takes 80 views rendered in the same way as input. These views are first learned separately and then fused through max-pooling operation the extract the most representative feature among all views for the whole 3D shape. This network is effective and efficient compared with volumetric data representation. However, the max-pooling operation only considers the most important views and discards information from other views, which fails to preserve comprehensive visual information.

MVCNN-MultiRes was proposed by Qi et al [15] to improve multi-view CNNs. Different from traditional view rendering methods, the 3D shape is projected to 2D via a convolution operation based on an anisotropic probing kernel applied to the 3D volume. Multi-orientation pooling is combined together to improve the 3D structure capturing capability. Then the MVCNN [76] is applied to classify the 2D projects. Compared with MVCNN [76], multi-resolution 3D filtering is introduced to capture multi-scale information. Sphere rendering is performed at different volume resolutions to achieve view-invariant and improve the robust to potential noise and irregularities. This model achieves better results in 3D object classification task compared with MVCNN [76].

3DMV [77] combines the geometry and imagery data as input to train a joint 3D deep architecture. Feature maps extracted from imagery data are first extracted and then mapped into the 3D feature extracted from the volumetric grid data derived from a differentiable back-projection layer. Because there exists redundant information among multiple views, a multiview pooling approach is applied to extract useful information from these views. This network achieved remarkable results in 3D objects classification. However, compared with models using one source of data such as LiDAR point or RGB images solely, the computation cost of this method is higher.

RotationNet [78] is proposed following the assumption that when the object is observed by a viewer from a partial set of full multiview images, the observation direction should be recognized to correctly infer the object’s category. Thus, the multiview images of an object are input to the RotationNet, which outputs its pose and category. The most representative characteristic of RotationNet is that it treats viewpoints which are the observation of training images as latent variables. Then unsupervised learning of object poses is conducted based on an unaligned object dataset, which can eliminate the process of pose normalization to reduce noise and individual variations in shape. The whole network is constructed as a differentiable MLP network with softmax layers as the final layer. The outputs are the viewpoint category probabilities, which correspond to the predefined discrete viewpoints for each input image. These likelihoods are optimized by the selected object pose.

However, there some limitation of 2D view-based models:

• •

  The first is that the projection from 3D space to 2D views can lose some geometrically-related spatial information.

• •

  The second is the redundant information among multiple views.

Due to the massive amount of data and the tedious labeling process, there exist limited reliable 3D datasets. To better exploit the architecture of deep networks and improve the model generalization ability, data augmentation is commonly conducted. Augmentation can be applied to both data space and feature space, while the most common augmentation is conducted in the first space. This type of augmentation can not only enrich the variations of data but also can generate new samples by conducting transformations to the existing 3D data. There are several types of transformations, such as translation, rotation, and scaling. Several requirements for data augmentation are summarised as:

• •

  There must exist similar features between original augmented data, such as shape;

• •

  There must exist different features between original and augmented data such as orientation.

Based on those existing methods, classical data augmentation for point clouds can be concluded as:

• •

  Mirror $x$ and $y$ axis with predefined probability [59, 93]

• •

  Rotation around z-axis with certain times and angles[59, 94, 93, 13]

• •

  Random (uniform) height or position jittering in certain range [95, 93, 67]

• •

  Random scale with certain ratio [59, 13]

• •

  Random occlusions or randomly down-sampling points within predefined ratio [59]

• •

  Random artefacts or randomly down-sampling points within predefined ratio [59]

• •

  Randomly adding noise, following certain distribution, to the points’ coordinates and local features [59, 96, 45].

The goal of semantic segmentation is to label each point as belonging to a specific semantic class. For AVs segmentation tasks, these classes cloud be a street, buildings, cars, pedestrians, trees or traffic lights. When applying DL for point cloud segmentation, classification of small features is required [38]. However, the LiDAR 3D point clouds are usually acquired in large scale, and they are irregularly shaped with changeable spatial contents. In the review of the recent five years papers related in this region, we group these papers into three schemes according to the types of data representation: point cloud based, voxel-based, and multi-view based models. There is limited research focusing on graph-based models, thus we combine the graph-based and point cloud based models together to illustrate their paradigms. Each type of model is represented by a compelling deep architecture as shown in Fig.7.

The detection(& localization) of 3D objects in LiDAR point clouds can be summarised as bounding box prediction and objectness prediction [14]. In this paper, we mainly survey the LiDAR-only paradigm, which takes advantage from accurate geo-referenced information. Overall, there are two ways for data representation in this paradigm: one detects and locates 3D objects directly from point clouds [118]; another first converts 3D points into regular grids, such as voxel grids or bird’s eye view images as well as front views, and then utilizes architectures in 2D detectors to extract object from images, the 2D detection results are finally back-projected into 3D space for final 3D object location estimation [50]. Fig.8 shows the representative network frameworks of the above-listed data representations.

Semantic object classification/recognition is crucial for safe and reliable driving of AVs in unstructured and uncontrolled real-world environments [67]. Existing 3D object detection are mainly focus on CAD data (e.g., ModelNet40 [30]) or RGBD data (e.g., NYUv2 [128]). However, these data have uniform point distribution, complete shapes, limited noise, occlusion and background clutter, which poses limit challenges for 3D classification compared with LiDAR point clouds [10, 12, 129]. Those compelling deep architectures applied on CAD data have been analyzed in the form of four types of data representations in section III. In this part, we mainly focus on the LiDAR data based deep models for the classification task.