Airborne LiDAR Point Cloud Classification with Graph Attention Convolution Neural Network

Traditionally, studies accomplish airborne LiDAR point cloud classification by calculating handcrafted features and employing the classic machine learning models. Zhang et al., adopt the surface growing algorithm to cluster point clouds and utilize SVM to classify segments according to thirteen features related to radiometry, geometry, topology and echo characteristics (Zhang et al.,, 2013). Lodha et al., select five features, namely, image intensity, LiDAR return intensity, normal variation, height, and height variation, and implement AdaBoost to classify 3D airborne LiDAR data into four groups: buildings, trees, grass, and road (Lodha et al.,, 2007). Chehata et al., apply Random Forests to select the most important features from multi-echo and full-waveform features, and classify airborne LiDAR point clouds on urban scenes (Chehata et al.,, 2009). However, these models ignore the contextual information of point clouds and predict the semantic label for each point individually, which creates noises in the classification results and inconsistency in the labels.

To resolve this issue, Niemeyer et al., implement Conditional Random Field (CRF) method, which can be employed to incorporate contextual information and learn relations among objects, to classify airborne LiDAR point clouds based on geometrical and intensity features (Niemeyer et al.,, 2011). Besides, Niemeyer et al., exploit ten types of features obtained from LiDAR point clouds and integrate Random Forest classifier into CRF to achieve reliable classification results, even in complicated urban scenes (Niemeyer et al.,, 2014).

Although the above non-deep learning methods achieve satisfying classification performance, they require manual calculation of features in advance, and the results are sensitive to the choice of features.

Deep learning, a novel machine learning method proposed in recent years, can automatically learn effective feature representations from a large amount of input data. CNNs, one of the most essential deep learning models, have made considerable progress in image classification tasks, such as object detection, semantic segmentation, and edge detection. However, because CNNs can handle only standard regular input format, direct application of CNNs to irregular and unordered 3D point clouds is infeasible.

To apply CNNs to airborne LiDAR point cloud classification, most researchers first convert the point cloud to regular 3D voxel grids or collections of images. Maturana and Scherer, develop the Voxnet model to transform point clouds to 3D voxel grids, followed by a 3D CNN that predicts the semantic label based on the occupancy grid (Maturana and Scherer,, 2015). However, 3D volumetric grids entail substantial memory consumption and computational cost.

Therefore, some researchers have aimed to transform point clouds into collections of feature images, and subsequently exert CNNs to extract high-level representations of features and perform point cloud classification. Yang et al., generate 2D feature images by extracting the local geometric features, global geometric features and full-waveform features of each point;the features are then input into 2D CNNs for point cloud classification (Yang et al.,, 2017). Moreover, Yang et al., change the previous image generation approach to a method that can implement the generation at different scales and design a multiscale CNN for final semantic classification based on five features, namely, intensity, eigenvalue features, planarity, sphericity, and variance of deviation angles (Yang et al.,, 2018). Zhao et al., propose a multiscale CNN that can automatically learn deep features of each LiDAR point by generating a set of contextual images from selected features of the LiDAR data, such as height, intensity and roughness (Zhao et al.,, 2018). However, these models cause spatial information loss and induce quantization error in the conversion process (Te et al.,, 2018).

As a pioneer work of directly applying deep learning models to raw point clouds, PointNet model (Qi et al., 2017a, ) employ MLP to learn the features of individual points and a symmetric function (e.g. max pooling) to encode global information. Although PointNet provide a unified and efficient approach to 3D recognition tasks, it cannot capture the local structure of point cloud. To solve this problem, PointNet++ (Qi et al., 2017b, ) is developed by constructing a hierarchical neural network that applies PointNet recursively on partitioning of point cloud generated via sampling layers and grouping layers. Following these two models, many researchers proposed various deep learning models for point cloud classification based on PointNet-like architectures (Li et al.,, 2018; Jiang et al.,, 2018; Thomas et al.,, 2019).

Considering the inherent topological information of point clouds, researchers propose graph convolution neural network (GCNN) for point cloud classification applied on unordered 3D point clouds (Te et al.,, 2018; Wang et al., 2018a, ). Similarly, Wang et al., 2019b design a dynamic graph convolution operations to capture local geometric structures by generating edge features between a point and its neighborhoods (Wang et al., 2019b, ). More recently, some studies try to introduce the attention mechanism to learn a more adaptive local summary of the neighborhood. For instance, Chen et al., propose the GAPNet model, which employs a multi-head mechanism to aggregate attention features for each point from its neighborhoods and applies stacked MLP layers to capture local geometric features from raw point cloud (Chen et al.,, 2019). Wang et al., 2019a introduce a graph attention convolution (GAC) to selectively focus on the most correlated part of the neighbors and train a graph attention convolution network for point cloud classification (Wang et al., 2019a, ). Note that these graph attention convolution neural networks just pay attention to local geometric structures, but our model also takes density distribution and global contextual relationships into consideration, which is the main difference of our graph attention convolution network from the previous ones in the computer vision field.

Specifically, in the field of airborne LiDAR point cloud classification, few studies apply deep learning models directly to raw point clouds. Yousefhussien et al., present a 1D-fully convolutional classification network that directly consumes 3D coordinates and three corresponding spectral features extracted from 2D georeferenced images for each point (Yousefhussien et al.,, 2017). Furthermore, Wang et al., 2018b design a deep neural network with spatial pooling (DNNSP) by adopting a max pooling layer to aggregate the point-based features into the cluster-based features, which are then input to another MLP for point cloud classification (Wang et al., 2018b, ). In addition, our previous work explores a directionally constrained point convolution (D-Conv) module to extract local features of 3D point sets from the projected 2D receptive fields and then introduces a multiscale fully convolutional neural network based on the above module (Wen et al.,, 2020).

However, these point cloud deep learning methods focus on extracting local features and ignore the global relationships between individual points. In addition, the above models employ standard convolution kernels with a regular receptive fields, which neglect the structural connections between points and fails to account for varying point density.

In this paper, we propose a GACNN for airborne LiDAR point cloud classification that can directly handle unstructured 3D point clouds by considering local structural features and global contextual information simultaneously. Specifically, our proposed graph attention convolution module dynamically learns convolution weights and adaptively adjusts the convolution kernel according to the local structural connection of the point cloud. Moreover, the unbalanced density distribution of the point cloud is taken into account by our module.

To further explore the local structure of point clouds, two attention modules are added to standard graph convolution in this study, i.e. edge attention and density attention, so the proposed graph convolution module can dynamically learn convolutional kernel shapes to adjust to the structure of point sets and take the varying density distribution of non-uniform sampled point clouds into account.

Previous works focus on extracting local features and obtaining global information by employing a symmetric function but ignore the spatial relationships among all points. To remedy this problem, we propose a graph global attention module to learn global contextual information of the point cloud. The Euclidean distance between every two individual points in each coordinate direction is calculated to obtain the distance matrix with dimension $N*N*3$, which can be represented as:

Similarly, we use a softmax layer to normalize the distance for each point, which can be expressed as:

where $\mathbb{D}_{ij}$ and $d_{ij}$ signify the distance and normalized distance between points $p_{i}$ and $p_{j}$, respectively. Another MLP layer is employed as nonlinear transform function to obtain the final attention weight from the normalized distance, which is given by:

In the last two sections, we introduce the local and global attention modules in detail. In this section, we first present how the two attention mechanisms are fused into the proposed graph attention convolution module. Then, we develop an end-to-end encoder-decoder network to learn multiscale features from raw point clouds.

We assess the performance of our model on the International Society for Photogrammetry and Remote Sensing (ISPRS) 3D labeling dataset, which is composed of airborne laser scanning data acquired with a Leica ALS50 system from Vaihingen, Germany (Cramer,, 2010). The LiDAR data are categorized into nine semantic classes (Niemeyer et al.,, 2014), that is power line, low vegetation (low_veg), impervious surface (imp_surf), fence/hedge, car, roof, facade, shrub, and tree. Scenes from three different areas are provided on the ISPRS 3D labeling website: one scene with 753,876 points is employed as the training set and the other two scenes with 411,722 points are used as the test set. Table 1 describes the number of points of each category for each set. The spatial XYZ coordinates, intensity, the return of number, the number of returns and semantic labels are provided for each set. Figure 4 shows the three scenes in the dataset, and the rendering color of each category is referenced to (Blomley et al.,, 2016).

According to the standard evaluation metrics of the ISPRS 3D labeling contest, precision, recall, and F1 score, overall accuracy (OA) are used to evaluate the performance of point cloud classification. In general, OA, which is specified as the percentages of correctly classified points in the test set, examines the classification accuracy for all categories. F1 score measures the classification performance for each category based on the precision and recall of the classification model and is a better choice than OA when there is a large difference in the number of points in each category. The OA, precision, recall and F1 score are designated as follows:

where TP, TN, FP and FN, respectively, denotes true positive (the fraction of positives correctly classified), true negative (the fraction of negatives correctly classified), false positive (the fraction of positives misclassified), and false negative (the fraction of negatives misclassified).

As stated in section 3.3, the raw 3D coordinates and optional features (intensity values acquired from the airborne LiDAR data and the height above ground features are adopted in this paper) of the point cloud are input into our model; however, the entire training point set cannot be directly fed into the network because of the limited GPU memory. To resolve this problem, we separated the whole scene into a set of 30 m*30 m*40 m cuboid regions. During the model training step, to make the model more robust and prevent overfitting, we first randomly select several cuboid regions, randomly choose 8,192 points from each cuboid region and indiscriminately drop 12.5% of these points. In terms of the model testing stage, all points from split cuboid regions in the test set (see Figure. 4 b)) are fed into the trained model to achieve point-to-point classification, although the number of points is different for each cuboid region. Note that the split small broken regions that occur at the edges of the scene are merged into the surrounding larger regions to ensure the integrity of each cuboid region.

The proposed model is implemented based on the Tensorflow framework. We employ the ADAM optimizer with a learning rate of 0.01 and divide the learning rate by 2 every 3,000 steps. The batch size is set to 8 for model training and to 1 for model testing. In the encoder network, the number of sampling points in each sampling layer is set to 1024, 512, 64, and 16, with the feature dimensions C1, C2, and C3 (see Figure 2) in each proposed graph attention convolution module set to (32, 32, 64), (64, 64, 128), (128, 128, 256) and (256, 256, 512), and the number of neighboring points K set to 32. In the decoder network, the feature dimensions C2 and C3 (without global attention in the decoder) in each proposed graph attention convolution module are (512, 512), (256, 256), (256, 128) and (128, 128), individually, and the number of neighboring points K is 16. Note that the model parameters are determined through a series of comparative experiments; we do not elaborate on the experimental details since that is not the focus of our paper.

As covered in the last section, the model training is conducted on the ISPRS dataset for 1000 epochs, taking 10 hours on a Titan Xp GPU until convergence. Subsequently the point cloud of each cuboid region of the test set (Scene 2 and 3) is directly input into the trained model in turn to predict the semantic labels of each point. Figure 5 and Figure 6 show the classification results and error maps of our GACNN model which correctly labelled the majority of the points in the test set.

To quantitatively assess the classification performance of our model for the points in each category, the classification confusion matrix, precision, recall and F1 score of each category are all calculated and listed in Table 2. Table 2 shows that the proposed model achieves satisfactory classification (F1 score greater than 75%) on six categories, namely, power line, low vegetation, impervious surfaces, car, roof, and tree. Acceptable classification performance is achieved on facade categories due to the mingling of facade category with the roof, shrub and tree categories, which causes the misclassification of facade points.

However, our model demonstrates poor classification performance for the fence/hedge and shrub categories, most likely because of the similar geographical distribution and topological features of these two categories. In addition, Table 2 shows that some points of the shrub category were misclassified as tree, which may also result from the mixing and lack of obvious boundaries between the shrub and tree categories. As shown in Figure 7, points of the shrub category are mixed with points of the fence/hedge category (marked with the left circle and right circle) and points of the tree category (marked with the right circle).

To evaluate the effectiveness of the proposed graph attention convolution module (i.e. the global attention module and local attention module), we design an ablation experiment to compare six models: a) GACNN model without the global and the local attention modules (w/o global, w/o local), b) GACNN model with the global attention module but without the local attention modules (w global, w/o local), c) GACNN model without the global attention module but with the local attention modules (w/o global, w/ local), d) GACNN model with the global and the local edge attention module (w/ global, w/ local_edge), e) GACNN model with global and local density attention module (w/ global, w/ local_density), and f) GACNN model with the global and the two local attention modules (w/ global, w/ local).

Table 3 shows the classification results of the above six models. Take local attention module as an example, it can be found that adding local attention module helps to improve the classification performance by comparing a) GACNN with c) GACNN model, and b) GACNN with f) GACNN model. To be more specific, the comparison of classification results between b) GACNN and d) GACNN model, as well as e) GACNN and f) GACNN model demonstrate the effectiveness of the first local module, namely edge attention module. Similarly, the value of the second local module, density attention module, can be illustrated by comparing b) GACNN with e) GACNN model, and d) GACNN with f) GACNN. We can implement alike analysis to verify the effectiveness of global attention module. Trough these comparisons, it can be found that each attention module helps to improve the classification performance to some extent.

To more intuitively demonstrate the effects of each attention module, we randomly select a region and plot the classification results of the six models, as shown in Figure 8. The three models without the local edge attention module (a, b and e) misclassify the roof category points as tree category points, which validates our assumption that the addition of the local edge attention module promotes the classification performance by dynamically learning the convolution kernel according to the local structure of the point cloud.

Furthermore, Figure 9 visualizes the feature maps of the global and local attention modules in the first encoder layer of the GACNN model on a randomly selected region. The input point cloud on this region is first dowsampled into 1024 points and then fed into our graph attention convolution module. Three attention feature maps of the sampled points are procured by implementing the procedures illustrated in Figure 2. The black circles and arrows in Figure 9 show the partial enlargement of the local neighboring points of the selected point. The learned local features of the two local attention modules correspond to the label of the neighboring points in general. Moreover, the global attention feature map shows that the features of the neighboring points differ when the distance between the selected point and the surrounding points varies. Finally, the output feature maps (two channels) of the graph attention convolution module are shown on the right-most side of Figure 9. The features of impervious surfaces and roofs are substantially captured by our graph attention convolution module, which further authenticates the effect of the global and local attention modules.

After illustrating the effectiveness of each attention module, we compare our model with other models submitted to the ISPRS 3D Labeling benchmark to exhibit the advantages of the proposed model. The top eight models with the highest performance on the benchmark, including UM (Horvat et al.,, 2016), WhuY2, WhuY3 (Yang et al.,, 2017), LUH (Niemeyer et al.,, 2016), BIJ_W (Wang et al., 2018b, ), RIT_1 (Yousefhussien et al.,, 2018), NANJ2 (Zhao et al.,, 2018) and WhuY4 (Yang et al.,, 2018)), are selected for performance comparisons. In addition, we also compare the proposed model with our previous D-FCN model (Wen et al.,, 2020), which has been shown to achieve better classification performance than that of the mentioned eight models. Table 4 displays OA and F1 score of our model and the comparison models on the ISPRS benchmark dataset. Our GACNN model achieves better classification performance in terms of average F1 score than the other models. Specifically, the proposed GACNN model achieves state-of-the-art classification performance for the power line, impervious surfaces categories.

In addition, Table 4 shows that the previous models, such as the NANJ2 model and D-FCN model,unilaterally achieves good results on a certain evaluation metric (overall accuracy or average F1 score), but performs poorly on the other indicator. By contrast, our model not only achieves the best performance on average F1 score, but also has a satisfying result on overall accuracy in comparison to other models. Meanwhile, it exhibits difficulty to simultaneously achieve the best overall accuracy and average F1 score on this benchmark dataset, since the number of points in various categories is extremely uneven (see Table 1). The categories with a large number of points receive more attention during model training, while the small categories are ignored, which leads to good OA performance but poor average F1 scores. On the other hand, performance enhancement on average F1 score will require the model to learn representative features on a small amount of data, which easily results in model overfitting and constrains on overall accuracy. Overall, our GACNN model achieves a satisfactory balance between OA and average F1 score.

Similarly, to further intuitively illustrate the classification results of different models, we plot the results of four models in Figure 10, including RIT_1, NANJ2, WhuY4, and D-FCN, which acquire the four best classification performance in terms of average F1 score . As shown in the Figure 10, the three models (a,b,and c) in the first row misclassify the points of facade category into the shrub and roof categories, which validates our assumption that facade category is difficult to distinguish from other categories (see Section 4.4). The main reason these models fail is that they cannot learn effective local neighboring features of the point cloud. Specifically, the RIT_1 model adopts a 1D fully convolutional network that cannot capture multiscale features, and NANJ2 and WhuY4 model lose 3D spatial information during the transformation of point cloud to 2D feature images. However, D-FCN and GACNN model learn effective local structural features by employing their core modules, thus obtaining better performance than that of the other models. Moreover, although most points in the low vegetation category are misclassified as impervious surfaces by all the models, our GACNN model still achieves an acceptable result.

Moreover, to fairly justify the superiority of our model, we compare our model with other prevalent point cloud deep learning models in the field of computer vision, including PointNet++ (Qi et al., 2017b, ), PointSIFT (Jiang et al.,, 2018), PointCNN (Li et al.,, 2018), PointCNN with A-XCRF model (Arief et al.,, 2019), KPConv(Thomas et al.,, 2019), DGCNN(Wang et al., 2019b, ) and GACNet(Wang et al., 2019a, ) (see Table 5). It can be seen from Table 5 that the proposed GACNN model have achieved better performance than these advanced point cloud deep learning models, which again demonstrate the superiority of our model. Please note that PointCNN with A-XCRF model is a post-processing method to address the overfitting of PointCNN, and our model has not adopted any post-processing technology. Furthermore, compared to DGCNN and GACNet, which are closely related to our work and recently proposed in the field of computer vision, our model achieves better accuracy and F1 score by a large margin. It demonstrates that adding attention mechanisms does improve model performance and the two attention mechanisms employed in this paper are more effective, obtaining superior classification performance than GACNet with only a local spatial attention mechanism.

Furthermore, to analyze the computational cost of our model, we compare the per sample inference time of the proposed model with recent state-of-the-art models. As shown in Table 6, it can be noticed that the inference time of our model is close to the comparing models. Our model achieves new state-of-the-art classification performance with a promising computation efficiency.

To demonstrate the generalization capability of the proposed model, we further conduct experiments on 2019 Data Fusion Contest Dataset. The 2019 Data Fusion Contest, organized by the Image Analysis and Data Fusion Technical Committee (IADF TC) of the IEEE Geoscience and Remote Sensing Society (GRSS), the Johns Hopkins University (JHU), and the Intelligence Advanced Research Projects Activity (IARPA), provides a large-scale Urban Semantic 3D (US3D) dataset covering approximately 100 square kilometers over Jacksonville, Florida and Omaha, Nebraska, United States. The point clouds in US3D dataset are labeled as five categories, including ground, trees, buildings, water and bridge, and are stored in ASCII text files with format x, y, z, intensity, return number.

This training set of Contest Dataset consists of 110 regions, from which the 100 regions are selected for the training dataset and the other 10 regions are used for testing dataset in this experiment. The number of points in this dataset and detailed category distribution is shown in Table 7. Similarly, we randomly select a 128m*128m*210m cuboid region from each region during training and arbitrarily choose 8192 points from the cuboid as model input. As for the testing stage, each region of the testing dataset is split into blocks of 128m*128m grids in the horizontal direction and all points is segmented blocks are input into the model for prediction. Note that the architecture and parameters of our model remain unchanged.

We first evaluate the effectiveness of the proposed graph attention convolution module and conduct an ablation experiment similar to what we have done in Section 5.1. The classification results of the six models which comprise of different attention modules are listed in Table 8, from which it can be found that the proposed local and global attention module do help to improve the classification performance to some extent as we discussed in Section 5.1.

The PointNet++ (Qi et al., 2017b, ), PointSIFT (Jiang et al.,, 2018), PointCNN (Li et al.,, 2018), KPConv (Thomas et al.,, 2019), DGCNN (Wang et al., 2019b, ), GACNet (Wang et al., 2019a, ) and DFCN model (Wen et al.,, 2020) are selected as comparison model, and the classification results on test regions are shown in Table 9. From Table 9, it can be seen that our model obtains the best classification performance on average F1 score and achieves the highest F1 score for three of the five categories, including ground, building and bridge deck. Moreover, Figure 11 displays the classification results of the four latest models among above prevalent point cloud deep learning models on a randomly selected test region, from which we can find that KPConv model and our model classify the points of bridge deck category more correctly. But there are some points of building category are misclassified by KPConv model. At the same time, from Table 9 and Figure 11, it can be found that our model, which incorporates local structural features and global contextual information simultaneously, performs better than DGCNN model and GACNet model, two closely related works in the field of computer vision, which further indicates the effectiveness of the proposed attention modules. As a result, the proposed model still outperforms other state-of-the-art models on the 2019 Data Fusion Contest Dataset and demonstrates a favorable generalization capability.