Projection-based Point Convolution for Efficient Point Cloud Segmentation

Recent developments in 3D scanning devices have incorporated a great amount of 3D data into machine vision applications, such as robotics, autonomous vehicles, and VR/AR. Point cloud is a popular type of data with 3D geometry, and thus it is significantly beneficial to have a reliable autonomous point cloud processing system for better 3D perception. Although computer vision algorithms have achieved remarkable improvements in image recognition, 3-dimensional data is naturally different from images that have only 2 spatial dimensions. Thus, researchers have studied novel methods to deal with 3D data effectively.

As convolutional neural networks (CNNs) have been proved to be effective in image recognition, researchers have attempted to extend the use of CNNs to 3D data processing. Several voxel-based methods [27, 40, 9] applied CNNs to 3D data, but this type of approach usually suffered from computational complexity of 3D convolutions. Higher voxel resolution led to a rapid increase in memory requirements and computation time, whereas lower resolution caused larger quantization errors. Alternatively, point-based methods [29, 30, 18, 46, 42] could alleviate the exhaustive memory requirements, by allocating memory that is linearly proportional to the number of points. However, during local information aggregation, frequent neighbor search and dynamic kernel computation led to delayed latency due to the irregular memory access and additional alignment computation, as pointed out in [24].

In this study, we aim to design a point convolutional module that does not have the limitations of 3D convolutions and point-based convolutions. Therefore, we choose 2D convolutional blocks as the fundamental operator. Even though projection-based methods for 3D data have been suggested before [34, 32, 17, 47], they showed degraded performance due to the nature of 2D convolutions, which distorts the 3D geometry and thus is not as expressive as 3D operators on 3D data. We intend to tackle a few problems that projection-based methods usually face. First, an N$\times$C point cloud needs to be transformed into a H$\times$W$\times$C feature map to be processed with 2D convolutions. In naive projection methods, e.g., removing one of the coordinates, information loss is inevitable due to the dimension reduction. Next, when projected features are processed through separate CNNs and merged at the final stage, intermediate features are not aware of context information from other projections, which limits the advantage of multi-view projection.

To this end, we propose Projection-based Point Convolution (PPConv), a point convolutional module for point cloud processing with 2D convolutions. In PPConv, two separate branches process point features: point branch applies MLPs to point features and projection branch uses 2D convolutions. In the projection branch, point clouds are projected through PointNet-based learnable projection, which minimizes the information loss during the dimension reduction. Furthermore, features from each branch are fused inside the convolutional module, so that the output of PPConv is always the aggregated information of all the branch features. For the feature fusion, we propose Importance-Weighted Fusion and Context-Aware Fusion modules that enables effective fusion of multiple features. By integrating all these modules, PPConv maximizes the effectiveness of 2D convolutions on point cloud processing.

We demonstrate performance and computational efficiency of PPConv by means of comparison with several state-of-the-art methods. To strictly measure the effect of convolutional modules, we use a backbone of PointNet++ [30] with architecture hyperparameters used in [24], and only replace the convolutional modules by PPConv. The experimental results on S3DIS [1] indoor scene segmentation demonstrate that PPConv shows superior efficiency among the compared methods, in terms of the trade-off between inference time and segmentation performance. Furthermore, PPConv also achieves comparable performance to the state-of-the-art on ShapeNetPart [48].

Contributions of this paper are as follows:

• •

  We propose the 2D convolution-based point convolutional module, which can be used as a building block of a point-based network for point cloud processing.

• •

  We propose novel feature fusion modules which can effectively fuse point features generated by different types of convolutional operations.

• •

  We compare the inference time of state-of-the-art methods and some representative studies on the same hardware environment, using their publicly available implementations, and show that PPConv is one of the most efficient modules in terms of the trade-off between inference time and performance.

Voxel-based methods use voxel representations for 3D data processing. VoxNet [27] used 3D CNNs on 3D data, such as raw LiDAR point clouds, RGB-D data, and CAD models, after transforming them into voxel representations. To address the computational burden of 3D convolutions, octree-based method [40] and sparse 3D convolution [9] were proposed, both of which reduced redundant computations at unoccupied spaces. Another study [4] proposed optimized sparse convolution for processing point cloud with varying density.

Point-based methods have recently focused on hierarchical feature generation on point clouds, following the pioneering work of PointNet [29]. PointNet++ [30] used PointNet on local point sets to extract local features, while some other works [18, 46, 23, 6] attempted to infer weights from the relative location or local structure of the points. Local projection was proposed in [36, 19], in which 2D convolutions were applied to extract local features. Various point-based convolutional operations [2, 51, 37, 26, 44, 7, 31, 43, 22] were recently proposed to properly capture local context of point clouds. Construction of graph was also suggested in other works [14, 42, 39, 20, 49] for effective point cloud processing. Some studies focused on the speed of point-based networks, by voxel-based neighbor search [45] or random down-sampling [12]. Other recent studies attempted to improve segmentation by predicting labels in each local region [8] or by matching categories with neighboring points [25].

Several other studies have attempted to combine the advantages of voxel-based and point-based methods. In PVCNN [24], point-wise MLP was used along with 3D convolutions on coarsely voxelized data. MLP extracted the individual point features, while 3D convolutions aggregated local information from adjacent voxels. These two features were added to produce the output of a PVConv module. The aid of MLP alleviated the requirement of high voxel resolution, because MLPs could focus on fine-grained features while voxel features captured larger contextual features. Lower voxel resolution led to significant improvement in computational efficiency, which is the main contribution of PVCNN. Our research is highly motivated by PVCNN and mainly compared with it. Following PVCNN, Sparse Point-Voxel Convolution [35] was suggested to process sparse point cloud with PVConv, and FusionNet [50] was proposed to extract local features with an improved voxel feature aggregation module.

In Projection-based methods, 2D CNNs have been used in 3D shape recognition or scene analysis. CNNs have been applied to rendered images [34] or depth images [32] transformed from 3D data. Projection-based methods have also been proposed for object detection [15, 53, 41], scene segmentation [16, 47], registration [17], and 3D reconstruction [28, 21]. Furthermore, projection onto higher-dimensional lattices was suggested for effective feature aggregation [33].

Most of the recent point cloud processing networks use either voxel-based 3D convolutions or point-based convolutional operations, as summarized in section II. However, each group of methods has critical limitations as pointed out in [24]. In voxel-based methods, memory usage and computational cost grow cubically with respect to the voxel resolution, which is considered to be a severe problem in scalablity. To alleviate this problem, some recent studies [4, 35, 9, 50] proposed to use sparse 3D convolution, which helps reduce redundant computations and thus enables to use small voxels in large scenes. In this paper, we take a different strategy to reduce computation by not using 3D convolutions at all, but instead processing point clouds with only 1D and 2D convolutions.

Point-based methods [18, 46, 37] suffer from delayed processing time caused by the additional computations needed for nearest neighbor search or kernel weight interpolation. In point clouds, the neighboring points are not saved in the memory contiguously: local point grouping operations require either radius search or k-nearest neighbor search. Another problem is that the input point locations vary for every sample, so the kernel points must be aligned with them before computing features. Based on these observations, we aim to use grid-based convolutions as the core operation to avoid inefficient computations. Thus, our design goal can be summarized as to process 3D point clouds with 1D and 2D operations on a grid structure.

In this section, we introduce Projection-based Point Convolution (PPConv) module. The overall framework is depicted in Fig. 1. As a point-based convolutional method, PPConv module takes a point cloud $P=\{p_{1},p_{2},\cdots,p_{N}\}\in\mathbb{R}^{N\times(3+C_{in})}$ as input and outputs the transformed point cloud $P^{\prime}\in\mathbb{R}^{N^{\prime}\times(3+C_{out})}$. Each point $p_{i}$ contains its point coordinates $c_{i}$ and point features $f_{i}$, i.e., $p_{i}=(c_{i},f_{i})$. In the first layer of the network, the input may be only the 3D coordinates of each point, or may have extra features such as RGB values or normalized coordinates ($C_{in}=6$ when RGB values and normalized 3D coordinates are used). In the other layers, each point has its coordinates $c_{i}$ and features $f_{i}$, which are the output of the previous layer ($C^{l}_{in}=C^{l-1}_{out}$, where $l$ is the layer index). In PPConv, $P$ is processed through two separate pathways: the point branch and the projection branch. In the point branch, MLP transforms each point feature individually. In the projection branch, the point cloud is projected onto a 2D plane and subsequently processed using 2D convolutions. Then, the feature map is backprojected to the point space and combined with the other features through fusion module. Each branch is further explained in the following subsections.

In order to demonstrate the effect of PPConv module, we follow the PointNet++ [30] architecture as used in PVCNN [24]. In PointNet++, Set Abstraction (SA) module aggregates the local geometry of each sampled point through a local PointNet. Feature Propagation (FP) modules use MLPs to aggregate multi-level features, which consist of interpolated higher-level features and previously extracted lower-level features from the corresponding SA module. We refer the readers to PointNet++ paper [30] for further details.

We follow the layer configurations of PVCNN++ architectures to purely compare the effectiveness of the convolutional module. For PPConv, we use the grid resolution of 64 for the first layer, which is determined through ablation study (in Table XI). Hereafter, PointNet++ architecture equipped with PPConv is denoted as PPCNN++. In Section VI, PPCNN++ architectures with different configurations are indicated by PPCNN++(config). For example, PPCNN++ model with 3 projection branches (x-, y-, and z-axis projection) and Importance-Weighted Fusion (IWF) module is denoted as PPCNN++(x,y,z,IWF).

Following recent state-of-the-art methods [37, 4, 31], we use cross entropy loss for each point. As semantic segmentation task targets for a class prediction for every input point, we simply apply the point-wise classification loss to each point and average those values to get the loss value for the input point cloud.

In this section, we present analysis on components of PPConv by providing experimental evidence of the effect of each module. The analysis was performed using S3DIS dataset using experimental protocol of PVCNN [24]. For every experiment in ablation studies, we trained the model 5 times with different random seeds, and report the averaged performance.

First, we demonstrate the effect of each branch of PPConv module: projection branch and point branch. We trained the model without each branch and report test performances in Table VIII. For the model without point branch, feature fusion was performed by adding 3 projection branch features and then applying a single-layer MLP. The results show that removing either branch shows significant performance drop, supporting that each module extracts complementary features to each other.

Next, we evaluated PPCNN++ models with different number of projection branches. We trained 3 different models, each with 1 (z), 2 (x,z), and 3 (x,y,z) projection branches. Every model included point branch in this experiment. The results are shown in Table IX, which shows increasing performance as projection branch is added. On the other hand, less projection branch results in faster runtime, which gives possible choices over the trade-off between runtime and performance.

We also evaluated different projection methods to demonstrate their effect on the segmentation performance. PointNet-based projection was mainly used throughout the experiments in the paper. We evaluated simpler projection methods, averaging features and bilinear interpolation, which was explained in Section IV-B1. The results in Table X present that PointNet-based projection is clearly better than the other methods, justifying the use of PointNet-based projection for segmentation.

For the grid resolution during projection, we tested a few choices in order to determine the optimal value. We evaluated four different resolutions: 32$\times$32, 48$\times$48, 64$\times$64, and 96$\times$96. Through experimental results, which is shown in Table XI, we observed that 64$\times$64 shows the best performance and chose this value as the default experimental setting in this paper.

Then, we present experimental results of variations in 2D convolutional modules. As 2D convolutional modules deal with projected feature maps, which are of the same shape as general image feature maps, any techniques that can be combined with 2D convolutions can be used. We test two of the most popular modules: residual connection and Squeeze-and-Excitation [11] module. We trained three different models: without any of the two components, with only residual connection, and with both modules. The segmentation performance, as shown in Table XII, increases as each module is added to the 2D convolutional module.

Finally, we trained models with different fusion strategies to demonstrate the effectiveness of proposed fusion modules. The default fusion adds 3 projection branch features and then concatenate it with the point branch feature. Then, a single-layer MLP is applied to produce the output feature with the pre-defined channel dimension. We proposed 2 different fusion modules in Section IV-C, and provide the experimental results in Table XIII. The results show that both fusion modules contribute to performance improvement over the default method, while runtime is also increased due to the additional MLPs and matrix computations. Context-Aware Fusion module shows noticeably larger improvement while increase of runtime is comparable to that of Importance-Weighted Fusion.

We proposed PPConv, an efficient local aggregation module for 3D point cloud processing. The experimental observations demonstrated that efficient 2D convolutional modules can facilitate faster computation while keeping performance not far behind from that of 3D convolutional modules. PPConv shows remarkable efficiency in terms of the trade-off between inference time and segmentation performance, providing an efficient alternative for point-based convolutional models. PPConv can process both object part segmentation and scene segmentation under appropriate network architectures, thus can be used as a generic building block for 3D point cloud processing networks.