DV-ConvNet: Fully Convolutional Deep Learning on Point Clouds  with Dynamic Voxelization and 3D Group Convolution

Given an input point cloud and a sampling centroid, we first draw $K\times D$ nearest neighbouring points (both dark grey and light grey points in Fig. 2a) around the sampling centroid using the k-nearest-neighbours (k-NN) searching algorithm. Next, $K$ points (dark grey points in Fig. 2a) are randomly picked among these $K\times D$ neighbours and are fitted into a cubic voxel kernel centering at the sampling centroid, while the kernel radius is determined such that the farthest neighbouring point can just be included. Thus, the receptive field of each convolution kernel can dynamically adapt to the local points density in each convolution layer.

Each sampling kernel consists of $S^{3}$ uniform regular grids ($S$ is known as kernel size in this context), while the representative feature of each grid is calculated based on the points falling into it. For a grid occupied by one single point, the feature of that point is treated as the feature representation; for a grid not occupied by any point, zero values are assigned for that grid following the practice in [8]. If a grid is occupied by more than one point, a max-pooling strategy is applied, where the channel-wise maximum across all the interior points is used as the representative feature for that grid. This is analogous to the max-pooling method used in traditional CNNs, but ours occurs within each grid instead of over the entire kernel. Indices of the points that contribute to the grid features are recorded during the network forward propagation and they are used to guide the gradient backward propagation, therefore, our dynamic voxelization operation is fully differentiable. For each grid, only the first five points encountered during the forward propagation are taken into consideration, while the rest are neglected. Since the computation expense varies cubically along with kernel size $S$, to control the computation expense within a rational range, $S$ is fixed as 3 for all the DV-Conv layers. During the dynamic voxelization operation, $N$ sampling centroids are drawn from the input point cloud using the farthest point sampling method, thus, the dynamic voxelization operation converts a point cloud in $\mathbb{R}^{C_{in}}$ into $N\times S^{3}$ voxels in $\mathbb{R}^{C_{in}}$, where $C_{in}$ is the number of feature channels of the point cloud.

We highlight the efficiency of the dynamic voxelization operation when compared with the global voxelization, which is commonly used in previous works [24, 34, 27, 42]. Though the invention of sparse convolution alleviates the computation inefficiency issue induced by the point cloud sparsity, how to choose the proper resolution for the global voxelization preprocessing is still a crucial yet challenging problem, since an improperly selected resolution may cause either severe information loss or massive computation overheads, as illustrated in Fig. 1b and c. In contrast, as shown in Fig. 1d, both the locations and receptive fields of the convolution kernels are more concentrated at the locations where the points are densely gathered, such as the floral discs and vase, and vice versa. Moreover, the introduction of dilation rate $D$ further adjusts the receptive fields to include richer features without additional computation expense, and in this way, the computation power is more efficiently allocated.

Specifically, [11] shares the concept of point-wise convolution with our work, but the similarity ends there. Dynamic voxelization sampling is the key to solve the following architecture limitation of [11] which severely hamper its real-world applications: [11] is unable to organize into a hierarchical network architecture, because each layer in [11] always shares the same receptive field. Unless dynamic voxelization sampling is introduce, the convolution layer in [11] cannot adapt to the dynamic change of feature points, not to mention the encoder-decoder architecture for segmentation tasks. The performance on benchmark datasets in Table 1 and 5 further illustrates the inherent difference between [11] and our work, despite their similar network architecture for the classification task.

Mathematically, the convolution between 3D feature voxels $\mathbf{F}$ and a 3D convolution kernel $\mathbf{W}$ can be expressed using Eq. 1:

where $\star$ is the convolution operator, $\mathbf{c}$ is the coordinate of voxel in $\mathbf{F}$, $\mathbf{c}=[x,y,z]^{T}\in\mathbb{Z}^{3}$, $t$ stands for the translation for $\mathbf{W}$, and we omit the summation among feature channels for simplicity. Eq. 1 can be interpreted as follows: shifting $\mathbf{W}$ to the convolution location $\mathbf{c}$ in $\mathbf{F}$ using translation $t$, is equivalent to finding the corresponding location of weight in $\mathbf{W}$ by using the inverse translation $t^{-1}$ on $\mathbf{c}$, that is $[t\mathbf{W}]_{\mathbf{c}}=\mathbf{W}_{t^{-1}\mathbf{c}}$. For any translation $s$ acts on $\mathbf{F}$, we have:

Note that we leverage the substitution: $s^{-1}\mathbf{c}=\mathbf{c}^{\prime}$, and $\mathbf{c}^{\prime}$ is still defined in $\mathbb{Z}^{3}$. Thus, a 3D convolution is equivariant to translation by its definition.

Now we extend the summation domain from $\mathbf{c}\in\mathbb{Z}^{3}$ to $g\in G$ as illustrated in Eq. 3, where $G$ is a group of transformations which are equivariant to a wider range of transformations like rotation and reflection, and $G$ is also defined in $\mathbb{Z}^{3}$.

Similarly, the conclusion like Eq. 2 still holds, which means, for any $g^{\prime}\in G$, $\mathbf{W}\star[g^{\prime}\mathbf{F}]=g^{\prime}[\mathbf{W}\star\mathbf{F}]$, in other words, the group convolution is equivariant to the transformation group $G$. In this work, we employ $p4$ and $p4m$ symmetry groups as our $G$s, and extend them to $\mathbb{Z}^{3}$ from their originally defined domain $\mathbb{Z}^{2}$.

$\mathbf{p4}$ / $\mathbf{p4m}$ Group. The $p4$ group comprises a group of rotation transformations by $0$, $\pi/2$, $\pi$, and $3\pi/2$, regarding any rotation center in a square grid, and it is easy to conclude that $p4$ is a symmetry group regrading rotation by $\pi/2$. Expressing $p4$ symmetry group in a homogeneous coordinate system, gives:

where $r\in\left\{0,1,2,3\right\}$ and $(x,y)\in\mathbb{Z}^{2}$. To extend the $p4$ group to $\mathbb{Z}^{3}$, we generalize its working domain from a square grid to cubic space by adding an additional $z$-axis. Putting the rotation center at the centroid of the cube and keeping the rotation at the $x$-$y$ plane, Eq. 4 can be rewritten as:

The $p4m$ group extends the $p4$ symmetry by involving additional horizontal mirroring (reflection) transformations:

where $m\in\left\{0,1\right\}$. When $m=0$, $p4m$ group reduces to $p4$.

Implementation. Given the feature voxels $\mathbf{F}\in\mathbb{R}^{N\times S^{3}\times C_{in}}$ yielded from the dynamic voxelization operation, we define a convolution kernel $\mathbf{W}\in\mathbb{R}^{S^{3}\times C_{in}\times C_{out}}$ to convolve with $\mathbf{F}$, where $C_{in}$ and $C_{out}$ refer to the number of feature channels for input and output. According to the definition of group convolution, for each $g\in G$, we are able to transform the kernel $\mathbf{W}$ to $\mathbf{W}_{g}$ using $\mathbf{W}_{g}=g\mathbf{W}$. We use the notation $\mathbf{W}_{G}\in\mathbb{R}^{S^{3}\times C_{in}\times C_{out}\times n}$ to represent a concatenation of $\mathbf{W}_{g}$ transformed via all the $g\in G$, where $n$ is the total number of transformations $g$ in group $G$, $n=4$ for ${G_{p4}}$ and $n=8$ for ${G_{p4m}}$. A visualization of $\mathbf{W}_{G_{p4}}$ and $\mathbf{W}_{G_{p4m}}$ is given in the Fig.3 regarding different transformation parameters $r$ and $m$ in Eq. 6.

We evaluate DV-ConvNet on ModelNet40 [42] with a 9,843 / 2,468 training / testing split, and to make a fair comparison, we evaluate our model based on both “pre-aligned” and “unaligned” mode, following the practice in [22]. We use the dataset provided in [26], and randomly select 1,024 points with their normals as our input point cloud. Anisotropic triaxial scaling in the range $[0.95,1.05]$ is used for data augmentation in both evaluation modes. The test results and comparisons with other works are given in Table 1. It can be seen that DV-ConvNet equipped with $p4$ group achieves the best performance among all the other methods on both “pre-aligned” and “unaligned” evaluation modes. The reason why $p4m$ based network performs less well is due to the number of trainable parameters being only a half of the network equipped with $p4$ group.

We evaluate the segmentation performance on the ShapeNet-part [5] dataset using a 14,007 / 2,874 training / testing split. We use the dataset provided by [26] and randomly sample 2,048 points with their normals as our input point cloud. For point cloud which originally has less than 2,048 points, we pad them into 2,048 points using random duplication and assume the category for each point cloud is priorly known and the irrelevant predictions are masked following the practice in [22, 8]. The testing results on ShapeNet-part dataset are presented in Table 2. Compared with other works, we are on-par with the state-of-the-art performance. Interestingly, DV-ConvNet performs quite well for inputs with symmetrical shapes, such as “rocket” and “cap”, where all the other methods yield less satisfactory results.

We also evaluate the segmentation performance on S3DIS [2], a large-scale dataset for in-door scene point cloud semantic segmentation . We crop each room into $1.5\times 1.5$ $m^{2}$ tiles and pad an additional 0.2 m offset at the surroundings for each tile. The points in the padding area only provide additional ambient information and they are not linked with the loss calculation in both training and testing phases. We randomly sample 4,096 points (including offset) with their RGB values as our input. Our evaluation is conducted based on both 1-fold and 6-fold cross validation, and the results are shown in Table 3 and Table 4, respectively.

The $p4$ group based DV-ConvNet reaches the state-of-the-art performance regarding all the three main evaluation indices for 6-fold cross validation, and it also achieves the best IoU performance for 6 out of 13 categories. Compared with the previous state-of-the-art PointCNN [22], our proposed DV-ConvNet yields much higher prediction accuracy, boosting the $mAcc$ index by 9.5% at most. We compare to the VV-Net equipped with group convolution only, thus it has the same voxelized input as ours. However, compared to its VAE equipped version, our method still overperforms it regarding the mpIoU index in ShapeNet (84.6% vs. 84.2%) and the OA index in S3DIS (90.06% vs. 87.78%).

To better illustrate the computational efficiency of DV-ConvNet, we compare the runtime statistics between our method and other works. The statistics are summarized in Table. 5, calculated based on the same hardware configuration with a batch size of 32 during the training phase. Through the comparison, our DV-ConvNet achieves the best performance on ModelNet40 (Table. 1), while using the least FLOPs and processing time.

PointNet++ [28] and PointCNN [22] take raw point cloud coordinates as input, even though they have a relatively small number of parameters, they still involve massive FLOPs, since they employ rather complex MLPs based network architectures to learn shape representations from the point coordinates. PCNN [37] is a continuous convolution based method, as aforementioned, with the expensive polynomial based convolution kernel, it has the highest processing time cost among all the methods. For the Minkowski-ConvNet [6], we do not follow the architecture suggested (ModelNet40 evaluation is not reported in its original paper), since it has a rather complicated ResNet based architecture. Instead, we setup a similar 4-layer architecture as our DV-ConvNet for fair comparison, and based on this configuration, we notice that Minkowski-ConvNet can only achieve  88% OA on ModelNet40. We believe it encounters severe quantization artifact when the resolution is not sufficient. Increasing resolution will lead to a better performance, but more FLOPs will be involved as well.

We plot the overall accuracy (OA) against training epoch curve of DV-Conv and PointCNN [22] in Fig. 5, based on the ModelNet40 under the "unaligned" evaluation mode. Since the shape information are explicitly preserved by voxels, our DV-ConvNet is able to converge at a considerably faster speed, leading to more efficient model development and tuning procedure. DV-ConvNet achieves 90% OA after about 20 training epochs, while by contrast, PointCNN takes about 40 epochs.

During dynamic voxelization operation, a grid is assigned with multiple candidate points (any points falling into that grid), and the channel-wise maximum among all the candidate points is chosen based on the max-pooling operation. By doing so, we enlarge the receptive field while keeping the computation expense within a rational range and avoid losing critical information.

$K\times D$ controls the size of receptive field. Here we study the impact of $K$ and $D$ on the model performance by increasing one and decreasing the other accordingly, so that the relative receptive field in each convolution layer remains unchanged. Evaluation results on ModelNet40 are shown as tests 1-4 in Table 6. The best performance is achieved when $D=2$.

Apart from the max-pooling methods, we also test the performance based on average pooling, where the channel-wise averages are used. The results are shown as test 5 in Table 6, and we find that the performance drops by 0.6% when max-pooling is replaced by average-pooling. In addition, the sampling strategy using fixed sampling radius instead of k-NN is used, where the sampling radius is kept the same as the relative receptive field of each convolution layer (shown as test 6 in Table 6). Again, there is an obvious performance decrease, and this harmonizes well with the intuition: voxelization with self-adaptive resolution fits better for those points inhomogeneously distributed in space.

Apart from the $p4$ and $p4m$ based DV-ConvNet models, we also evaluate the performance on a baseline model to verify the effectiveness of group convolution. Our baseline model shares the same network architecture with DV-ConvNet, while the only difference is that all the group convolutions are replaced by plain 3D convolutions. The comparison between test 2 and test 7 further verifies that our DV-ConvNet is indeed able to benifit from group convolution techniques.