Laplacian Unit: Adaptive Local Detail-Preserving Filtering for 3D Point Cloud Understanding

Early attempts to apply deep learning on 3D point clouds project raw point clouds onto regular 2D (view) or 3D (voxel) grids to enable successful grid-based convolution operations. View-based methods [su2015multi, kanezaki2018rotationnet, feng2018gvcnn] project point clouds onto several 2D planes from different viewpoints. Multi-view images are subsequently processed by using 2D CNNs. On the other handIn contrast, voxel-based methods [maturana2015voxnet, zhou2018voxelnet, graham20183d, choy20194d] project point clouds onto 3D regular voxel grids and apply 3D convolutions. The performance of view-based methods heavily relies relies heavily on the choice of projection planes, whereas voxel-based methods suffer from a substantial memory consumption. Moreover, fine-grained geometrical details are lost due to owing to the conversions.

The point-based Point-based methods, in contrast to the projection-based methods, operate directly on irregular point clouds without any conversion. Point clouds are naturally unordered; hence, such a type of method must respect permutation invariance, which ensures that the results are insensitive to permutations of points. In general, the point-based methods can be roughly categorized based on the type of the local aggregation operator they use [liu2020closer]. Pioneered by PointNet [qi2017pointnet], point-wise pointwise MLP-based –based methods [qi2017pointnet++, zhang2019shellnet, lan2019modeling, wang2019dynamic] update the feature of a query point using a series of shared MLPs followed by a symmetric aggregation function (e.g., max-pooling), which guarantees the whole process to be permutation invariantthat the entire process is permutation-invariant. Given a query point, neighborhood points and their associated features are collected through a neighborhood search algorithm such as k-nearest-neighbor k-nearest neighbor (kNN) or radius search. Subsequently, features , which that are either the input features of points or the concatenation of different features (e.g., positions, input features, or edge features) [qi2017pointnet++, wang2019dynamic, cui2021geometric], are transformed by shared MLPs. Updated query point features are obtained by spatially summarizing the transformed neighborhood features by using a symmetric aggregation function. On the other handIn contrast, inspired by the success of convolution in image processing, numerous effort has been spent on realizing efforts have been made to realize convolution on 3D point clouds. In general, these methods can be categorized into 2 two types depending on how they generate convolution filters: adaptive weight-based and –based and pseudo gridpseudo-grid -based –based methods. Adaptive weight-based weight–based methods dynamically update the weights of convolution filters by transforming the input positions/features. The generated weights are applied to the input features for the convolution. For instance, spatial features such as relative positions (relative to query positions) [wu2019pointconv, wang2018deep] and Euclidean distance [liu2019relation] are used to generate filters. Moreover, some works studies use additional edge features [simonovsky2017dynamic] for inputwhile others utilize , whereas others utilize the attention mechanism [wang2019graph, zhao2019pointweb] to further modulate the generated filters. On the other hand, pseudo grid-based In contrast, pseudo-grid–based methods build artificial convolution kernels by mimicking the standard convolutions in 2D image convolutions. ConcretelySpecifically, the input features are projected onto artificial kernel points, which are additional ”grid points” with associated weight matrices (convolution filter), using similarity-based methods such as the trilinear interpolation [mao2019interpolated], Gaussian kernel [shen2018mining, atzmon2018point], or the linear correlation [thomas2019kpconv]. Subsequently, the convolution is performed on the kernel points to which the point cloud features are projected.

Based on these basic local aggregation operators, some works focus studies have focused on enhancing the network capability by improving the down-sampling downsampling procedures [xu2020grid, yang2019modeling], injecting rotation invariance [fan2021scf], or the performing careful supervision [gong2021omni].

The difference term $x_{j}-x_{i}$ is frequently referred to as the edge feature in graph-based methods [wang2019dynamic, guo2020pct], where the edge feature is used as an additional input feature for local operations. As shown in Table 2, in its simplest form, the edge feature-augmented feature–augmented method updates the feature of a query point by aggregating the non-linearly nonlinearly transformed neighborhood features (usually uses max-pooling), where each feature is a concatenation of $x_{j}-x_{i}$ and $x_{i}$. In other words, the fusion of local and global features is performed before the aggregation, and the local feature is restricted to the pairwise difference. In addition, in some cases [wang2019dynamic], edge feature-augmented methods perform the neighborhood search feature–augmented methods perform neighborhood searches in the feature space instead of the 3D space.

We think believe that the pairwise difference is not sufficiently informative for local-global fusion because it merely describes the difference between two connected points. Performing kNN in the feature space can enlarge the receptive field, ; however, it ignores the natural 3D layout of objects in a complex scene. Moreover, the feature-space kNN is much more computationally expensive than the 3D-space counterpart. Furthermore, the non-linear nonlinear transformations are applied to the concatenation of $x_{i}$ and $x_{j}-x_{i}$, which doubles increases the parameters required while increasing the optimization difficulties complicates the optimization because the function is no longer a residual function. In contrast, the Laplacian Unit unit treats the local feature as a patch instead of a pairwise difference, in which local properties are sufficiently described, hence thereby making the local feature more discriminative. We expect that the local-global fusion would be easier when local features are more meaningful. The neighborhood search of Laplacian Units units is performed in 3D space. Thus, it is fully aware of the natural 3D layout. Moreover, the Laplacian Unit unit belongs to the residual learning framework, which is known to ease facilitate optimization.

To compare the effectiveness of Laplacian Units units and edge feature augmentation, we compare the network equipped with Laplacian Units units with the edge feature-augmented feature–augmented network (denoted as Edge edge feature augmentation in Table. 11) in Sec. 5.3.

As shown in Table 2, Laplacian Smoothingsmoothing [taubin1995signal] formulates the mesh smoothing as low-pass filtering using the discrete Laplacian. It receives 3D coordinates as input and generates 3D displacement vectors which are used for moving that are used to move the vertices.

The dampening factor $\lambda$ is tuned manually manually tuned as a hyperparameter. In addition, the process is often repeated several times (the number repetition is also decided manually) to obtain the desired amount of smoothness.

In comparison with Laplacian smoothing, we design the Laplacian Unit unit as an architectural unit that is optimized along with the backbone networks. While the input is restricted to 3D coordinates for the Laplacian smoothing, the proposed Laplacian Unit unit takes arbitrary dimensional vectors as input and returns vectors of the same dimension. Unlike Laplacian smoothing, the Laplacian Unit unit considers both spatial and feature relations to model the complex structure of point clouds. Furthermore, the Laplacian Unit unit performs smoothing and sharpening simultaneously by using a learned calibration function, whereas the dampening factor (a scaler) of the Laplacian Smoothing Laplacian smoothing is handled in a hand-crafted manner. Such a way of calibration calibration method is likely to be sub-optimal suboptimal and only realizes the low-pass filtering. Moreover, while the Laplacian Smoothing Laplacian smoothing needs to be applied multiple times to obtain the desired smoothing effect, Laplacian Units aims units aim to obtain the desired amount of smoothness by single non-linear nonlinear filtering.

Since Laplacian Smoothing Because Laplacian smoothing is not directly applicable to the deep learning framework, we formulate the learned Laplacian Smoothing smoothing, where $lambda$, as shown in Table. 2is optimized with , is optimized using the network (denoted as Learned Laplacian Smoothing learned Laplacian smoothing in Table. 11). Note that we make the original Laplacian Smoothing smoothing stronger by removing the constraint of $0<\lambda<1$, which makes it possible to perform low-pass and high-pass filtering adaptively. Then, the effectiveness of the Laplacian Unit unit and learned Laplacian smoothing is are compared in Sec. 5.3).

The performance is measured using overall accuracy (OA). We use the Adam [kingma2015adam] optimizer and trained the model for 250 epochs with a batch size of 32. The initial learning rate is set to 0.001 and decayed by a factor of 10 when it plateaus. We adopt the same training strategy as in the previous work [uy2019revisiting] for fair comparisons. ConcretelySpecifically, 1,024 points are randomly sampled from 2,048 points as input. The input data are augmented by random rotations rotation (rot.) and Gaussian jittering (jit.). Moreover, to further investigate the effectiveness of Laplacian Units units on the different training strategies, we adopt another training strategy where in which the number of points is increased to 2,048. The random Random scaling (scale.) and the random translation (trans.) are applied used for data augmentation.

The results are reported presented in Table 3. Under the same training strategy as in the previous work [uy2019revisiting], significant improvements are observed when Laplacian Units units are integrated into AW and PG. In particular, PG + LU achieves the state-of-the-art performance by surpassing all previous works (the bold text in the middle part of Table 3), demonstrating the effectiveness of Laplacian Unitsunits. Note that it even outperforms networks that adopt the additional fine-grained ground truth information (BGA-PN++ and BGA-DGCNN) and the network that uses the additional data augmentations (SimpleView). We speculate that the inter-point relations exploited by convolution-based methods (AW and PG) are further enhanced by Laplacian Units. On the other handunits. In contrast, the performance of the PM drops when the Laplacian Unit unit is added. Unlike convolution-based methods, the MLP-based method does not explicitly infer inter-point relations using positional information; thus, we conjecture that the training strategy is not sufficient for the MLP-based method to obtain such an inductive bias, which is injected by design in convolution-based methods , during the training.

The performance of the methods using the second training strategy is shown in the bottom part of Table 3. Under this training strategy, we observe that the Laplacian Units Laplacian units consistently provide performance improvements. Specifically, in contrast to the first training strategy, Laplacian Units units successfully augment PM, achieving the best performance among networks (the bold texts text at the bottom of Table 3).

We use randomly sampled 2,048 points with their surface normal features as input. The input data are augmented by the random anisotropic scaling and random translation. We train the models for 150 epochs. We use the SGD optimizer with an initial learning rate of 0.1, which is decayed by the a factor of 10 when it plateaus. As a common practice, we use the voting for post-processing, following [thomas2019kpconv]. The performance metric used in this task is the instance-wise average intersection over union (Ins. mIoU) [qi2017pointnet++], and class-wise average IoU (Cat. mIoU).

As shown in Table 4, integrating Laplacian Units units into the network architecture is beneficial for all networks. Specifically, Laplacian Units units provide a significant performance gain ranging from 0.2 to 0.4 mIoU points to the backbones, which lead the PM and PG to achieve state-of-the-art performance. We conjecture that employing Laplacian Units units explicitly enables each point to recognize how they are different from each other; thus, the predictions become more boundary-aware. As shown in Fig. 8, assisted by Laplacian Unitsunits, the networks are able to recognize inter-part boundaries. In addition, to verify the effect of the Laplacian Units Laplacian units on existing architectures, we integrate Laplacian Units Laplacian units are integrated into PointNet [qi2017pointnet] and RSCNN [liu2019relation]. PointNet serves as a representative of the basic architecture, whereas RSCNN serves as a cutting-edge onearchitecture. The results are shown in the middle rows of Table 4. Laplacian Units units again manage to lift their performance significantly, which demonstrates its effectiveness on both basic and advanced networks.

During the training, following [zhao2019pointweb], we randomly sample one point from all scenes and extract a 1m$\times$1m pillar centered by this point’s horizontal coordinates. Subsequently, 4,096 points are randomly sampled from the points contained in the extracted pillar. These points are taken as an element of a mini-batch. During the test, we regularly slide a pillar with a stride of 0.5m to ensure every point is tested at least once. Input The input feature consists of 3D coordinates, RGB, and normalized 3D coordinates with respect to the maximum coordinates in a room. We use the SGD optimizer and train the models for 100 epochs in which one epoch is set to 1.5k iterations. The batch size is set to 32. The initial learning rate is set to 0.1 and decayed by a factor of 10 every 25 epochs. Following [thomas2019kpconv], we augment augmented the input with a random vertical rotation, a random anisotropic scaling, a Gaussian jittering, and a random color dropout. The performance metrics are the point average IoU (mIoU), overall accuracy (OA), and class-averaged accuracy (mAcc).

The results are shown listed in Table 5. The consistent Consistent relative performance improvements are observed by inserting Laplacian Units units into the backbone networks. Notably, though although PM achieves good performance, it is further boosted by Laplacian Units units by 0.6 mIoU points. For AW and PG, Laplacian Units even provide units provide a greater performance gain. To investigate the effect of Laplacian Units units on existing architectures, we integrate multiple Units units into both basic (PointNet [qi2017pointnet]) and advanced (KPConv (rigid) [thomas2019kpconv]) methods. Results The results (middle rows of Table 5) reveal that Laplacian Units units can boost them significantly. Notably, with Laplacian Unitsunits, KPConv (rigid) even outperforms its strong variant KPConv (deform), which is based on the deformable convolution, indicating that the information encoded by Laplacian Units is units is more beneficial than tailoring receptive fields. Laplacian Units units make use of local smoothness statistics explicitly inside the network, which force forces networks to be more boundary-aware and produce smooth predictions within that boundariesboundary. This speculation is further verified by the qualitative results shown in Fig. 9). In some cases, LU-Net is able to detect a subtle object which that is missed completely by the counterpart which that has no units integrated (third row of Fig. 9). In addition, LU-Net also produces smoother predictions, which are shown in the first, second, and last rows of Fig. 9).

During the training, we randomly take sample 8,192 points from a sphere (3m radius) sampled regularly from a scene. For testing, we regularly take spheres from the scene so that each point is tested at least once. We use the SGD optimizer and train models for 150 epochs in which one epoch is set to 500 iterations. The batch size is set to 32. The initial learning rate is set to 0.01 and decayed by a factor of 10 every 30 epochs. The same augmentation strategy used as the for indoor scene segmentation is used. The performance is measured in terms of the point average IoU (mIoU) and overall accuracy (OA).

The results of the outdoor scene segmentation are presented in Table 6. In general, Laplacian Units manage to units improve the performance of the backbones . Like in backbones consistently. Similar to the indoor segmentation case, a greater performance improvement is observed for both convolution-based methods compared with the one that of PM. This might suggest that Laplacian Units units favor more convolution methods in which spatial correlations are well-exploited compared with the MLP-based method. In outdoor scenes, many artificial objects consist of simple planes. Therefore, we think believe that the Laplacian Unit unit would be effective to classify for classifying such objects because it enforces smooth predictions within object boundaries by explicitly considering inter-point differences. For instance, despite the gap of in mIoU between Ours (AW) + LU and RFCR [gong2021omni], Ours(AW) + LU outperforms RFCR by 0.2 OA points. This reveals that Ours(AW) + LU does better in simple classes having performs better in geometrically simple classes with more samples, e.g.for example, ground and buildings.

The results concerning for the components in Eq. 10are reported , are listed in Table 10. With all components present, the network achieves the best performance, suggesting that the combination of all components is indeed vital. The performance drops by 0.1 mIoU point by removing the $\mathcal{T}$. In such a case, the Laplacian Unit unit becomes a linear filter; thus, its representational power is weakened compared with the one of the non-linear that of the nonlinear filtering. However, performance drops significantly (0.6 mIoU points) when $\mathcal{M}$ is removed, revealing that the channel relations of raw DPL need to be exploited to become useful. The significance of the local-global fusion is evidently shown as the performance sharply drops 0.7 mIoU points when the global feature, $x_{i}$, is removed from the formulation. Finally, performance drops dramatically by 2.6 mIoU points when only raw DPL remains, verifying our design choices.

We examine the sensitivity of performance on the size of the local neighborhood used in the DPL. The results are reported presented in Table 10. Among the tested sizes, we observe stable performanceexcept the , except for one of the smallest size sizes (4). Interestingly, most sizes outperform the default onesize, which is 16, indicating that performance may be further improved by looking for the best size for the task at hand.

We further compare compared the proposed Laplacian Units units to closely related works, including the edge feature-augmented edge feature–augmented networks and learned Laplacian Smoothing. Concretelysmoothing. Specifically, the edge feature-augmented feature–augmented network concatenates the $x_{j}-x_{i}$ term to the input feature during the local aggregation, which is exactly equal to the first row of the $x_{i}^{\prime}$ column in Table 2. On the other hand, as we mentioned in Sec 3.4, we convert the dampening factor $\lambda$ to into a learnable network parameter to make the original Laplacian Smoothingsmoothing  [taubin1995signal] learnable. The results are shown listed in Table 11. We empirically verify that Laplacian Units units provide the most significant relative improvement over the baseline (first row of Table 11). A minor increase in the performance is observed by adopting learnable Laplacian Smoothing while smoothing, whereas edge feature augmentation does not affect the performance. Therefore, the careful handling of $x_{j}-x_{i}$ is crucial for achieving good performance.

We investigate the impact of inserting positions (resolutions) of Laplacian Unitsunits. Specifically, only one unit is inserted into each position to analyze the impact of individual positions. The results are shown listed in Table 12. In general, we can still observe performance improvements when only one Laplacian Unit unit is integrated at a certain level. For AW and PG, it seems that more improvements are obtained by inserting into deeper positions, while peak values appear in the shallow and deep levels at the same time for PM. On the other handIn contrast, the improvements are rather limited compared to the Full versionwith the full version, in which units are inserted at all positions. Therefore, the effect of Laplacian Units units can be accumulated throughout from different positions in the network, which verifies our integration strategy.

We use pre-trained models , (i.e., models (S) from the part segmentation task, ) for the following analyses. Since the Laplacian Unit performs non-linear Because the Laplacian unit performs nonlinear filtering on the input, we hypothesize that the behavior of the Laplacian Unit unit can be tracked by monitoring the smoothness of the features. The smoothness [dong2016learning] can be defined as

where $Y_{l}=\{y_{l,i}\}_{i=0}^{n_{l}\times d_{l}}$ is the feature vectors vector of the point cloud in the at level $l$that are normalized into , which are normalized to a unit hypersphere (the maximum norm of the feature vectors is 1one), $W_{ij}\in\mathbb{R}_{x\geq 0}$ denotes the weight of the edge between $y_{l,i}$ and $y_{l,j}$, $L_{l}$ = $D_{l}-W_{l}$ is the graph Laplacian, and $D_{l,ii}=\sum_{j}W_{l,ij}$ is the diagonal weighted degree matrix. The sparse weight matrix $W_{l}$ is a kNN graph ($k=16$ in this study), where $W_{ij}=1$ when there exist exists an edge between point points $i$ and $j$. Because we are interested in the smoothness of the output features with reference to input features, we subsequently define relative smoothness to quantify such a change:

where $Y_{l,in}$ and $Y_{l,out}$ are the features before and after applying the Laplacian Unit in the unit at level $l$. Laplacian Unit The Laplacian unit smooths the input when $RS$ is larger than 1 one, whereas it sharpens the feature when $RS$ is smaller than 1. one.

The $RS$ for each level using the ShapeNet dataset is calculated, and some examples are shown in Fig. 10. The result shows results show that the Laplacian Unit unit adaptively performs smoothing and sharpening adaptively depending on the input classes and levels to which they are integrated. Furthermore, the result suggests results suggest that the networks in general tend to desire smoother features in the encoder levelswhile , whereas sharper features are produced in the decoder levels. We conjecture that the smoother features effectively summarize the coarse shape information from local details in the encoderwhile , whereas the interpolated features are sharpened for point-wise pointwise prediction in the decoder. Moreover, Laplacian Units units for different models perform similarly for some classes, whereas they perform differently in other classes. This suggests that the network types, i.e.that is, local aggregation operations used in the network, influence differently on the behavior of Laplacian Unitsunits differently. The qualitative results are shown in Fig. 11. In the first row, the features with relatively large magnitudes before applying Laplacian Units are smoothedand hence units are smoothed, and hence, the feature distributions appear more uniform. On the other handIn contrast, sharpening makes feature distributions more variable, as isolated extreme features appear in several positions of the shapes after applying Laplacian Unitsunits. Nevertheless, we can observe that the smoothing and sharpening occur simultaneously on at the point level in practice. For instance, the wings of the airplane in the second row of Fig. 11 becomes smoother while isolated become smoother when some points with high responses appear on its the body, which further verifies the flexible functionalities of Laplacian Unitsunits.