Gradient-based Point Cloud Denoising with Uniformity

Early point cloud denoising methods [17, 18, 19, 20, 21, 22, 23] are mainly optimization-based, which rely heavily on geometric priors and have difficulty balancing between detail preservation and denoising effectiveness. In recent years, with the advent of point-based neural networks such as PointNet [24], deep-learning-based denoising methods have emerged and achieved promising results. In general, deep-learning-based methods can be roughly divided into three categories: displacement-based [25, 26, 27, 28, 29, 30], downsample-upsample-based [31], and gradient-based [12, 11] methods. PointCleanNet [27] is the pioneer of displacement-based denoising methods, which employs an architecture based on PointNet to estimate the single-step corrective displacement of noisy points. Subsequently, GPDNet [29] enhances the denoising network by graph convolutions to exploit the local structure of the neighborhood. However, the estimation of single-step corrective displacement may not be sufficiently accurate thus these methods generally suffer from shrinkage and outliers. To address these issues, DMRDenoise [31] proposes to explicitly learn the underlying manifold of noisy point clouds from a downsampled subset of points with less noises via a differentiable pooling layer. However, the downsampling stage also discards geometric details and leads to over-smoothing. Recently, gradient-based methods [11, 12] formalize point cloud denoising as an iterative process of increasing the log-likehood of each point via estimating the gradient of the underlying distribution. Compared with displacement-based methods which only consider the position of each input point, gradient-based methods model 3D continuous distribution supported by a 2D manifold. Compared with downsample-upsample method, they preserve more informative details at low noise levels. Although they have outperformed previous approaches, gradient-based methods neglect the relationship among points while iterative denoising, which leads to serious distribution non-uniformity.

In this work, we demonstrate that the distribution non-uniformity stems from the point independence assumption, and propose to extend previous gradient-based methods using UniNet. Our method achieves comparable denoising performance and can significantly achieve distribution uniformity, which largely benefits downstream tasks.

Although distribution uniformity has not received sufficient attention in point cloud denoising, it has been explored in point cloud upsampling. For upsampling, the generated points should be located on the underlying surface and cover the surface with a uniform distribution. PU-Net [15] proposes to distribute the generated points more uniformly on the object surface via a repulsion loss, which penalizes a point if it is too close to its neighborhoods. Subsequently, PU-GAN [14] adopts a chi-squared model to measure the uniformity and proposes the uniform loss to enhance the output point distribution uniformity. The follow-up Dis-PU [13] disentangles the upsampling task based on its multi-objective nature and adopts a spatial refiner network to generate more uniform dense point sets. All these previous methods demonstrate that high-quality point clouds should not only faithfully locate on the underlying surface, but also cover the surface with a uniform distribution. Regretfully, point clouds produced by existing gradient-based denoisers lack such uniformity, which significantly obstructs downstream tasks such as surface reconstruction as illustrated by our experiments.

Given a clean point cloud $\bm{Y}=\{\bm{y}_{i}\}_{i=1}^{n}$ containing $n$ points, each point $\bm{y}_{i}$ can be viewed as a sample from the underlying surface of a 3D object, which can be represented as a 3D distribution $\mathcal{P}$ supported by a 2D manifold. However, point clouds are often perturbed by noises due to the inherent limitation of depth sensors, which can be formalized as applying a degradation operator $\mathcal{D}$ to the clean point cloud $\bm{Y}$. We represent the noisy point cloud as $\bm{X}=\mathcal{D}(\bm{Y})$ and $\mathcal{D}$ is often modeled as an additive noise term $\bm{N}=\{\bm{n}_{i}\}_{i=1}^{n}$ sampled from the noise distribution $\mathcal{N}$, i.e., $\bm{x}_{i}=\bm{y}_{i}+\bm{n}_{i}$.

Let $q(\cdot)$ denote the probability density function of 3D distribution $\mathcal{P}$. Gradient-based denoising methods [11, 12] formalize the process of denoising the input noisy point cloud $\bm{X}$ as finding a transformed point cloud $\widetilde{\bm{X}}=\{\widetilde{\bm{x}}_{i}\}_{i=1}^{n}$ that maximize $q(\bm{X})$. To solve this optimal likelihood problem, previous methods view each point as an independent sample from the point distribution. Therefore, the log-likelihood of a point cloud can be written as the summation of point-wise log-likelihoods:

Existing gradient-based methods aim at learning the first-order derivative of the log-density function to denoise the input point cloud iteratively. Thus, the denoising process can be written as:

where $\bm{X}^{(0)}=\bm{X}$ and $s_{t}$ is the step size at the $t$-th step.

Based on the point independence assumption,the log-likelihood $q(\bm{X})$ is transformed into the summation of $n$ point-wise log-likelihoods $q(\bm{x}_{i})$ to reduce the computational complexity. However, high-quality point clouds should not only be faithfully located on the underlying surface of 3D objects, but also cover the surface with a uniform distribution [13, 14]. As shown in Fig. 3 (a)(b), neglecting local point interactions leads to severe distribution non-uniformity of the generated point clouds.

On the other hand, due to the lack of ground truth for the gradient field $\nabla_{\bm{X}}\sum_{i=1}^{n}\log q(\bm{x}_{i})$, previous methods use the center $\overline{\bm{y}}$ of $K$ nearest points $\bm{y}_{k}$ of the noisy point $\bm{x}_{i}$ to approximate the ground truth gradient for $\bm{x}_{i}$. However, this approximation results in errors that could not be ignored when the curvature is large or points are sparsely distributed in the local region. Errors accumulate with iterative gradient ascent, which exacerbates distribution uniformity.

Based on the above analysis, we argue that the core of solving the distribution non-uniformity problem is to model the interactions among points while iteratively denoising the noisy points, as shown in Fig. 3 (c)(d). However, due to the inherent permutation invariance of point clouds, it is difficult to arrange all points in a specific order and consider previous states when learning the gradient field. To this end, we propose to disentangle the gradient field $\nabla_{\bm{X}}\log q(\bm{X})$ into two parts:

where $\nabla_{\bm{X}}\log g(\bm{x}_{1},...,\bm{x}_{n})$ is a complement of point-wise gradient that aims at capturing feature interactions among points while denoising. Compared with Eq. 2, the iterative denoising process of GPCD++ can be written as:

where $\bm{X}^{(0)}=\bm{X}$. Previous work [13] shows the challenge for a single network to meet both uniformity and proximity-to-surface at the same time. Therefore, in our implementation, the function $g(\cdot)$ takes one-step denoised points ${\bm{x}_{i}^{(t)}}^{\prime}=\bm{x}_{i}^{(t)}+s_{t}\nabla_{\bm{x}_{i}}\log q(\bm{x}^{(t)}_{i})$ as input instead of $\bm{x}_{i}^{(t)}$. The corresponding network is constrained to focus more on distribution uniformity and is easier to train.

As illustrated in Fig. 4, our proposed GPCD++ consists of three main parts, including a feature extraction unit, a gradient field estimation unit, and a lightweight network named UniNet, which is used for modeling feature interactions among points. Given the input noisy point cloud $\bm{X}$, the feature extraction unit produces point-wise features $\bm{H}=\{\bm{h}_{i}\}_{i=1}^{n}$ that encode local geometric information. The gradient field estimation unit takes both the output features $\bm{H}$ and the noisy point cloud $\bm{X}^{(t)}$ at the $t$-th step as input and then predicts the point-wise gradient vectors $\nabla_{\bm{x}}\log q(\bm{x}_{i})$ pointing to the underlying surface of the 3D objects. Finally, UniNet aims at converging the denoised point cloud ${\bm{X}^{(t)}}^{\prime}=\bm{X}^{(t)}+s_{t}\nabla_{\bm{X}}\sum_{i=1}^{n}\log q(\bm{x}^{(t)}_{i})$ towards uniform distribution on the surface. It is worth noting that the denoising backbone can be implemented as any gradient-based denoising network such as Score-based Denosing Network [11] or Point Set Resampling [12]. For completeness, we first briefly introduce the gradient-based network architecture, then describe the proposed UniNet which leverages only two graph convolution layers to achieve distribution uniformity while maintaining proximity-to-surface.

If we train GPCD++ from scratch, the low-quality output point clouds of the denoising backbone lead to an unstable training process of UniNet at the early training stage. Thus, we pretrain the feature extraction unit and the gradient field estimation unit on the same dataset using the loss function $\mathcal{L}_{g}$, which follows previously proposed training objective [11, 12], and then freeze them while training UniNet with the loss function $\mathcal{L}_{u}$.

$\mathcal{L}_{u}$ measures the difference between the output point cloud and the ground truth point cloud. Chamfer Distance (CD) and Earth Mover’s Distance (EMD) are two permutation-invariant loss functions widely used in various tasks on point clouds (e.g., completion, upsampling, denoising). Supposing $\bm{X}^{\prime\prime}=\{\bm{x}^{\prime\prime}_{i}\}_{i=1}^{N}$ and $\bm{Y}=\{\bm{y}_{i}\}_{i=1}^{N}$ denotes the output and ground truth point cloud, the loss functions are defined as:

Here CD directly calculates the average closest point distance between two point clouds $\bm{X}^{\prime\prime}$ and $\bm{Y}$, while EMD finds a bijection $\phi:\bm{X}^{\prime\prime}\rightarrow\bm{Y}$ which minimizes the average distance between corresponding points. Compared with CD, EMD can provide stronger supervision to train UniNet, so we use EMD to regularize the output point clouds for point cloud denoising. Notably, our method does not require other loss terms, since the network trained with EMD is already able to refine the point clouds for proximity-to-surface and distribution uniformity at the same time.

Following previous works [11, 12], we collect 20 meshes from the training subset of PU-Net and use Poisson disk sampling to generate ground truth point clouds ranging from 10k points to 50k points. After normalization into the unit sphere, point clouds are perturbed by Gaussian noise with zero mean and a standard deviation from 0.5% to 2% of the bounding ball radius. Each training point cloud is split into patches before fed into the network to reduce memory use. In training, we use random sampling to sample patches(patch size is 1K).

For testing, we compare our model with state-of-the-art methods on the testing subset of PU-Net [15] and PointCleanNet [27] datasets, which contain 20 meshes and 10 meshes, respectively. Three metrics are employed to evaluate point cloud denoising methods, including Chamfer distance, Point-to-Mesh Distance for denoising effectiveness, and modified uniformity metric [14] for point uniformity. We normalize the denoised point clouds into a unit sphere before computing the metrics. For uniformity metric, we adopt the farthest sampling to pick $M$ seed points, and then crop a point set $S_{j}$ at each seed using ball query. The uniformity metric can be written as:

where $U_{\text{imbalance}}(S_{j})=\frac{(|S_{j}|-\widehat{n})^{2}}{\widehat{n}}$ denotes the deviation of $|S_{j}|$ from $\widehat{n}$. Here $\widehat{n}$ is the expected number of points in $S_{j}$. $U_{\text{clutter}}(S_{j})$ measures the deviation of each point’s distance to its nearest neighbor, denoted by $d_{j,k}$, and is defined as $U_{\text{clutter}}(S_{j})=\frac{1}{|S_{j}|}\sum_{k=1}^{|S_{j}|}\frac{(d_{j,k}-\widehat{d})^{2}}{\widehat{d}}$. $\widehat{d}$ is the expected distance between a point and its nearest neighbor. Compared with original uniform loss, our modified one leverages the ground truth point cloud to compute the expected distance $\widehat{d}$ and expected number $\widehat{n}$, which provides a more accurate estimation, especially for objects with complex geometric shape.

To demonstrate the generalization, we test our proposed model with point clouds perturbed by several types of noise, including isotropic Gaussian noise, Laplace noise, discrete noise, anisotropic Gaussian noise, uni-directional Gaussian noise, uniform noise, simulated sensor noise and real noise. The following are the setting of corresponding hyperparameters, where the scale parameter $s$ is set to 1%, 2% and 3% of the bounding sphere radius to simulate different levels of noise.

• •

  Isotropic Gaussian noise.

  $\displaystyle p(x;s)=\frac{1}{\sqrt{2\pi}s}e^{-\frac{x^{2}}{2s^{2}}}$

• •

  Laplace noise.

  $\displaystyle p(x;s)=\frac{1}{2s}e^{-\frac{|x|}{s}}$

• •

  Discrete noise.

  $\displaystyle p(x;s)=\left\{\begin{matrix}0.1&(\pm s,0,0),(0,\pm s,0),(\pm 0,0,\pm s)\\ 0.4&(0,0,0)\\ 0&\text{otherwise}\end{matrix}\right.$

• •

  Anisotropic Gaussian noise.

  	$\displaystyle x$	$\displaystyle\sim\mathcal{N}(0,\Sigma)$
  	$\displaystyle\text{where}\;\;\Sigma$	$\displaystyle=s^{2}\times\begin{bmatrix}1&-\frac{1}{2}&-\frac{1}{4}\\ -\frac{1}{2}&1&-\frac{1}{4}\\ -\frac{1}{4}&-\frac{1}{4}&1\end{bmatrix}$

• •

  Uni-directional Gaussian noise. We only perturb the point clouds along x-axis using Gaussian noise.

• •

  Uniform noise.

  $\displaystyle p(x;s)=\left\{\begin{matrix}\frac{3}{4\pi s^{3}}&||x||_{2}\leq s\\ 0&\text{otherwise}\end{matrix}\right.$

• •

  Simulated sensor noise. We adopt the Blensor [35] simulation package to simulate realistic noise.

• •

  Real noise. We test our method on the real-world dataset Paris-rue-Madame [36], which is obtained from real laser scanner.

In the training phase, we perform data augmentation on the fly by randomly scaling and rotating the input point clouds. We adopt Adam optimizer with an initial learning rate $2e-4$ to train UniNet. The learning rate is multiplied by $0.8$ on epoch 30, 60 and 90.

In the testing phase, for uniformity metric, the number of seed points $M$ is set as $rN$, where $N$ is the number of points and $r$ is $0.05$. We crop each set $S_{j}$ with radius $r_{d}=\sqrt{p}$ for each $p\in\{$ 0.4%, 0.6%, 0.8%, 1.0% $\}$, and compute Eq. 7 four times, then sum up the results.

Following previous works [11, 12], we first perturb the point clouds with isotropic Gaussian noise to compare our models with state-of-the-art point cloud denoising methods. They include both optimization-based methods such as bilateral filtering (Bilateral)  [17], jet fitting (Jet)  [19], MRPCA [22], GLR  [37], and learning-based methods including Point Clean Net (PCN)  [27], GPDNet [29], DMRDenoise (DMR)  [31], Score-based Denoising Network (Score-based)  [11], Point Set Resampling via Gradient Field (PSR)  [12]. Notably, the last two methods are gradient-based methods, both of which can be employed as the backbone in our proposed framework GPCD++, thus we implement two versions of GPCD++, named GPCD++(Score) and GPCD++(PSR), to make a comparison.

The experimental results are summarized in Tab. I, where we adopt the results of previous works reported in Point Set Resampling [12] following the same evaluation protocol. We observe noticeable performance gains from not using UniNet in GPCD++(Score) in the first few steps of gradient descent iteration, thus we activate UniNet after 20 steps in GPCD++(Score). It is worth noting that our proposed GPCD++(PSR) surpasses previous state-of-the-art methods on both benchmark datasets, with an obvious improvement on CD and P2M metrics. Compared with Score-based Denoising Network and Point Set Resampling, the corresponding GPCD++ versions almost achieve a consistent improvement on Chamfer Distance while taking both distribution uniformity and proximity-to-surface into consideration, which demonstrates the effectiveness of UniNet. We also note that the performance of GPCD++ on P2M metrics is slightly worse than that of its backbone network in a few cases. The reason is that EMD adopts a point-to-point bijection function $\phi$ to regularize each point and sometimes the correspondence is incorrect. UniNet moves the point $x$ away from the underlying surface and slightly degrades the performance on P2M metrics.

To explore whether our proposed UniNet improves the distribution uniformity of the output point clouds, we compare GPCD++ with its corresponding backbone on PU dataset using uniformity metric. As shown in Tab. II, GPCD++ significantly outperforms its backbone network under different levels of noise on uniformity metric, while slightly affecting denoising in a few cases. With regard to Chamfer Distance, GPCD++ can generate point clouds of higher quality for downstream tasks.

Tab. III illustrates the quantitative results of our proposed GPCD++ under various types of noise, including Laplace noise, discrete noise, anisotropic Gaussian noise, uni-directional Gaussian noise and uniform noise. We take the results reported by Point Set Resampling [12] with the identical evaluation protocol. It is worth noting that we only train our model using Gaussian noise without further fine-tuning. The experimental results demonstrate our model has better generalization capability than previous methods under unseen noise models.

We also test on simulated noise (with ground truth mesh) of a real sensor provided by Blensor [35] package and the results are shown in Tab. IV. GPCD++(Score) and GPCD++(PSR) have only 10.7% and 7.1% more parameters, respectively, than the corresponding backbones. For inference time, GPCD++(PSR) has a consistent and significant improvement over Point Set Resampling with only 16.6% more inference time on average. Notably, GPCD++(Score) only adds neglectable inference time compared with Score-based Denoising Network, a patch-based denoising method that also requires time to split the noisy input into patches. In summary, our method outperforms previous methods with minimal cost regarding inference time and model size.

In particular, another alternative solution is to run sequentially the gradient-based denoiser and then a method to achieve distribution uniformity. To evaluate in this setting, we choose PSR as the denoiser and adopt Graph Laplacian (GL) [38] to resample the denoised point cloud. As shown in Tab. V, this “denoise then resample” baseline slightly improves Chamfer Distance, but brings additional noise (see P2M) and significant computational overhead.

We conduct qualitative studies on PU dataset and then visualize the denoising results generated by our proposed GPCD++(PSR) and previous state-of-the-art methods. Due to the limit of space, we only compare our proposed methods with Score-based Denoising Network and Point Set Resampling for the following reasons, (1) they have a significantly better performance on PU dataset than other methods, (2) they are gradient-based methods closely related to our method. Specifically, the input point cloud is perturbed by isotropic Gaussian noise with a scale parameter 3%, and perturbed by simulated sensor noise with a scale parameter 1%. As seen in Fig. 6, although both Score-based Denoising Network and Point Set Resampling successfully converge each noisy point towards the underlying surface, the denoised point clouds suffer from distribution non-uniformity. Our method can effectively resolve this problem and produce higher-quality point clouds.

Fig. 7 shows the denoising results of various methods on the real-world dataset Paris-rue-Madame [36]. It is worth noting that our method can not only generate clean point clouds from the noisy input, but also preserve subtle structures and achieve distribution uniformity.

To study the impact of model size and the neighborhood size, we perform two experiments on the PU dataset. The number of nearest neighbors in KNN decides the size of local regions from which each point aggregates information. Tab. VI shows the results of our proposed GPCD++(PSR) with different hyperparameters, where $K$ is the number of nearest neighbors in KNN and $L$ is the number of dense edge convolution layers. We observe no significant improvement when $K$ is larger than 8, which means UniNet only exploits local information to refine the input point clouds. The number of convolution layers decides how we reach a balance between representational power and model size. Although the larger models perform better, the improvements are moderate. Therefore, for a better trade-off between performance and model size, we choose to use $L=2$ in our implementation.

We observe noticeable performance gains from not using UniNet in the first few steps of GPCD++(Score), so we evaluate our proposed GPCD++(Score) and GPCD++(PSR) under different activation steps. As illustrated in Fig. 8(a), GPCD++(Score) achieves the best performance when the activation step is set to be 20. In contrast, Fig. 8(b) shows that introducing UniNet into the gradient descent iteration of GPCD++(PSR) has a consistent improvement, and the improvement gets smaller when the activation step is larger. We assume that the difference stems from denoising efficiency of the backbone network in GPCD++. Compared with Score-based denoising net, PSR can produce high-quality denoised point clouds on early iterations, thus UniNet in GPCD++(PSR) is activated from the first step.