Score-Based Point Cloud Denoising

Prior to the emergence of deep-learning-based denoising, the point cloud denoising problem is often formulated as an optimization problem constrained by geometric priors. We classify them into four categories: (1) Density-based methods are most relevant to ours as they also involve modeling the distribution of points. [36] uses the kernel density estimation technique to approximate the density of noisy point clouds. Then, it removes outlying points in low-density regions. To finally obtain a clean point cloud, it relies on the bilateral filter [9] to reduce the noise of the outlier-free point cloud. Therefore, this method focuses on outlier removal. (2) Local-surface-fitting-based methods approximate the point cloud with a smooth surface using simple-form function approximators and then project points onto the surface [1]. [9, 4, 16, 6] proposed jet fitting and bilateral filtering that take into account both point coordinates and normals. (3) Sparsity-based methods first reconstruct normals by solving an optimization problem with sparse regularization and then update the coordinates of points based on the reconstructed normals [2, 31, 33]. The recently proposed MRPCA [22] is a sparsity-based denoiser which has achieved promising denoising performance. (4) Graph-based methods represent point clouds on graphs and perform denoising using graph filters such as the graph Laplacian [28, 37, 12, 14, 13]. Recently, [37] proposed graph Laplacian regularization (GLR) of a low-dimensional manifold model for point cloud denoising, while [12] proposed a paradigm of feature graph learning to infer the underlying graph structure of point clouds for denoising. To summarize, optimization-based point cloud denoising methods rely heavily on geometric priors. Also, there is sometimes a trade-off between detail preservation and denoising effectiveness.

The advent of point-based neural networks [24, 25, 32] has made deep point cloud denoising possible. The majority of existing deep learning based methods predict the displacement of each point in noisy point clouds using neural networks, and apply the inverse displacement to each point as illustrated in Figure 2(a). PointCleanNet (PCNet) [26] is the pioneer of this class of approaches, which employs a variant of PointNet as its backbone network. [23] proposed GPDNet, which uses graph convolutional networks to enhance the robustness of the neural denoiser. [11] proposed an unsupervised point cloud denoising framework—Total Denoising (TotalDn). In TotalDn, an unsupervised loss function is derived for training deep-learning-based denoisers, based on the assumption that points with denser surroundings are closer to the underlying surface. The aforementioned displacement-prediction methods generally suffer from two types of artifacts: shrinkage and outliers, as a result of inaccurate estimation of noise displacement. Instead, [21] proposed to learn the underlying manifold (surface) of a noisy point cloud for reconstruction in a downsample-upsample architecture as illustrated in Figure 2(b). However, although the downsampling stage discards outliers in the input, it may also discard some informative details, leading to over-smoothing.

In this work, we propose a novel framework that distinguishes significantly from the aforementioned methods. Our method is motivated by the distribution model of noisy point clouds. It denoises point clouds via gradient ascent guided by the estimated gradient of the noisy point cloud’s log-density as illustrated in Figure 2(c). Our method is shown to alleviate the artifacts of shrinkage and outliers, and achieve significantly better denoising performance.

Score matching is a technique for training energy-based models—a family of non-normalized probability distributions [18]. It deals with matching the model-predicted gradients and the data log-density gradients by minimizing the squared distance between them [17, 30]. Our proposed training objectives are similar to the score matching technique. The score matching technique in generative modeling aims at approximating unconditional distributions about data (e.g., images), while our model estimates the noise-convolved distribution of points.

Score matching has been applied to developing generative models for 3D shapes. [3] proposed an auto-encoder architecture ShapeGF that also has a score-estimation network which served as a decoder. However, ShapeGF is different from our model in at least the following three aspects. First, ShapeGF is designed for 3D point cloud generation and models the noise-free 3D distribution $p$, while our method models the noise-convolved distribution $p*n$ and aims at denoising the point cloud based on the score of $p*n$. Second, since ShapeGF is a general auto-encoder for 3D shapes, it does not have the generalizability to out-of-distribution shapes. For instance, when trained on the ShapeNet dataset [5], it can hardly generalize to shapes beyond the categories in ShapeNet. In contrast, our model is generalizable to arbitrary 3D shapes because our score function is defined on a local basis, which learns the building blocks of 3D shapes rather than the entire shapes themselves. This way narrows down the latent space of 3D geometry representation and makes it possible for the network to learn and reconstruct 3D details. Third, to recover 3D shapes, ShapeGF requires a latent code of the shape obtained from the encoder, but their encoder is not meant to learn representations for denoising or other detail-demanding tasks.

Given a noisy point cloud ${\bm{X}}=\{{\bm{x}}_{i}\}_{i=1}^{N}$ consisting of $N$ points as input, we model the underlying noise-free point cloud as a set of samples from a 3D distribution $p$ supported by 2D manifolds, and assume the noise follows a distribution $n$. Then the distribution of the noisy point cloud can be modelled as the convolution between $p$ and $n$, denoted as $p*n$.

In order to denoise the noisy input ${\bm{X}}$, we propose to estimate the score of the noise-convolved distribution $p*n$, i.e., $\nabla_{\bm{x}}\log[(p*n)({\bm{x}})]$—the gradient of the log-probability function, only from ${\bm{X}}$. Then, we denoise ${\bm{X}}$ using the estimated scores of $p*n$ via gradient ascent, thus moving noisy points towards the mode of the distribution that corresponds to the underlying clean surface. The implementation of the proposed method mainly consists of the following three parts:

1. 1.

   The score estimation network that takes noisy point clouds as input and outputs point-wise scores $\nabla_{\bm{x}}\log[(p*n)({\bm{x}}_{i})](i=1,\ldots,N)$ (Section 3.2).

2. 2.

   The objective function for training the score estimation network (Section 3.3).

3. 3.

   The score-based denoising algorithm that leverages on the estimated scores to denoise point clouds (Section 3.4).

Given a noisy point cloud ${\bm{X}}=\{{\bm{x}}_{i}\}_{i=1}^{N}$ as input, the score estimation network predicts $\nabla_{\bm{x}}\log[(p*n)({\bm{x}}_{i})]$ for each point in ${\bm{X}}$. We estimate the score for each point ${\bm{x}}_{i}$ on a local basis, i.e., the network aims at estimating the score function in the neighborhood space around ${\bm{x}}_{i}$, denoted as ${\mathcal{S}}_{i}({\bm{r}})$. Localized score functions are fundamental to the model’s generalizability because in this way the model focuses on the basic fragments of 3D shapes rather than the entire shapes themselves, narrowing down the latent space of 3D geometry representation.

The estimation of ${\mathcal{S}}_{i}({\bm{r}})$ is realized by a neural network which consists of a feature extraction unit and a score estimation unit. The feature extraction unit produces features that encode both the local and non-local geometry at each point. The extracted features are subsequently fed as parameters into the score estimation unit to construct score functions.

We denote the input noisy point cloud as ${\bm{X}}=\{{\bm{x}}_{i}\}_{i=1}^{N}$ and the ground truth noise-free point cloud as ${\bm{Y}}=\{{\bm{y}}_{i}\}_{i=1}^{N}$. Using the ground truth ${\bm{Y}}$, we define the score for some point ${\bm{x}}\in{\mathbb{R}}^{3}$ as follows:

where $\operatorname{NN}({\bm{x}},{\bm{Y}})$ returns the point nearest to ${\bm{x}}$ in ${\bm{Y}}$. Intuitively, ${\bm{s}}({\bm{x}})$ is a vector from ${\bm{x}}$ to the underlying surface.

The training objective aligns the network-predicted score to the ground truth score defined above:

where ${\mathcal{N}}({\bm{x}}_{i})$ is a distribution concentrated in the neighborhood of ${\bm{x}}_{i}$ in ${\mathbb{R}}^{3}$ space. Note that, this objective not only matches the predicted score on the position of ${\bm{x}}_{i}$ but also matches the score on the neighboring areas of ${\bm{x}}_{i}$ as illustrated in Figure 3. This is important because a point moves around during gradient ascent, which relies on the score defined on the neighborhood of its initial position. Such definition of training objective also distinguishes our method from previous displacement-based methods [26, 23], as the objectives of those methods only consider the position of each point while our proposed objective covers the neighborhood of each point.

The final training objective is simply an aggregation of the objective for each local score function:

Given a noisy point cloud ${\bm{X}}=\{{\bm{x}}_{i}\}_{i=1}^{N}$ as input, we first need to construct the local score function ${\mathcal{S}}_{i}$ for each point ${\bm{x}}_{i}\in{\bm{X}}$. Specifically, we first feed the input point cloud ${\bm{X}}$ to the feature extraction unit to obtain a set of point-wise features $\{{\bm{h}}_{i}\}_{i=1}^{N}$. Next, by substituting ${\bm{x}}_{i}$, ${\bm{h}}_{i}$ and some 3D coordinate ${\bm{x}}\in{\mathbb{R}}^{3}$ into Eq. 1, we obtain ${\mathcal{S}}_{i}({\bm{x}})$ as the estimated score at ${\bm{x}}$.

In principle, we can solely use ${\mathcal{S}}_{i}$ to denoise ${\bm{x}}_{i}$. However, to enhance the robustness and reduce the bias of estimation, we propose the ensemble score function:

where $k\operatorname{NN}({\bm{x}}_{i})$ is ${\bm{x}}_{i}$’s $k$-nearest neighborhood.

Finally, denoising a point cloud amounts to updating each point’s position via gradient ascent:

where $\alpha_{t}$ is the step size at the $t$-th step. We suggest two criteria for choosing the step size sequence $\{\alpha_{t}\}_{t=1}^{T}$: (1) The sequence should be decreasing towards 0 to ensure convergence. (2) $\alpha_{1}$ should be less than 1 and not be too close to 0, because according to Eq. 2, the magnitude of the score is approximately the distance from each point to the underlying surface (approximately the length of ${\bm{s}}({\bm{x}})$ in Eq. 2). Thus, performing gradient ascent for a sufficient number of steps with a proper step size less than 1 is enough and avoids over-denoising.

It is worth noting that, unlike some previous deep-learning-based denoisers such as PCNet [26] and TotalDn [11] that suffer from shape shrinkage, we do not observe any shrinkage induced by our method. Thus, we have no need to post-process the denoised point clouds by inflating them slightly as in those works. This shows that our method is more robust to shrinkage compared to previous ones.

To begin with, we consider the distribution of a noise-free point cloud ${\bm{Y}}=\{{\bm{y}}_{i}\}_{i=1}^{N}$ as sampled from a 3D distribution $p({\bm{y}})$ supported by 2D manifolds. Since $p({\bm{y}})$ is supported on 2D manifolds, it is discontinuous and has zero support in the ambient space, i.e., $p({\bm{y}})\rightarrow\infty$ if ${\bm{y}}$ exactly lies on the manifold, otherwise $p({\bm{y}})=0$.

Next, we consider the distribution of noisy point clouds. A noisy point cloud can be denoted as ${\bm{X}}=\{{\bm{x}}_{i}={\bm{y}}_{i}+{\bm{n}}_{i}\}_{i=1}^{N}$, where ${\bm{n}}_{i}$ is the noise component from a distribution $n$. Here, we assume that the probability density function $n$ is continuous and has a unique mode at 0. These assumptions are made for analysis. We will show by experiments that in some cases where the assumptions do not hold, the proposed method still achieves superior performance (see Section A in the supplementary material). Under the continuity assumption of $n$, the density function of the distribution with respect to ${\bm{x}}_{i}$ can be expressed as a convolution of $p$ and $n$:

It can be shown by taking the derivative of both sides that the noise-free point cloud ${\bm{Y}}$ from the noise-free distribution $p$ exactly lies on the mode of $q$ if the mode of $n$ is 0. When the assumption of uni-modality holds, $q({\bm{x}})$ reaches the maximum on the noise-free manifold.

Suppose the density function $q({\bm{x}})$ is known. Based on the above analysis, denoising a point cloud ${\bm{X}}=\{{\bm{x}}_{i}\}_{i=1}^{N}$ amounts to maximizing $\sum_{i}\log q({\bm{x}}_{i})$. This can be naturally achieved by performing gradient ascent until the points converge to the mode of $q({\bm{x}})$. The gradient ascent relies only on the score function $\nabla_{\bm{x}}\log q({\bm{x}})$—the first-order derivative of the log-density function. As shown in the previous subsection, $q({\bm{x}})$ reaches the maximum on the underlying manifold under some mild assumptions. Hence, the vector field $\nabla_{\bm{x}}\log q({\bm{x}})$ consistently heads to the clean surface as demonstrated in Figure 4.

However, the density $q$ is unknown during test-time. Instead of estimating $q$ from noisy observations, we only need the gradient of $\log q$ during the denoising, which is more tractable. This motivates the proposed model—score-based denoising.

The training objective defined in Eq. 3 matches the predicted score to the ground truth score function. The magnitude of the estimated score may not exactly equal to that of the real score function. However, this is not an issue in the denoising task, since as long as the directions of estimated gradients are accurate, the points will converge to the underlying surface with sufficient number of steps at a suitable step size of gradient ascent.

We first use isotropic Gaussian noise to test our models and baselines. The standard deviation of noise ranges from 1% to 3% of the shape’s bounding sphere radius. As presented in Table 1, our model significantly outperforms previous deep-learning-based methods in all settings and, surpasses optimization-based methods in the majority of cases.

Although the model is trained with only Gaussian noise, to test its generalizability, we use a different noise type—simulated LiDAR noise. Specifically, we use a virtual Velodync HDL-64E2 scanner provided by the Blensor simulation package [10] to acquire noisy point clouds. The scanning noise level is set to 1% following [21]. The results in Table 2 indicate that although our denoiser is trained using Gaussian noise, it is effective in generalizing to unseen LiDAR noise and outperforms previous methods.

Other noise models, including non-isotropic Gaussian noise, uni-directional noise, Laplace noise, uniform noise, and discrete noise are also used to evaluate our method and baselines. In most of these experimental settings, our model outperforms competing baselines. The detailed results are included in the supplementary material.

Figure 5 shows the denoising results from the proposed method and competitive baselines under Gaussian noise and simulated LiDAR noise. The color of each point indicates its reconstruction error measured by point-to-mesh distance introduced in Section 5.1. Points closer to the underlying surface are colored darker, otherwise colored brighter. As can be observed in the figure, our results are much cleaner and more visually appealing than those of other methods. Notably, our method preserves details better than other methods and is more robust to outliers compared to other deep-learning-based methods such as PCNet and DMRDenoise.

Further, we conduct qualitative studies on the real-world dataset Paris-rue-Madame [29]. Note that, since the noise-free point cloud is unknown for real-world datasets, the error of each point cannot be computed and visualized. As demonstrated in Figure 6, our denoising result is cleaner and smoother than that of PCNet, with details preserved better than DMRDenoise.

In addition, we present a denoising trajectory in Figure 7, which reveals the gradient ascent process of our method—noise reduces as points gradually converge to the mode of $p*n$.

More visual results regarding synthetic noise and real-world noise are provided in the supplementary material.

In summary, the demonstrated qualitative results are consistent with the quantitative results in Section 5.2, which again validates the effectiveness of the proposed method.

We perform ablation studies to assess the contribution of the proposed method’s main designs:

(1) Score-based denoising algorithm We replace the gradient ascent rule (Eq. 6) by directly adding the predicted score to the input coordinates, which is similar to end-to-end displacement-based methods:

We also apply this update rule iteratively following previous displacement-based methods [26, 11]. The number of iterations is fine-tuned to produce the best performance.

(2) Neighborhood-covering training objective We replace the objective in Eq. 3 with:

which is similar to the L2 objective [26] or the Chamfer distance [21, 23] employed in previous deep-learning-based models [26], considering only the position of ${\bm{x}}_{i}$, while ours covers the neighborhood of ${\bm{x}}_{i}$.

(3) Ensemble score function We replace the ensemble score function in Eq. 5 with the single score function ${\mathcal{S}}_{i}({\bm{x}})$.

As shown in Table 3, all the components contribute positively to the denoising performance. More results and analysis of the ablation studies can be found in the supplementary material.

Going beyond denoising, we show that the proposed method is applicable to point cloud upsampling. In particular, given a sparse point cloud with $N$ points as input, we perturb it with Gaussian noise independently for $r$ times, leading to a noisy dense point cloud consisting of $rN$ points. Subsequently, we feed the noisy dense point cloud to our denoiser to acquire the final upsampled point cloud.

We compare the denoising-based upsampling method with the classical upsampling network PU-Net [35] using the test-set of PU-Net. The quantitative results are presented in Table 4 and the qualitative comparison is shown in Figure 8. We see that the denoising-based upsampling method fairly outperforms PU-Net which is specialized in upsampling. This implies that the proposed score-based method for point clouds has the potential in tasks beyond denoising, which will be further explored as our future works.