Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions

To capture the local and global geometric information of irregular points, we adopt the Point 4D Convolution from P4Transformer [6], but simplify it to apply on the spatial domain only, thus termed as Point 3D Convolution (P3DConv). Specifically, for each interpolated point $p\in P_{I}$ and its associated feature $f$, we first search for its $k$-nearest neighbor $(p_{k},f_{k})$, and calculate the coordinate offset $(\delta_{x},\delta_{y},\delta_{z})$ between them for convolution kernel generation. Then the P3DConv on interpolated point $(p,f)$ is conducted as follow:

where $f^{\prime}$ is the convoluted feature, $K(p)$ is the $k$-nearest neighbor set of point $p$, $\gamma,\alpha,\beta$ all indicate an MLP-based transformation with the same output channel $d$, and $\odot$ represents the Hadamard product.

The detailed structure of our feature extractor is shown in Fig 3. Given an interpolated point cloud $P_{I}\in\mathbb{R}^{N\times 3}$, where $N$ is the number of points, an MLP first projects $P_{I}$ to a higher dimensional space $\mathbb{R}^{N\times d}$, followed by a stack of three dense blocks with intra-block dense connection [13]. Each dense block consists of three convolution groups, followed by a transition down layer. Inside each convolutional group, an MLP reduces the feature dimension, while a P3DConv layer extracts local features. The transition down layer is another MLP that reduces the feature channels and therefore following computational cost. All MLPs for feature extraction share the same output channel of $d$. In the end, a set of multi-scale local features $\{l^{0},l^{1},l^{2},l^{3}\}\in\mathbb{R}^{N\times d}$ is captured. By further applying a max pooling layer, a global feature $g\in\mathbb{R}^{1\times d}$ is obtained.

For any query point $p\in\mathbb{R}^{3}$, its point-to-point distance $F(p)$ is estimated based on the extracted local features $\{l^{0},l^{1},l^{2},l^{3}\}$ and global feature $g$.

To obtain the point-wise local features for each query point $p$, we follow [28] to conduct the feature interpolation, using the inverse distances of three-nearest neighbors in the initial interpolated point cloud as weights. With that, the point-to-point distance $F(p)$ can be estimated as follow:

where $\{l_{p}^{0},l_{p}^{1},l_{p}^{2},l_{p}^{3}\}$ are the interpolated multi-scale local features, $g$ is the global feature, and $\psi$ is a four-layer MLP.

During inference, the extracted local and global features are fixed. However in each iteration, since the points have moved, interpolated features are re-generated, thus new $F(p)$ can be estimated.

Unlike inference, there is no iterative optimization during training. Instead the interpolated points are jittered with Gaussian noise $\mathcal{N}(0,0.02^{2})$ to serve as query points, which simulates varying degrees of displacement in different iterations, and increases the smoothness and continuity of learned distance functions.

We apply L1 loss to minimize the error between the predicted distance $P2PNet(p)$ and the ground truth $F(p)$.

Two public datasets, PU-GAN [16] and PU1K [29] are used for evaluation. We follow the official training/testing splits and settings in original papers, where training is conducted on patch level. Compared to the PU-GAN dataset, PU1K is more challenging because it has a larger volume of data and more diverse categories.

During training, each input low-res patch contains $256$ points, while its corresponding high-res patch has $1024$ points. Thus the upsampling rate $R=4$. During testing, we follow [29] to generate point clouds from the test set of PU-GAN with Poisson disk sampling [46], as it only provides 3D meshes. All testing low-res point clouds from both datasets have $2048$ points, while the high-res counterparts contain $2048\times R$ points. Given the low-res input, we first generate the interpolated point cloud by our midpoint interpolation. Then we follow [29] to extract patches, apply gradient descent to update them, and finally merge them to obtain the full high-res output.

Besides aboved synthetic datasets, we also adopt the real-scanned ScanObjectNN [35] and KITTI [9] datasets for qualitative evaluation.

Following [17, 29, 20], we adopt the Chamfer distance (CD), Hausdorff distance (HD) and point-to-surface distance (P2F) as metrics.

For the PU-GAN dataset, we train PU-Net [45], MPU [43], PU-GAN [16], Dis-PU [17], PU-EVA [20], PU-GCN [29] and Neural Points (NePs) [8] with the default settings in the respective papers as baselines. For the PU1K dataset, we only choose PU-Net [45], MPU [43] PU-GCN [29] and PU-Transformer [29], following original papers.

Tab 1 shows that our method outperforms prior arts on all metrics. In particular, the performance gain on higher upsampling rate ($16\times$) is larger. As shown in Fig 4, previous methods tend to generate outliers caused by overestimation of 3D coordinates. This artifact is more severe as the upsampling rate goes higher. On the contrary, our results have much less outliers, more faithful surfaces and more fine-grained details, without obvious distinction between $4\times$ and $16\times$ upsampling.

In addition, we compare the efficiency of each method under $4\times$ setting, in terms of network parameters (Param.), as well as inference time which is measured end-to-end from loading the input low-res point cloud to generate the full high-res output, using a TITAN X GPU. Note that model size and inference time are not necessarily in proportion, because some of the expensive operations, e.g. $k$-nearest neighbor search, are not part of the network. That said, our method is the fastest with the fewest parameters.

Unlike most of previous methods [45, 16, 43, 17, 29, 30, 19, 32], our approach does not need retraining for different upsampling rates. Similarly, priort art NePs [8] also does not require retraining. Thus we conduct a comparison using the model trained on PU-GAN dataset. For both methods, we only vary the upsampling rate $R$ during inference while fixing all other parameters. Based on Tab 2, our method yields better accuracy under most of the metrics, except the P2F metric with $6\times$ and $7\times$ upsampling. Note that since P2F is asymmetrical, only from upsampled points to the ground truth surfaces but not vice versa[16], CD and HD metrics are more meaningful.

We also conduct the $4\times$ evaluation on the more challenging PU1K dataset, as reported in Tab 3. Our method still outperforms others on almost all metrics, except for the P2F metric, which is second to PU-Transformer [32]. Note that our model is much smaller than PU-Transformer (67.1Kb vs. 969.9Kb).

Using models trained on the PU-GAN dataset, we also conduct experiments on real-scanned point clouds from ScanObjectNN [35] and KITTI [9] datasets, as shown in Fig 5 and Fig 6. Since there is no ground truth high-res point cloud, we only qualitatively compare, and omit some methods that produce consistently worse results. Not only sparse and noisy, scanned data often have small holes or gaps, which makes them even more challenging. Fig 5 shows that our results are more complete, smooth and faithful, while other methods tend to keep the holes. In Fig 6, our result appears to be more complete and with more fine-grained details.

We adopt CurveNet [40] as the classification model, and utilize the same training and testing schema on the ModelNet40 dataset [39]. Specifically, the model is trained with 1024 points. For each testing point cloud, we uniformly subsample 256 points as the low-res input, and upsample them back to 1024 points with various methods (trained on the PU-GAN dataset).

We then compare the classification performance on the downsampled low-res point clouds (Low-res, 256 points), original test set (High-res, 1024 points) and upsampled point clouds (1024 points). As shown in the first two rows of Tab 4, the classification accuracy of the low-res point clouds is observably worse, while our upsampling method brings a significant improvement.

We utilize BallPivoting [1] to reconstruct meshes from the $4\times$ upsampled point clouds (8192 points) of PU-GAN dataset. From the last two rows of Tab 4, we find that the low-res point clouds (Low-res, 2048 points) already achieve a comparable performance, because they are directly sampled from the ground truth meshes. Although the improvement obtained by each method is marginal, our approach still yields the best.

As the point clouds captured by scanners are often noisy, it is necessary to evaluate the robustness of each method against noise. To be specific, we first generate some random noise offline, which is sampled from a standard Gaussian distribution $\mathcal{N}(0,1)$ and multiplied by a factor $\tau$, where $\tau$ denotes the noise level. Then we test on the low-res point clouds of PU-GAN dataset with added noise. And the training of all methods incorporate with Gaussian noise perturbation as augmentation strategy for fair comparison. As shown in Tab 5, our method achieves the best performance consistently, especially at high noise level. And Fig 7 provides the qualitative comparisons, which verifies that our result is cleaner with much less outliers.

Considering all previous evaluations of PU-GAN and PU1K datasets are conducted on the low-res point clouds with 2048 points, we further validate the robustness of our method against different input sizes. As Fig 8 shows, although our model is trained on the fixed-scale input data, it can generalize well to different scales during inference, even when the input is extremely sparse.

Compared with predicting 3D coordinates or residuals, merely regressing the point-to-point distance is a relatively easier task. To verify this point, we modify the output channel of P2PNet’s last MLP to predict the 3D coordinate offset for each interpolated point, and we employ a L2 loss to minimize the prediction error. Following [21, 41, 22], we update the interpolated points to ground truth locations in an auto-regression way:

where $\Delta(p^{t})$ denotes the predicted coordinate displacement, $\lambda$ is the step size, and we fine-tune it and iteration number $T$ to achieve the best performance, holding all other parameters the same. Moreover, we also report the results of end-to-end update, which repeats Eq 5 only once with step size $\lambda=1$.

As reported in Tab 6, our distance regression based method clearly achieves superior performance, although updating points auto-regressively can alleviate the misestimation of predicted coordinate offsets to some extent.

We employ the midpoint interpolation result $P_{I}$ to serve as both input of P2PNet and initial point cloud to be updated. For validating the effectiveness of midpoint interpolation, we replace the input of P2PNet with the low-res point cloud $P_{L}$. And we also sample points from the Gaussian distribution $\mathcal{N}(p,\sigma^{2})$ to get another initial point cloud, denoted as $P_{S}$. Finally, we combine different network inputs and initial point clouds to conduct the experiments, holding all other parameters the same.

From the second and forth rows of Tab 7, we conclude that using the denser interpolated point cloud $P_{I}$ as network input can achieve better performance, since it contains more geometrical information and benefits the feature extraction process. And the improvement is more evident in the comparison between the third and forth rows, it proves that our midpoint interpolation result provides a better initial position, which benefits the update process under the same number of iterations.

The EdgeConv based feature extractor [43] is widely used by previous work [16, 17, 43, 8, 48]. However, EdgeConv [43] utilizes most of parameters to refine each individual feature. While our P3DConv focuses on the feature aggregation achieved by generated convolution kernels, thus benefits the extraction of local and global features. For verifying the superior performance of our P3DConv, we replace the dense block in our P2PNet with EdgeConv in [43], and we also fine-tune the number of EdgeConv layers to achieve the comparable network parameters, as Tab 8 shows.