View Synthesis with Sculpted Neural Points

We use an MVS network (Ma et al., 2022) to extract a dense depth map $\{D_{1},\dots D_{m}\}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}}$ for each of the reference views. Each depth map is un-projected into a set of 3D points by applying the inverse projection in Eqn. 1. The points from all depth maps are combined, without any filtering, to form a larger set of original 3D points $P_{o}=\{p_{1},...,p_{N}\}$. We associate point $p_{i}\in\mathbb{R}^{3}$ with a learnable $K$-dimensional feature vector $f_{i}\in\mathbb{R}^{K}$ and a scalar $o_{i}\in[0,1]$ representing its opacity.

Given a featurized 3D point cloud and a target view, we use a differentiable rendering function with learnable parameters to map the point cloud into an RGB image. For each scene individually, we learn the parameters of this rendering function together with the point features through gradient descent to minimize photometric errors against the input images.

Spherical Harmonic Point Feature We use the spherical harmonics (SH) functions to model the view-dependent effects. Recently Yu et al. (2021b; a) brings up the attention of using SH in neural rendering. Unlike previous works that use SH directly in RGB space, we propose to use SH in high-dimensional feature space, where each element of a vector is modulated by a set of SH coefficients.

We use the SH basis up to degree $2$, which yields $9$ basis in total. This choice follows Yu et al. (2021b) and we find it sufficient to capture highly non-Lambertian surfaces in our experiments. For a 3D point $p_{i}$, the SH layer takes its feature vector $f_{i}\in\mathbb{R}^{K}$ and a target view direction $v$ as input, and outputs a modulated feature vector $s_{i}\in\mathbb{R}^{\frac{K}{9}}$. Specifically, we first compute the SH basis according to $v$, yielding a basis vector $b_{v}\in\mathbb{R}^{9\times 1}$. We then reshape $f_{i}$ into $f_{i}^{\prime}\in\mathbb{R}^{\frac{K}{9}\times 9}$, and finally compute $s_{i}$ with a dot product $s_{i}=f_{i}^{\prime}\cdot b_{v}$. Note that evaluating SH functions is cheap as it avoids complex matrix multiplication operations. We find that it leads to better performance and faster rendering speed compared to the MLP parameterization used in NeRF, as shown in Sec. 5.2.

Differentiable Soft Rasterization We use soft rasterization proposed in Liu et al. (2019); Lassner & Zollhofer (2021) to convert the view-dependent features into a 2D feature map $F$ given a target view. Soft rasterization blends the features of multiple points hit by a camera ray with weights depending on their depth and opacities. We refer the readers to Pulsar (Lassner & Zollhofer, 2021) for details.

Note that in addition to updating $f_{i}$, we also compute the gradient of the photometric loss w.r.t. the point positions $p_{i}$ and opacity $o_{i}$, and optimize them through gradient descent, following Lassner & Zollhofer (2021), which we show helpful in improving fine geometric details in our experiments.

Point Dropout Layer We find that the existing point rendering pipeline is prone to over-fitting, i.e., the image quality on test views is much worse than on training views. The reasons are two-fold: 1. The “training set” for view synthesis consists of only tens of images, and there are barely any data augmentation techniques that can be applied except random cropping. 2. Some points are covered by their neighbors in training views, but get visible in test views. The features for these points are not well-optimized. The blending weights are very low for these points when the rasterizer is “soft”.

To resolve the above issue, we propose to use a “Point Dropout Layer” before rasterization. In each forward pass, we randomly select a subset of points to feed into the rasterizer, whose size depends on the dropout rate $p_{d}$. Note that at inference time, we cannot simply rasterize all points and multiply the output by $p_{d}$ as in the neural network (NN) case, because the rasterization operation is non-linear in contrast to the matrix multiplication in NN. Since it is impossible to traverse all subsets, we simply rasterize $L$ multiple random subsets and average the output feature maps at inference time. When rendering videos, we find that sampling independently for each frame causes obvious flickering artifacts. Therefore, we use the same subsets across all frames, which leads to a better consistency.

Intuitively, the point dropout layer allows us to train on an ensemble of point clouds, and give all points a chance to get optimized even if they are covered, and thus alleviating the over-fitting problem. As a side effect, we also gain a speed-up because fewer points get rasterized. Although the design is simple, this idea has not been explored in previous works, and our experiments show that it significantly improves the image quality on test views.

2-D Rendering without BatchNorm Given a target view, we convert the 2D feature map $F$ into the RGB image $I_{t}$ with a 2D ConvNet. We use a U-Net (Ronneberger et al., 2015) with two downsampling layers and two upsampling layers, and optionally one more upsampling layer to produce high-resolution outputs. The intuition behind using a U-Net is that it can remove noise in the feature map. We use a dropout layer and relatively small point radius, leaving the rasterized feature map with tiny holes, which makes such denoising necessary. The large receptive field of U-Net is favorable for denoising.

Previous works (Aliev et al., 2020; Rakhimov et al., 2022) directly use the original U-Net design with BatchNorm (Ioffe & Szegedy, 2015) layers, which we find unsuitable for the view synthesis task for two reasons. First, the small training set size makes the estimation of the moving average in BatchNorm unstable. Second, the benefit of accelerated training is minimal since the network is shallow. Therefore, we use no normalization layer in our U-Net.

The MVS system we use produces a dense depth map for each input image. Like other depth-based systems (Yao et al., 2018; Yan et al., 2020; Chen et al., 2020), it adopts a fusion step that merges the depth maps from different views into a final point cloud. A geometry consistency check is often used to remove outlier points. Using the depth maps, the consistency check projects a pixel into another view, reprojects the corresponding point back, and sees if the original pixel is recovered up to a threshold. For example, COLMAP (Schönberger & Frahm, 2016) defines the consistency error $\psi^{i,j}_{\bm{p}}$ between view $i$ and $j$ for pixel $\bm{p}$ as:

where $\bm{q}$ is $\bm{p}$’s corresponding point in view $j$. Yan et al. (2020) further propose an improved version, Dynamic Consistency Checking (DCC), which achieved the state-of-the-art filtering results. The main problem of this type of forward-backward consistency check is that it tends to be over-aggressive in filtering out points, resulting in highly incomplete geometry that is detrimental to view synthesis. In datasets such as DTU and LLFF, many areas are only visible in a small number of views. Those areas can easily be filtered out by this check as no confident match could be found.

Therefore, we take the raw depth maps from the multiview stereo system and propose a new technique for our own consistency checking geared toward view synthesis. We check only the forward consistency to maximize completeness while still removing outliers. Formally, a pixel $\bm{p}$ in view $i$ passes the check if and only if $\bigcap\limits_{j=1}^{m}\left[D^{j}(\Pi^{-1}(\bm{p},d_{\bm{p}}^{i},C^{i}))\geq\delta_{d}\cdot d_{\bm{q}}^{j}\right]$, where $\bm{q}$ is $\bm{p}$’s corresponding point in view $j$ (same as in Eqn. 2), $d_{\bm{q}}^{j}$ is the predicted depth of $q$ in view $j$, $D^{j}(\cdot)$ is the depth of a point in view $j$ (the $z$ value of a 3D point in camera $j$’s coordinate), and $\delta_{d}$ is a hyperparameter controlling the relative tolerance.

Intuitively, our point pruning method keeps a point as long as it is not significantly closer than the original surface to any reference view camera. It filters out the points that are floating in the free space between the actual surface and the camera, which are likely to be outliers. It also keeps all points that are only visible in a small number of views. Although the position of such points may not be accurate, it is useful to keep them as candidates for further optimization.

As Fig. 2 shows, after pruning, the point cloud can have holes, either due to points being pruned or incorrect depth estimates (e.g. depth estimates that are close to infinity or zero). Previous works (Lassner & Zollhofer, 2021; Yifan et al., 2019) tackled this problem by performing gradient-based updates to the point locations. However, such updates are limited to local changes of existing points and are unable to recover large areas of missing geometry.

We thus introduce a technique to add new points to the pruned point cloud. The basic idea is to find a set of 3D points that, if added to the point cloud, could help minimize the photometric error after optimization of the point features. Note that these new points do not need to be perfect; they just need to be a superset of the ground truth geometry, because the extraneous points can get optimized through the subsequent gradient-based optimization. On the other hand, an excessive number of new points can lead to overfitting and slower rendering, so a good balance is needed. Our point adding algorithm consists of two steps:

• •

  Optimizing with existing points: We optimize the features and opacity of the current points through gradient descent until convergence. For the $i$-th input image, we extract an error map between the rendered and ground-truth image: $E_{i}=||I^{gt}_{i}-I^{render}_{i}||_{1}$. Note that we use $f_{i}\in\mathbb{R}^{27}$, which is converted to $s_{i}\in\mathbb{R}^{3}$ by the SH layer. $s_{i}$ is directly treated as RGB values during rasterization, and we use no U-Net in this step, as the U-Net hallucination makes $E_{i}$ less informative.

• •

  Proposing new points: For a pixel $(u,v)$ in an input view $i$, we check if its rendering error $E_{i}(u,v)$ is bigger than a pre-defined threshold $\delta_{e}$. If so, we sample 3D points uniformly along the ray of the pixel within the bounds of the scene, and search for points that do not occlude any of the existing points in any of the input views. If multiple such points exist, we propose the closest $M$ points, where $M$ is a hyperparameter. We go through all pixels in all input views, collect all the proposed points, and add them to the existing point cloud.

The design of this algorithm builds upon the assumption that our rendering pipeline can approximate the radiance of each point arbitrarily well on the input images and that any high rendering error can only be caused by incomplete geometry, as those areas having no points covered can only take the default background color. Based on this assumption, we propose new points for pixels with high rendering errors, but exclude points that occlude the existing surface in other views. We propose up to $M$ closest points and choose $M=5$ in our experiments to strike a good balance between geometry coverage and rendering speed. We can alternate between gradient-based optimization and point adding for multiple rounds, but in practice we find one round of point adding to be sufficient for good results. We present the full details of the point adding algorithm in Appendix C.

Results on DTU The quantitative comparison is presented in Tab. 1. Results show that our model has better SSIM and LPIPS, and slightly worse PSNR compared to NeRF. We present the visualizations in Fig. 13, Appendix F. We claim to use LPIPS as the major quality metric, as we find that PSNR and SSIM may not reflect actual visual quality because they are highly sensitive to small pixel shifts (See Appendix. A).

Results on LLFF The quantitative results are shown in Tab. 1. Similar to DTU, our method achieves consistently better SSIM and LPIPS, while being slightly worse in PSNR. Qualitative comparisons are shown in Fig. 3 and Fig. 12, Appendix F. Compared to NeRF, our model can reconstruct very fine details. Our method also has significantly better visual quality compared NPBG and SynSin.

Results on Tanks&Temples We present the numbers in Tab. 2. All baselines numbers are from Point-NeRF (Xu et al., 2022). Our method achieves comparable quality as Point-NeRF, while being significantly better than other baselines. We present qualitative comparisons in Fig. 16, Appendix F.

Results on NeRF-Synthetic Results are shown in Tab. 3. Our method achieves comparable performance to NeRF while being worse than Point-NeRF, which is also reflected in Fig. 4. Our method is better at capturing the reflective drum surfaces, while struggles with the microphone which has fine geometry. Our explanation is that our view-dependent point features are very expressive in modeling high-frequency textures, while our point cloud is not accurate enough in cases of fine geometries.

We conduct ablation studies of our proposed designs. We show results in Tab. 4. In the 1^st block, We compare with several baselines on point cloud refinement, including 1) the filtering algorithm DCC (Yan et al., 2020) which achieves SOTA performance on MVS, 2) using the raw MVS point cloud without any pruning or adding, 3) pruning with the proposed point pruning but no point adding, 4) using the same point cloud as the complete model, while keeping the point positions and opacity values fixed during gradient updates. Also see Fig. 5 for qualitative comparisons. Results show that all proposed geometry refinement components contribute to the final model. While point pruning contributes to sharper object boundaries near the head of the plush, point adding is especially helpful for filling large holes on the table in the rabbit scene. See Fig. 11, Appendix C for visualizations of the point cloud generated by each method.

The 2^nd block shows that using SH reduces the layer latency by 82% and improves the PSNR by $0.38$dB compared to MLP. For the MLP baseline, we use a 2-layer MLP with 256 hidden units, which takes as input the concatenation of the point feature and the positional-encoded view direction, following NeRF. See also Fig. 14, Appendix F for visualization of the non-Lambertian effect learned by our model. The 3^rd block shows that using dropout layer improves the PSNR by $1.28$dB, and the model is not sensitive to the dropout rate. The 4^th block shows that not using any normalization layers improves the PSNR by $1.49$dB compared to using BatchNorm, and by $0.60$dB compared to InstanceNorm. Replacing U-Net with a 2-layer 1$\times$1 Conv network gives significantly worse results.

We show that our system supports, with high fidelity, scene editing operations such as scene composition, object deformation, and erasing. Results are shown in the inset figure on the right. Composition is achieved by first co-training two scenes with separate point features and a shared U-Net, then putting the points into a single scene at inference time. For deformation, we export the sculpted point cloud into MeshLab (Cignoni et al., 2008), where we manually select the moving part and its axis of rotation. For erasing, we filter out points based on their $z$ coordinates.

Compared to existing neural rendering pipelines that support scene editing, our system has two main advantages: 1. Fine-grained editing: Previous works (Lombardi et al., 2019; Yu et al., 2021b; Yang et al., 2021b) use explicit representations like voxel grids, which are typically limited in resolution. Therefore, they can only achieve object-level operations such as composition. In comparison, we represent object surfaces densely with millions of points, so we can do fine-grained editing such as object deformation. 2. Ease of use: Previous works doing scene editing with NeRF either require a special interface to take user inputs (Liu et al., 2021), or a complex pipeline that uses meshes as an intermediate representation (Yuan et al., 2022). In contrast, our point cloud representation is directly supported by nearly all graphics toolboxes such as MeshLab or Blender, which allows users to edit the scene intuitively without any specialized tool.

We compare our model’s inference speed, training time, and model size with a few baselines on the NeRF-Synthetic dataset, shown in Tab. 5. All speeds are benchmarked using an RTX 3090 GPU. Compared to NeRF, our model is more than 100$\times$ faster in inference and requires only 33% training time. PlenOctrees (Yu et al., 2021b) bakes the radiance field into a voxel-based cache, resulting in faster rendering speed but also a significantly larger model size and longer training time. NPBG (Aliev et al., 2020) achieves faster inference speed with their one-pixel point splats, but at the cost of worse visual quality. Finally, we are about 25$\times$ faster than Point-NeRF (Xu et al., 2022) in rendering while other metrics are roughly the same.