Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection

Multi-View Stereo (MVS) is a technique that aims to reconstruct fine-grained scene geometry from multi-view images. Traditional MVS methods can be classified into four categories based on the output scene representation: volumetric-based methods [6, 41], mesh-based methods [10], point cloud-based methods [11, 24], and depth map-based methods [4, 12, 38, 39, 47]. Among them, depth map-based methods are the most flexible, estimating depth maps for each view by utilizing photometric consistency between reference and neighboring images [11], then fusing all the depth maps into dense point clouds. Surface reconstruction methods [6, 18, 22], such as screened Poisson Surface Reconstruction [18], are then employed to reconstruct surfaces from the point clouds.

Recent learning-based MVS methods use deep learning for improved reconstruction. SurfaceNet [15] and LSM [17] were the first volumetric learning-based MVS pipelines proposed to regress surface voxels. More recently, MVSNet [50] extracts deep image features and warps them into the reference camera frustum to construct a 3D cost volume via differentiable warping.

However, learning-based MVS methods may still produce unsatisfactory results in certain scenarios, such as surfaces with specular reflections, regions with low texture, and non-Lambertian regions. In these cases, photometric consistency across multi-view images is not guaranteed, which can lead to severe artifacts and missing parts in the reconstruction results.

Recently, learning-based approaches using implicit surface representations have been proposed. In these representations, a neural network maps continuous points in 3D space to an occupancy field [29, 36] or a Signed Distance Function (SDF) [34]. These methods perform multi-view reconstruction with additional supervision corresponding to the occupancy value or SDF for each point. However, supervision for these methods is not always available with only multi-view 2D images, which limits their scalability.

The introduction of volumetric approaches in NeRF [30], which combines classical volume rendering [16] with implicit functions for novel view synthesis, has attracted a lot of attention towards 3D reconstruction using neural implicit surface representations and volume rendering [31, 52, 45]. Unlike NeRF, which aims to render novel view images while keeping the geometry unconstrained, these methods define the surface in a more explicit manner and are therefore better suited for surface extraction. UNISURF [31] uses the occupancy field [29], while IDR [53], VolSDF [52], and NeuS [45] use the SDF field [34] as an implicit surface representation. Despite their promising performance on 3D reconstruction, these methods fall short of recovering the correct geometry of objects with reflections, leading to ambiguous surface optimization. Our approach builds upon NeuS [45], but we believe it can be adapted to fit any volumetric neural implicit framework.

We discuss rendering and reconstruction for objects with reflections. Recent works [3, 42, 55, 57, 43] have investigated rendering view-dependent reflective appearance by decomposing a scene into shape, reflectance, and illumination for novel view synthesis and relighting. However, the recovered meshes are not explicitly validated, and the geometry is often unsatisfactory. Reconstruction, on the other hand, aims to recover explicit geometry, which is still under-explored due to the inherent challenges. For instance, PM-PMVS [5] formulates the reconstruction task as joint energy minimization over surface geometry and reflectance, while nLMVS-Net [49] formulates MVS as an end-to-end learnable network, leveraging surface normals as view-independent surface features for cost volume construction and filtering. However, none of these methods combines neural implicit surfaces with volume rendering for reconstruction.

Warping-based consistency learning is widely used in both multi-view stereo [44, 48, 47, 54] and neural implicit surface learning [7, 9] for 3D reconstruction by exploiting inter-image correspondence with differentiable warping operations. Generally, consistency learning in MVS-based pipelines is performed at the CNN feature level. For example, MVSDF [54] warps the predicted surface point to its neighboring views and enforces pixel-wise feature consistency, while ACMM [47] warps the coarse predicted depth to form a multi-view aggregated geometric consistency cost to refine finer scales. Consistency learning in neural implicit surface-based pipelines, on the other hand, is typically performed at the image level. NeuralWarp [7] warps the sampled points along a ray to source images to obtain their RGB values and optimizes them jointly with a radiance network, while Geo-NeuS [9] warps the gray-scale patch centered on the predicted surface point to its neighboring images to guarantee multi-view geometry consistency. However, they ignore view-dependent radiance and are limited when multi-view consistency is not reasonable due to reflection, which can lead to artifacts when minimizing patch similarity regardless of reflection. Furthermore, visibility identification is not well-handled, and both rely on cumbersome preprocessing to determine source images. Alternatively, we leverage inconsistency to reduce ambiguity for high-fidelity reconstruction.

Volume rendering [16] is used in NeRF [30] for novel view synthesis. The idea is to represent the continuous attributes (i.e., density and radiance) of a 3D scene with neural networks. $\alpha$ compositing [28] aggregates these attributes along a ray $\mathbf{r}$ to approximate the pixel RGB values by:

where $T_{i}=\exp\left(-\sum_{j=1}^{i-1}\alpha_{j}\delta_{j}\right)$ and $\alpha_{i}=1-\exp\left(-\sigma_{i}\delta_{i}\right)$ denote the transmittance and alpha value of sampled point, respectively. $\delta_{i}$ is the distance between neighboring sampled points. $P$ is the number of sampled points along a ray. $\sigma_{i}$ and $\mathbf{c}_{i}$ are predicted attributes by the neural networks conditioned on position $\mathbf{x}=(x,y,z)$ and view direction $\mathbf{d}=(\theta,\phi)$. The training object $\mathcal{L}$ of NeRF is the mean square error between the ground-truth pixel color $\mathbf{C}(\mathbf{r})$ and the rendering color $\hat{\mathbf{C}}(\mathbf{r})$ formulated as

where $\mathcal{R}$ is the set of all rays shooting from the camera center to image pixels.

However, density-based volume rendering lacks a clear definition of the surface, which makes it difficult to extract precise geometry. Alternatively, Signed Distance Function (SDF) clearly defines the surface as the zero-level set, making SDF-based volume rendering more effective in surface reconstruction. Following NeuS [45], the attributes of a 3D scene include signed distance and radiance parameterized by a geometry network $f$ and a radiance network $c$ by:

where the geometry network $f$ maps a spatial position $\mathbf{x}$ to its signed distance $f(\mathbf{x})$ to the object and the radiance network $c$ predicts the color conditioned on position $\mathbf{x}$ and view direction $\mathbf{d}$ to model the view-dependent radiance. To aggregate the signed distances and colors of sampled points along a ray $\mathbf{r}$ for pixel color approximation, we utilize volume rendering similar to that of NeRF. The key difference is the formulation of $\alpha_{i}$, which is calculated from the signed distance $f\left(\mathbf{x}\right)$ rather than density $\sigma_{i}$ as

where $\Phi_{s}(x)=\left(1+e^{-sx}\right)^{-1}$ and $1/s$ is a trainable parameter which indicates the standard deviation of $\Phi_{s}(x)$.

For multi-view reconstruction, multi-view consistency is the promise for accurate surface reconstruction. However, for reflective pixels, the geometry network often predicts ambiguous surfaces, which devastates the multi-view consistency. To overcome the issue, we propose to reduce the influence of reflective surfaces by a reflection-aware photometric loss, which adaptively lowers the weights assigned to reflective pixels. To achieve this, we first define the reflection score, which allows us to identify reflective pixels.

A naive solution is to treat the uncertainty defined in NeRF-W [27] as the reflection score. This approach models the radiance value of a scene as a Gaussian distribution and considers the predicted uncertainty as the variance. By minimizing the negative log-likelihood of the Gaussian distribution, a large variance reduces the importance of a pixel with high uncertainty. Ideally, the reflective pixels should be assigned a large variance to attenuate its influence for reconstruction. However, the implicit uncertainty learned by the MLP is defined on a single ray without considering the multi-view context. Therefore, it may not accurately localize reflective surfaces without explicit supervision.

Similar to NeRF-W [27], we also formulate rendering the color of a ray as a Gaussian distribution $\hat{\mathbf{C}}(\mathbf{r})\sim(\overline{\mathbf{C}}(\mathbf{r}),\overline{\beta}^{2}(\mathbf{r}))$, where $\overline{\mathbf{C}}(\mathbf{r})$ and $\overline{\beta}^{2}(\mathbf{r})$ are the mean and variance, respectively. We adopt Eq. (1) to query $\overline{\mathbf{C}}(\mathbf{r})$. However, unlike NeRF-W, which only defines the implicit variance based on the information of a single ray, we explicitly define the variance based on the multi-view context. Specifically, we utilize multi-view pixel colors that refer to the same surface point to determine the variance.

To obtain the multi-view pixel colors, we project the surface point $\mathbf{x}$ onto all $N$ images $\left\{\mathbf{I}_{i}\right\}_{i=1}^{N}$ and use bilinear interpolation to obtain the corresponding pixel colors $\left\{\mathbf{C}_{i}(\mathbf{r})\right\}_{i=1}^{N}$. For simplicity, omitting the subscript, the pixel color $\mathbf{C}$ is obtained by

where $\operatorname{interp}$ indicates bilinear interpolation, $\mathbf{K}$ denotes the internal calibration matrix, $\mathbf{R}$ denotes the rotation matrix, $\mathbf{T}$ denotes the translate matrix and $\cdot$ is matrix multiplication.

Considering the fact that only the local region of partial images contains reflection, we treat reflection localization as an anomaly detection problem, with the expectation that reflective surfaces will be regarded as the anomaly and assigned a high reflection score. To this end, we estimate a view-dependent reflection score $\overline{\beta}^{2}_{i}(\mathbf{r})$ by an anomaly detector empirically, which uses the Mahalanobis distance [26] as the reflection score (i.e., variance) as follows:

where $\gamma$ is the a scale factor to control the scale of the reflection score and $\mathbf{\Sigma}$ is the empirical covariance matrix. As reflections do not dominate the majority of training images, if the currently rendered pixel color $\mathbf{C}_{i}(\mathbf{r})$ is contaminated by reflection, a larger reflection score is generated as most of the relative divergences become large.

Then, we extend the photometric loss in Eq. (2) to a reflection-aware one by minimizing the negative log-likelihood of distribution of rays $\mathbf{r}$ from a batch $\mathcal{R}$ similar to NeRF-W [27] and ActiveNeRF [32] as follows:

Since $\overline{\beta}^{2}(\mathbf{r})$ is estimated by Eq. (6) explicitly instead of implicitly learning by MLP, it is a constant and can be removed from the objective function. Besides, following previous works [45, 52, 9] for multi-view reconstruction, we use L1 loss instead of L2 loss. Finally, our reflection-aware photometric loss is quite simple formulated as

Note that we slightly abuse the term ${\overline{\beta}^{2}(\mathbf{r})}$, which should be corresponding to L2 loss.

The computation of reflection score using all pixel colors $\left\{\mathbf{C}_{j}(\mathbf{r})\right\}_{j=1}^{N}$ assumes that the point on the surface has a valid projection on all source images. However, in practice, this assumption is not true due to self-occlusion. The projected pixel colors are meaningless if the point is invisible in the source images, and the corresponding pixel colors should not be used in Eq. (6).

To address this issue, we design a visibility identification module, which leverages intermediate reconstructed meshes to identify visibility, as illustrated in Fig. 3. Specifically, given a pixel $p_{i}$ corresponding to a ray $\mathbf{r}_{i}$, the implicit surfaces on ray $\mathbf{r}_{i}$ can be represented according to the signed distance of sampled points $\mathbf{x}$ as follows:

Since an infinite number of points exist along a ray, we need to sample discrete points on this ray. Based on the sampled points and their signed distances, we can determine the intervals where surface points exist by

If the sign of the sampled point $f(\mathbf{x}_{j})$ is different from the sign of the next sampled point $f(\mathbf{x}_{j+1})$, the interval $[\mathbf{x}_{j},\mathbf{x}_{j+1}]$ intersects with the surface. The intersection point set $\hat{S}_{i}$ can be obtained by linear interpolation by

In practice, the ray $\mathbf{r}_{i}$ may intersect with the object at multiple surfaces. For our reflection score computation, only the first intersection is meaningful, and it is formulated as

where $\mathbf{x}\in\hat{S}_{i}$ and $\mathcal{D}(\cdot,\mathbf{o}_{i})$ indicate the distance between point $\mathbf{x}$ and the origin of the ray $\mathbf{r}_{i}$, respectively.

After capturing the predicted surface point $\mathbf{x}_{i}^{*}$, we can calculate the distances $\left\{d_{j}^{*}\right\}_{j=1}^{N}$ between this point and all camera locations $\left\{\mathbf{o}_{j}\right\}_{j=1}^{N}$ as follows:

Meanwhile, we compute the distances $\left\{d_{j}\right\}_{j=1}^{N}$ from all camera locations $\left\{\mathbf{o}_{j}\right\}_{j=1}^{N}$ to the first intersection of the intermediate reconstructed meshes by ray casting [37]. Based on these two distances, the visibility of images $\mathbf{I}_{j}$ can be estimated by

where $\mathbb{I}(\cdot)$ is an indicator function. Based on the approximation of visibility, we eliminate the invisible pixel colors $\{\mathbf{C}_{j}(\mathbf{r})\mid v_{j}=0\}$ that are used in the calculation of the reflection score in Eq. (6). We then refine the reflection score as follows:

We provide some examples in Fig. 4 to illustrate the estimated reflection score.

As suggested in Ref-NeRF [43], conditioning the radiance on reflection direction in reflective scenes can lead to a more accurate radiance fields, which has been shown to be beneficial for reconstruction in NeuralWarp [7]. Inspired by this, we reparameterize the radiance network as a function of the reflection direction about the surface normal, and the formulation in Eq. (3) becomes

where $\hat{\mathbf{d}}$ is the reflection direction calculated by

where $\mathbf{n}$ is the surface normal formulated by

Compared to Ref-NeRF [43], the reflection direction is more precise in our framework due to the well-estimated surface normal, which leads to a more accurate radiance fields. Compared to NeuralWarp [7] that ignores reflection in their framework, we take the view-dependent radiance into consideration and estimate a more accurate radiance fields. As a result, our method is more reliable and promising for multi-view reconstruction of objects with reflection.

During reconstructing a scene, our total loss function is

$\mathcal{L}_{\text{color}}$ is the reflection-aware photometric loss in Eq. (8) with reflection score estimated by Eq. (15). The rendered color is computed by Eq. (1) with radiance parameterized in Eq. (16). $\mathcal{L}_{\text{eik}}$ is an eikonal term [13] to regularize the gradients of geometry network formualated as

In our experiments, we choose $\alpha$ as 0.1.

To evaluate the effectiveness of our model, we conducted experiments on objects from several datasets, including Shiny Blender [43], Blender [30], SLF [46], and Bag of Chips [35]. The selected objects in these datasets are glossy.

We implemented our model based on NeuS [45]. The structure of the geometry network and radiance network was the same as that of NeuS. Please refer to our supplement for more details. To estimate visibility in an on-the-fly manner, the intermediate reconstruction result was updated every 500 iterations with a resolution of 128 to decrease the computation cost. $\gamma$ was set to 5. We trained our model for 200k iterations, which took approximately 7 hours on a single NVIDIA RTX 3090 Ti GPU for the reconstruction task. After convergence, a mesh can be extracted from the signed distance functions (SDFs) in a predefined bounding box using Marching Cubes [25] with a resolution of 512.

We compared the reconstruction quality of our method with several other methods, including IDR [53], three baseline methods for multi-view reconstruction (UNISURF [31], VolSDF [52], and NeuS [45]), two warp-based consistency learning methods (NeuralWarp [7] and Geo-NeuS [9]), and two methods specifically designed for modeling objects with reflection (Ref-NeRF [43] and PhySG [55]). The quantitative results are shown in Tables 1 and 2, where accuracy is indicated with the term “Acc” [1]. As COLMAP and NeRF-W fail to recover reflective surfaces, we did not compare with them. Instead, we presented the reconstructed meshes with severe artifacts of COLMAP in the supplement. For NeRF-W, it focuses on novel view synthesis in the wild and failed to produce meshes.

For PhySG and Ref-NeRF, we obtained the MAE from the Ref-NeRF paper, as these methods focus on novel view synthesis, the reconstruction accuracy is not reported in Ref-NeRF paper. For other methods, we implemented the released code on the corresponding datasets. IDR [53] was unable to generate meaningful reconstructions for the helmet and toaster objects, so we only reported the results for the car and coffee objects. Our method significantly outperformed all other compared methods by a large margin. As can be seen qualitatively in Figures 1 and 5, our method yields promising improvements in both geometry accuracy and surface normals. Ref-NeRF adopts Integrated Positional Encoding similar to mip-NeRF [2], which makes its geometry parameters dependent on view direction, making the meshes inaccessible. Furthermore, our results achieve higher rendering quality (PSNR) compared to Ref-NeRF except for a failure case as reported in Table 3. This indicates that more accurate surface normals can lead to improved novel view synthesis quality since the conditioned reflection direction is calculated more accurately compared to Ref-NeRF. Note that, due to memory limitations, we only sampled 1024 rays instead of 4096 $\times$ 4 as in Ref-NeRF.

We conducted an ablation study on the Shiny Blender dataset to evaluate the effectiveness of each component in our model. The experimental results are reported in Table 4. “NeuS w/ RS” indicates that we computed the reflection-aware photometric loss with the reflection score estimated by Eq. (6), without accounting for visibility. This leads to a smaller improvement in performance, suggesting that using the reflection score as a source of variance can still be beneficial in improving geometry. “NeuS w/ Ref” only replaces the radiance dependency with the reflection direction without considering the reflection score. The performance improved over the baseline, except for one failure case, which we discussed and provided per-scene results for in the supplementary material. “Ref-NeuS w/o Ref” indicates that we further identified visibility for reflection score estimation using Eq. (15). The performance significantly improved, highlighting the importance of an unbiased reflection score in accurate geometry recovery. Finally, we used both reflection-aware photometric loss and reflection direction-dependent radiance in our model. The performance significantly improved compared to using only one of these methods, indicating that both methods are complementary for achieving accurate surface reconstruction.