Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

Our framework has three training stages to gradually decompose the shape, materials, and illumination. The input to our framework is a set of images. In the first stage, we reconstruct the surface from a (possibly glossy) appearance decomposing the color into diffuse and specular components. After that, we use the reconstructed radiance field to extract direct illumination visibility and indirect illumination in the second stage. Having them decomposed from the radiance field allows for the recovery of the direct illumination map and materials’ bidirectional reflectance distribution function (BRDF), which we perform in the final stage.

Volume rendering. The radiance $C$ of the pixel corresponding to a given ray $\bm{r}(t)=\bm{o}+t\bm{d}$ at the origin $\bm{o}\in\mathbb{R}^{3}$ towards direction $\bm{d}\in\mathbb{S}^{2}$ is calculated using the volume rendering equation, which involves an integral along the ray with boundaries $t_{n}$ and $t_{f}$ ($t_{n}$ and $t_{f}$ are parameters to define the near and far clipping plane). This calculation requires the knowledge of the volume density $\sigma$ and directional color $\bm{c}$ for each point within the volume.

The volume density $\sigma$ is used to calculate the accumulated transmittance $T(t)$:

It is then used to compute a weighting function $w(t)=T(t)\sigma(\bm{r}(t))$ to weigh the sampled colors along the ray $\bm{r}(t)$ to integrate into radiance $C(\bm{r})$.

Current inverse rendering methods first recover implicit neural surfaces, typically represented as SDFs, from multi-view images to recover shape information, then freeze the parameters of neural surfaces to further recover the material. However, this approach does not consider specular reflections that produce highlights and often models this inconsistent color as bumpy surface geometry as depicted in Fig. 1. This incorrect surface reconstruction has a negative impact on subsequent material reconstruction. We propose a neural surface reconstruction method that considers the appearance, diffuse color, and specular color of glossy surfaces at the same time, whose architecture is given in Fig. 3. Our inspiration comes from the following observations. First, according to Geo-NeuS, using SDF point cloud supervision can make the colors of surface points and volume rendering more similar. We abandoned the idea of using additional surface points to supervise SDFs and directly use two different MLPs to predict the surface rendering and volume rendering results, and narrow the gap between these two colors using network training. In addition, when modeling glossy surfaces, Ref-NeRF proposes a method of decomposing appearance into diffuse and specular components, which can better model the glossy appearance. We propose to simultaneously optimize the volume appearance, surface diffuse radiance, and specular radiance of surface points, which can achieve an improved surface reconstruction of glossy surfaces.

To compute color $\bm{c_{i}}$ on the point $\bm{r}(t_{i})$, we define a color mapping $\mathsf{M}_{c}:({\bm{x},\bm{n},\bm{d}},\bm{v}_{f})\mapsto\bm{c}$ from any 3D point $\bm{x}$ given its feature vector $\bm{v}_{f}$, normal $\bm{n}$ and ray direction $\bm{d}$.

We define two neural networks to predict diffuse and specular components separately. Diffuse radiance refers to the scattered and spread-out light that illuminates a surface or space evenly and without distinct shadows or reflections. We define a mapping $\mathsf{M}_{d}:({\bm{x},\bm{n}},\bm{v}_{f})\mapsto\bm{c}_{d}$ for diffuse radiance that maps surface points $\bm{x}$, surface normals $\bm{n}$, and feature vectors $\bm{v}_{f}$ to diffuse radiance. For simplicity, we assume that the diffuse radiance is not related to the outgoing viewing direction $\bm{\omega}_{o}$.

Specular radiance is usually highly dependent on the viewing direction. Ref-NeRF (Verbin et al., 2022) proposes to use the reflection direction instead of the viewing direction to model the glossy appearance in a compact way. However, from Eq. 7, we can observe that specular radiance is also highly dependent on the surface normal, which is particularly important when reconstructing SDF. In contrast to Ref-NeRF, we further condition specular radiance on the surface normal. Therefore, we define specular radiance $\mathsf{M}_{s}:({\bm{x},\bm{\omega}_{r},\bm{n}},\bm{v}_{f})\mapsto\bm{c}_{s}$, which maps surface points $\bm{x}$, outgoing viewing direction $\bm{\omega}_{o}$, surface normals $\bm{n}$, and feature vectors $\bm{v}_{f}$ to specular radiance, where $\bm{\omega}_{r}=2(\bm{\omega}_{o}\cdot\bm{n})\bm{n}-\bm{\omega}_{o}$.

Surface rendering focuses the rendering process on the surface, allowing for a better understanding of highlights on the surface compared to volume rendering, but requires calculating surface points. We sample $n$ points on the ray $\{\bm{r}(t_{i})|i=1,...,n\}$. We query the sampled points to find the first point $\bm{r}(t_{i}^{\prime})$ whose SDF value is less than zero $\mathsf{S}_{\theta}(\bm{r}(t_{i}^{\prime}))<0$. Then the point $\bm{r}(t_{i^{\prime}-1})$ sampled before $\bm{r}(t_{i}^{\prime})$ has the SDF value greater than or equal to zero $\mathsf{S}_{\theta}(\bm{r}(t_{i^{\prime}-1}))\geq 0$. To account for the possibility of rays interacting with objects and having multiple intersection points, we select the first point with a negative SDF value to solve this issue.

We use two neural networks to predict the diffuse radiance and specular radiance of two sampling points $\bm{r}(t_{i^{\prime}-1})$ and $\bm{r}(t_{i}^{\prime})$. The diffuse radiance of the two points calculated by the diffuse network $\mathsf{M}_{d}$ will be $\bm{c}_{d}^{i^{\prime}-1}$ and $\bm{c}_{d}^{i^{\prime}}$. The specular radiance of the two points calculated by the specular network $\mathsf{M}_{s}$ will be $\bm{c}_{s}^{i^{\prime}-1}$ and $\bm{c}_{s}^{i^{\prime}}$. Therefore, the diffuse radiance and specular radiance of the surface point $\bm{x}$ can be calculated as follows.

The final radiance of the intersection of the ray and the surface is calculated by a tone mapping as follows.

where $\gamma$ is a pre-defined tone mapping function that converts linear color to sRGB (Verbin et al., 2022) while ensuring that the resulting color values are within the valid range of [0, 1] by clipping them accordingly.

The surface radiance loss $\mathcal{L}_{\text{sur}}$ is measured by calculating the $\mathcal{L}_{1}$ distance between the ground truth colors ${C}^{\text{gt}}$ and the surface radiances ${C}^{\text{sur}}$. During the training process, only a few rays have intersection points with the surface. We only care about the set of selected rays $\mathcal{R}^{\prime}$, which satisfies the condition that each ray exists point whose SDF value is less than zero and not the first sampled point. The loss is defined as follows.

$\mathcal{L}_{\text{reg}}$ is an Eikonal loss term on the sampled points. Eikonal loss is a regularization loss applied to a set of sampling points $X$, which is used to constrain the noise in signed distance function (SDF) generation.

We use weights $\lambda_{\text{sur}}$ and $\lambda_{\text{reg}}$ to balance the impact of these three losses. The overall training weights are as follows.

At this stage, we focus on predicting the lighting visibility and indirect illumination of a surface point $\bm{x}$ under different incoming light direction $\bm{\omega}_{i}$ using the SDF in the first stage. Therefore, we need first to calculate the position of the surface point $\bm{x}$. In stage one, we have calculated two sampling points $\bm{r}(t_{i^{\prime}-1}),\bm{r}(t_{i}^{\prime})$ near the surface. As Geo-NeuS (Fu et al., 2022), we weigh these two sampling points to obtain a surface point $\bm{x}$ as follows.

The weights of the light visibility network are optimized by minimizing the loss between the calculated ground truth values and the predicted values of a set of sampled incoming light directions $\Omega_{i}\subset\mathbb{S}^{2}$. This pre-integrated technique can reduce the computational burden caused by the integration for subsequent training.

Reconstructing good materials and lighting from scenes with highlights is a challenging task. Following prior works (Zhang et al., 2022b, 2021a), we use the Disney BRDF model (Burley and Studios, 2012) and represent BRDF $f_{s}(\bm{\omega}_{i}\ |\ \bm{\xi}_{s},\lambda_{s},\bm{\mu}_{s})$ via Spherical Gaussians (Zhang et al., 2021a). Direct (environment) illumination is represented using $K_{e}=128$ SGs:

and render diffuse radiance and specular radiance of direct illumination in a way similar to Eq. 6.

where $\bm{d}_{a}$ is diffuse albedo.

To reconstruct a more accurate specular reflection effect, we use an additional neural network $\mathsf{M}_{sa}:(\bm{x},\bm{\omega}_{r})\mapsto\bm{s}_{a}\in[0,1]$ to predict the specular albedo. The modified BRDF $f^{a}_{s}$ is as follows:

For indirect illumination, the radiance is extracted directly from another surface and does not consider light visibility. The diffuse radiance and specular radiance of indirect illumination are as follows

Our final synthesized appearance is $C=L_{d}+L_{s}+L^{\text{ind}}_{d}+L^{\text{ind}}_{s}$ and can be supervised by the following RGB loss.

Our full model is composed of several MLP networks, each one of them having a width of 256 hidden units unless otherwise stated. In Stage 1, the SDF network $\mathsf{S}_{\theta}$ is composed of 8 layers and includes a skip connection at the 4-th layer, similar to NeuS (Wang et al., 2021). The input 3D coordinate $\bm{x}$ is encoded using positional encoding with 6 frequency scales. The diffuse color network $\mathsf{M}_{d}$ utilizes a 4-layer MLP, while the input surface normal $\bm{n}$ is positional-encoded using 4 scales. For the specular color network $\mathsf{M}_{s}$, a 4-layer MLP is employed, and the reflection direction $\bm{\omega}_{r}$ is also positional-encoded using 4 frequency scales.

In Stage 2, the light visibility network $\mathsf{M}_{\nu}$ has 4 layers. To better encode the input 3D coordinate $\bm{x}$, positional encoding with 10 frequency scales is utilized. The input view direction $\omega_{i}$ is also positional-encoded using 4 scales. The indirect light network $\mathsf{M}_{\text{ind}}$ in stage 2 comprises 4 layers.

In stage 3, the encoder part of the BRDF network consists of 4 layers, and the input 3D coordinate is positional-encoded using 10 scales. The output latent vector $\bm{z}$ has 32 dimensions, and we impose a sparsity constraint on the latent code $\bm{z}$, following IndiSG (Zhang et al., 2022b). The decoder part of the BRDF network is a 2-layer MLP with a width of 128, and the output has 4 dimensions, including the diffuse albedo $\bm{d}_{a}\in\mathbb{R}^{3}$ and roughness $r\in\mathbb{R}$. Finally, the specular albedo network $\mathsf{M}_{sa}$ uses a 4-layer MLP, where the input 3D coordinate $\bm{x}$ is positional-encoded using 10 scales, and the input reflection direction $\bm{\omega}_{r}$ is positional-encoded using 4 scales.

The learning rate for all three stages begins with a linear warm-up from 0 to $5\times 10^{-4}$ during the first 5K iterations. It is controlled by the cosine decay schedule until it reaches the minimum learning rate of $2.5\times 10^{-5}$, which is similar to NeuS. The weights $\lambda_{\text{sur}}$ for the surface color loss are set for 0.1, 0.6, 0.6, and 0.01 for DTU, SK3D, Shiny, and the IndiSG dataset, respectively. We train our model for 300K iterations in the first stage, which takes 11 hours in total. For the second and third stages, we train for 40K iterations, taking around 1 hour each. The training was performed on a single NVIDIA RTX 4090 GPU.

To evaluate the effectiveness of material and lighting reconstruction, we use the dataset provided by IndiSG (Zhang et al., 2022b), which has self-occlusion and complex materials. Each scene has 100 training images of 800 $\times$ 800 resolution. To evaluate the quality of material decomposition, the dataset also provides diffuse albedo, roughness, and masks for testing.

Apart from that, we also compare against more specialized methods for individual quantitative and qualitative comparisons to provide additional context for our results. For surface reconstruction quality, we compare our method to NeuS (Wang et al., 2021) and Geo-NeuS (Fu et al., 2022). NeuS is a popular implicit surface reconstruction method that achieves strong results without reliance on extra data. Geo-NeuS improves upon NeuS by using additional point cloud supervision, obtained from structure from motion (SfM) (Schönberger and Frahm, 2016). We also show a qualitative comparison to Ref-NeRF (Verbin et al., 2022), which considers material decomposition, but due to modeling geometry using density function, it has difficulty extracting smooth geometry.

We first demonstrate quantitative results in terms of Chamfer distance. IndiSG and PhySG share the same surface reconstruction method, but PhySG optimizes it together with the materials, while IndiSG freezes the underlying SDF after its initial optimization. We list the numerical results for IndiSG and PhySG for comparison. NVDiffrec is not as good for surface reconstruction as we verify qualitatively in Fig. 5. For completeness, we also compare our method against NeuS and Geo-NeuS. First, we list quantitative results on the DTU dataset and SK3D dataset in Tab. 2. It should be noted that NeuS and Geo-NeuS can only reconstruct surfaces from multi-views, while our method and IndiSG can simultaneously tackle shape, material, and lighting. As shown in the table, Geo-NeuS achieves better performance on the DTU dataset because the additional sparse 3D points generated by structure from motion (SfM) for supervising the SDF network are accurate. However, on the SK3D scenes with glossy surfaces, these sparse 3D points cannot be generated accurately by SFM, leading to poor surface reconstruction by Geo-NeuS. In contrast, our approach can reconstruct glossy surfaces on both DTU and SK3D without any explicit geometry information. Compared with IndiSG, PhySG cannot optimize geometry and material information well simultaneously on real-world acquired datasets with complex lighting and materials. Our method is the overall best method on SK3D. Most importantly, we demonstrate large improvements over IndiSG and PhySG, our main competitors, on both DTU and SK3D.

We further demonstrate the qualitative experimental comparison results in Fig. 4. It can be seen that although Geo-NeuS has the best quantitative evaluation metrics, it loses some of the fine details, such as the small dents on the metal can in DTU 97. By visualizing the results of the SK3D dataset, we can validate that our method can reconstruct glossy surfaces without explicit geometric supervision.

We then show the qualitative results for surface reconstruction compared with NeuS, Ref-NeRF, IndiSG, PhySG, and NVDiffrec on the Shiny dataset, which is a synthetic dataset with glossy surfaces. From Fig. 5, we can observe that NeuS is easily affected by highlights, and the geometry reconstructed by Ref-NeRF has strong noise. PhySG is slightly better than IndiSG on the Shiny synthetic dataset with jointly optimizing materials and surfaces, such as toaster and car scenes, but still can not handle complex highlights. NVDiffrec works well on the teapot model but fails on other more challenging glossy surfaces. Our method is able to produce clean glossy surfaces without being affected by the issues caused by highlights. Overall, our approach demonstrates superior performance in surface reconstruction, especially on glossy surfaces with specular highlights.

In Tab. 3, we evaluate the quantitative results in terms of PSNR metric for material and illumination reconstruction on the IndiSG dataset compared with PhySG, NVDiffrec, and IndiSG. For completeness, we also compare to the case where the specular albedo improvement was not used in Stage 3 (See in Eq. 24 in Section 3.5). Regarding diffuse albedo, although NVDiffrec showed a slight improvement over us in the balloons scene, we achieved a significant improvement over NVDiffrec in the other three scenes. Our method achieved the best results in material reconstruction. Moreover, our method achieves the best results in illumination quality without using the specular albedo improvement. Additionally, our method significantly outperforms other methods in terms of rendering quality and achieves better appearance synthesis results. We present the qualitative results of material reconstruction in Fig. 6, which shows that our method has better detail capture compared to IndiSG and PhySG, such as the text on the balloon. Although NVDiffrec can reconstruct the nails on the backrest, its material decomposition effect is not realistic. The materials reconstructed by our method are closer to ground truth ones. We also demonstrate the material decomposition effectiveness of our method on Shiny datasets with glossy surfaces, as shown in Fig. 7. We showcase the diffuse albedo and rendering results of NVDiffrec, IndiSG, and our method. The rendering results indicate that our method can restore the original appearance with specular highlights more accurately, such as the reflections on the helmet and toaster compared to the IndiSG and NVDiffrec methods. The material reconstruction results show that our diffuse albedo contains less specular reflection information compared to other methods, indicating our method has better ability to suppress decomposition ambiguity caused by specular highlights.

In addition to the synthetic datasets with ground truth decomposed materials, we also provide qualitative results on real-world captured datasets such as DTU and SK3D in Fig. 8. From the DTU data, we can observe that our method has the ability to separate the specular reflection component from the diffuse reflection component, as seen in the highlights on the apple, can, and golden rabbit. Even when faced with a higher intensity of specular reflection, as demonstrated in the example showcased in SK3D, our method excels at preserving the original color in the diffuse part and accurately separating highlights into the specular part.