NeuS: Learning Neural Implicit Surfaces
by Volume Rendering for Multi-view Reconstruction

In order to apply a volume rendering method to training the SDF network, we first introduce a probability density function $\phi_{s}(f({\bf x}))$, called S-density, where $f({\bf x})$, ${\bf x}\in\mathbb{R}^{3}$, is the signed distance function and $\phi_{s}(x)=se^{-sx}/(1+e^{-sx})^{2}$, commonly known as the logistic density distribution, is the derivative of the Sigmoid function $\Phi_{s}(x)=(1+e^{-sx})^{-1}$, i.e., $\phi_{s}(x)=\Phi_{s}^{\prime}(x)$. In principle $\phi_{s}(x)$ can be any unimodal (i.e. bell-shaped) density distribution centered at $0$; here we choose the logistic density distribution for its computational convenience. Note that the standard deviation of $\phi_{s}(x)$ is given by $1/s$, which is also a trainable parameter, that is, $1/s$ approaches to zero as the network training converges.

Intuitively, the main idea of NeuS is that, with the aid of the S-density field $\phi_{s}(f({\bf x}))$, volume rendering is used to train the SDF network with only 2D input images as supervision. Upon successful minimization of a loss function based on this supervision, the zero-level set of the network-encoded SDF is expected to represent an accurately reconstructed surface $\mathcal{S}$, with its induced S-density $\phi_{s}(f({\bf x}))$ assuming prominently high values near the surface.

Requirements on weight function. The key to learn an accurate SDF representation from 2D images is to build an appropriate connection between output colors and SDF, i.e., to derive an appropriate weight function $w(t)$ on the ray based on the SDF $f$ of the scene. In the following, we list the requirements on the weight function $w(t)$.

1. 1.

   Unbiased. Given a camera ray $\mathbf{p}(t)$, $w(t)$ attains a locally maximal value at a surface intersection point $\mathbf{p}(t^{*})$, i.e. with $f(\mathbf{p}(t^{*}))=0$, that is, the point $\mathbf{p}(t^{*})$ is on the zero-level set of the SDF $\mathbf{(}{\bf x})$.

2. 2.

   Occlusion-aware. Given any two depth values $t_{0}$ and $t_{1}$ satisfying $f(t_{0})=f(t_{1})$, $w(t_{0})>0$, $w(t_{1})>0$, and $t_{0}<t_{1}$, there is $w(t_{0})>w(t_{1})$. That is, when two points have the same SDF value (thus the same SDF-induced S-density value), the point nearer to the view point should have a larger contribution to the final output color than does the other point.

An unbiased weight function $w(t)$ guarantees that the intersection of the camera ray with the zero-level set of SDF contributes most to the pixel color. The occlusion-aware property ensures that when a ray sequentially passes multiple surfaces, the rendering procedure will correctly use the color of the surface nearest to the camera to compute the output color.

Next, we will first introduce a naive way of defining the weight function $w(t)$ that directly using the standard pipeline of volume rendering, and explain why it is not appropriate for reconstruction before introducing our novel construction of $w(t)$.

Naive solution. To make the weight function occlusion-aware, a natural solution is based on the standard volume rendering formulation [29] which defines the weight function by

where $\sigma(t)$ is the so-called volume density in classical volume rendering and $T(t)=\exp(-\int_{0}^{t}\sigma(u){\rm d}u)$ here denotes the accumulated transmittance along the ray. To adopt the standard volume density formulation [29], here $\sigma(t)$ is set to be equal to the S-density value, i.e. $\sigma(t)=\phi_{s}(f(\mathbf{p}(t)))$ and the weight function $w(t)$ is computed by Eqn. 3. Although the resulting weight function is occlusion-aware, it is biased as it introduces inherent errors in the reconstructed surfaces. As illustrated in Fig. 2 (a), the weight function $w(t)$ attains a local maximum at a point before the ray reaches the surface point $\mathbf{p}(t^{*})$, satisfying $f(\mathbf{p}(t^{*}))=0$. This fact will be proved in the supplementary material.

Our solution. To introduce our solution, we first introduce a straightforward way to construct an unbiased weight function, which directly uses the normalized S-density as weights

This construction of weight function is unbiased, but not occlusion-aware. For example, if the ray penetrates two surfaces, the SDF function $f$ will have two zero points on the ray, which leads to two peaks on the weight function $w(t)$ and the resulting weight function will equally blend the colors of two surfaces without considering occlusions.

To this end, now we shall design the weight function $w(t)$ that is both occlusion-aware and unbiased in the first-order approximation of SDF, based on the aforementioned straightforward construction. To ensure an occlusion-aware property of the weight function $w(t)$, we will still follow the basic framework of volume rendering as Eqn. 3. However, different from the conventional treatment as in naive solution above, we define our function $w(t)$ from the S-density in a new manner. We first define an opaque density function $\rho(t)$, which is the counterpart of the volume density $\sigma$ in standard volume rendering. Then we compute the new weight function $w(t)$ by

How we derive opaque density $\rho$. We first consider a simple ideal case where there is only one surface intersection, and the surface is simply a plane that approaches infinitely far off the camera. Since Eqn. 4 indeed satisfies the above requirements under this assumption, we derive the underlying opaque density $\rho$ corresponding to the weight definition of Eqn. 4 using the framework of volume rendering. Then we will generalize this opaque density to the general case of multiple surface intersections.

Specifically, in the simple case of a single plane intersection, it is easy to see that the signed distance function $f(\mathbf{p}(t))$ is $-|\cos(\theta)|\cdot(t-t^{*})$, where $f(\mathbf{p}(t^{*}))=0$, and $\theta$ is the angle between the view direction $\mathbf{v}$ and the outward surface normal vector $\mathbf{n}$. Because the surface is assumed a plane, $|\cos(\theta)|$ is a constant. It follows from Eqn. 4 that

Recall that the weight function within the framework of volume rendering is given by $w(t)=T(t)\rho(t)$, where $T(t)=\exp(-\int_{0}^{t}\rho(u){\rm d}u)$ denotes the accumulated transmittance. Therefore, to derive $\rho(t)$, we have

Since $T(t)=\exp(-\int_{0}^{t}\rho(u){\rm d}u)$, it is easy to verify that $T(t)\rho(t)=-\frac{{\rm d}T}{{\rm d}t}(t)$. Further, note that $|\cos(\theta)|\phi_{s}(f(\mathbf{p}(t)))=-\frac{{\rm d}\Phi_{s}}{{\rm d}t}(f(\mathbf{p}(t)))$. It follows that $\frac{{\rm d}T}{{\rm d}t}(t)=\frac{{\rm d}\Phi_{s}}{{\rm d}t}(f(\mathbf{p}(t)))$. Integrating both sides of this equation yields

Taking the logarithm and then differentiating both sides, we have

This is the formula of the opaque density $\rho(t)$ in the ideal case of single plane intersection. The weight function $w(t)$ induced by $\rho(t)$ is shown in Figure 2(b). Now we generalize the opaque density to the general case where there are multiple surface intersections along the ray $\mathbf{p}(t)$. In this case, $-\frac{{\rm d}\Phi_{s}}{{\rm d}t}(f(\mathbf{p}(t)))$ becomes negative on the segment of the ray with increasing SDF values. Thus we clip it against zero to ensure that the value of $\rho$ is always non-negative. This gives the following opaque density function $\rho(t)$ in general cases.

Based on this equation, the weight function $w(t)$ can be computed with standard volume rendering as in Eqn. 5. The illustration in the case of multiple surface intersection is shown in Figure 3.

The following theorem states that in general cases (i.e., including both single surface intersection and multiple surface intersections) the weight function defined by Eqn. 10 and Eqn. 5 is unbiased in the first-order approximation of SDF. The proof is given in the supplementary material.

Discretization. To obtain discrete counterparts of the opacity and weight function, we adopt the same approximation scheme as used in NeRF [29], This scheme samples $n$ points $\{\mathbf{p}_{i}=\mathbf{o}+t_{i}\mathbf{v}|i=1,...,n,t_{i}<t_{i+1}\}$ along the ray to compute the approximate pixel color of the ray as

where $T_{i}$ is the discrete accumulated transmittance defined by $T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})$, and $\alpha_{i}$ is discrete opacity values defined by

which can further be shown to be

The detailed derivation of this formula for $\alpha_{i}$ is given in the supplementary material.

To train NeuS, we minimize the difference between the rendered colors and the ground truth colors, without any 3D supervision. Besides colors, we can also utilize the masks for supervision if provided.

Specifically, we optimize our neural networks and inverse standard deviation $s$ by randomly sampling a batch of pixels and their corresponding rays in world space $P=\left\{C_{k},M_{k},\mathbf{o}_{k},\mathbf{v}_{k}\right\}$, where $C_{k}$ is its pixel color and $M_{k}\in\{0,1\}$ is its optional mask value, from an image in every iteration. We assume the point sampling size is $n$ and the batch size is $m$. The loss function is defined as

The color loss $\mathcal{L}_{color}$ is defined as

Same as IDR[49], we empirically choose $\mathcal{R}$ as L1 loss, which in our observation is robust to outliers and stable in training.

We add an Eikonal term [10] on the sampled points to regularize the SDF of $f_{\theta}$ by

The optional mask loss $\mathcal{L}_{mask}$ is defined as

where $\hat{O}_{k}=\sum_{i=1}^{n}T_{k,i}\alpha_{k,i}$ is the sum of weights along the camera ray, and BCE is the binary cross entropy loss.

Hierarchical sampling. In this work, we follow a similar hierarchical sampling strategy as in NeRF [29]. We first uniformly sample the points on the ray and then iteratively conduct importance sampling on top of the coarse probability estimation. The difference is that, unlike NeRF which simultaneously optimizes a coarse network and a fine network, we only maintain one network, where the probability in coarse sampling is computed based on the S-density $\phi_{s}(f(\mathbf{x}))$ with fixed standard deviations while the probability of fine sampling is computed based on $\phi_{s}(f(\mathbf{x}))$ with the learned $s$. Details of hierarchical sampling strategy are provided in supplementary materials.

We conducted the comparisons in two settings, with mask supervision (w/ mask) and without mask supervision (w/o mask). We measure the reconstruction quality with the Chamfer distances in the same way as UNISURF [31] and IDR [49] and report the scores in Table 1. The results show that our approach outperforms the baseline methods on the DTU dataset in both settings – w/ and w/o mask in terms of the Chamfer distance. Note that the reported scores of IDR in the setting of w/ mask and NeRF and UNISURF in the w/o mask setting are from IDR [49] and UNISURF [31].

We conduct the qualitative comparisons on the DTU dataset and the BlendedMVS dataset in both settings, w/ mask and w/o mask, in Figure 4 and Figure 5, respectively. As shown in Figure 4 for the setting of w/ mask, IDR shows limited performance for reconstructing thin metals parts in Scan 37 (DTU), and fails to handle sudden depth changes in Stone (BlendedMVS) due to the local optimization process in surface rendering. The extracted meshes of NeRF are noisy since the volume density field has not sufficient constraint on its 3D geometry. Regarding the w/o mask setting, we visually compare our method with NeRF and COLMAP in the setting of w/o mask in Figure 5, which shows our reconstructed surfaces are with more fidelity than baselines. We further show a comparison with UNISURF [31] on two examples in the w/o mask setting. Note that we use the qualitative results of UNISURF reported their paper for comparison. Our method works better for the objects with abrupt depth changes. More qualitative images are included in the supplementary material.

To evaluate the effect of the weight calculation, we test three different kinds of weight constructions described in Sec. 3.1: (a) Naive Solution. (b) Straightforward Construction as shown in Eqn. 4. (e) Full Model. As shown in Figure 6, the quantitative result of naive solution is worse than our weight choice (e) in terms of the Chamfer distance. This is because it introduces a bias to the surface reconstruction. If direct construction is used, there are severe artifacts.

We also studied the effect of Eikonal regularization [10] and geometric initialization [1]. Without Eikonal regularization or geometric initialization, the result on Chamfer distance is on par with that of the full model. However, neither of them can correctly output a signed distance function. This is indicated by the MAE(mean absolute error) between the SDF predictions and corresponding ground-truth SDF, as shown in the bottom line of Figure 6. The MAE is computed on uniformly-sampled points in the object’s bounding sphere. Qualitative results of SDF predictions are provided in the supplementary material.

Proof of Theorem 1: Suppose that the ray is going from outside to inside of the surface. Hence, we have $-(\nabla f(\mathbf{p}(t))\cdot\mathbf{v})>0$, because by convention the signed distance function $f(\mathbf{x})$ is positive outside and negative inside of the surface.

Recall that our S-density field $\phi_{s}(f(\mathbf{x}))$ is defined using the logistic density function $\phi_{s}(x)=se^{-sx}/(1+e^{-sx})^{2}$, which is the derivative of the Sigmoid function $\Phi_{s}(x)=(1+e^{-sx})^{-1}$, i.e. $\phi_{s}(x)=\Phi_{s}^{\prime}(x)$.

According to Eqn. 5 of the paper, the weight function $w(t)$ is given by

where

By assumption, $-(\nabla f(\mathbf{p}(t))\cdot\mathbf{v})>0$ for $t\in\left[t_{l},t_{r}\right]$. Since $\phi_{s}$ is a probability density function, we have $\phi_{s}(f(\mathbf{p}(t)))>0$. Clearly, $\Phi_{s}(f(\mathbf{p}(t)))>0$. It follows that

which is positive. Hence,

As a first-order approximation of signed distance function $f$, suppose that locally the surface is tangentially approximated by a sufficiently small planar patch with its outward unit normal vector denoted as $\mathbf{n}$. Because $f(\mathbf{x})$ is a signed distance function, locally it has a unit gradient vector $\nabla f=\mathbf{n}$. Then we have

where $\theta$ is the angle between the view direction $\mathbf{v}$ and the unit normal vector $\mathbf{n}$, that is, $\cos(\theta)=\mathbf{v}\cdot\mathbf{n}$. Here $|\cos(\theta)|T(t_{l})\cdot\Phi_{s}(f(\mathbf{p}(t_{l})))^{-1}$ can be regarded as a constant. Hence, $w(t)$ attains a local maximum when $f(\mathbf{p}(t))=0$ because $\phi_{s}(x)$ is a unimodal density function attaining the maximal value at $x=0$.

We remark that in this proof we do not make any assumption on the existence of surfaces between the camera and the sample point $\mathbf{p}(t_{l})$. Therefore the conclusion holds true for the case of multiple surface intersections on the camera ray. This completes the proof. $\Box$

In this section we show that the weight function derived in naive solution is biased. According to Eqn. 3 of the paper, $w(t)=T(t)\sigma(t)$, with the opacity $\sigma(t)=\phi_{s}(f(\mathbf{p}(t)))$. Then we have

Now we perform the same first-order approximation of signed distance function $f$ near the surface intersection as in Section B.1. In this condition, the above equation can be rewritten as

Here $\cos(\theta)$ can be regarded as a constant. Now suppose $\mathbf{p}(t^{*})$ is a point on the surface $\mathbb{S}$, that is, $f(\mathbf{p}(t^{*}))=0$. Next we will examine the value of $\frac{{\rm d}w}{{\rm d}t}(t)$ at $t=t^{*}$. First, clearly, $T(t^{*})>0$ and $\phi_{s}(f(\mathbf{p}(t^{*})))^{2}>0$. Then, since $\phi_{s}^{\prime}(0)=0$, we have

Hence $w(t)$ in naive solution does not attain a local maximum at $t=t^{*}$, which corresponds to a point on the surface $\mathbb{S}$. This completes the proof. $\Box$