Ray-Distance Volume Rendering for
Neural Scene Reconstruction

Our goal is to recover the scene geometry given a set of projected 2D views of a 3D scene. To achieve this, NeRF [23] optimizes a continuous 5D function to represent the scene, which predicts the volume density $\sigma({\mathbf{p}})$ and view-dependent color $\textbf{c}({\mathbf{p}},{\mathbf{r}})$ for each sampled point ${\mathbf{p}}\in\mathbb{R}$ and ray direction ${\mathbf{r}}\in\mathcal{S}^{2}$. With $\Omega\subset\mathbb{R}^{3}$ denoting the scene geometry filled with physical presence, the predicted density $\sigma({\mathbf{p}})$ approximates the ground truth scene density $\mathbf{1}_{\Omega}(\textbf{p})$, such that $\mathbf{1}_{\Omega}(\textbf{p})=1$ if $\textbf{p}\in\Omega$ and 0 otherwise. After that, the classic volume rendering is employed to render the color ${\textbf{C}}({\mathbf{r}})$, and $N$ points $\{\textbf{p}_{i}=\textbf{o}+z_{i}\mathbf{r},i\in[1,N]\}$ are sampled along the camera ray ${\mathbf{r}}$, with $z_{i}$ indicating the depth from the camera center o to the sampled point. The color is accumulated along the ray:

where $\delta_{i}$ is the interval of neighboring points.

Despite the great success in novel view synthesis, NeRF has difficulty reconstructing satisfactory actual surfaces from the generated volume density. To address this challenge, recent methods [46, 50] propose to predict the SDF and then transform it to volume density, which aligns with the geometry bias that the surface points have a higher density than other non-surface points and corresponds to a higher weight during volume rendering.

For any sampled point $\textbf{p}\in\mathbb{R}^{3}$, the SDF $d_{\Omega}(\textbf{p})$ is with an absolute value representing the shortest distance from p to the surface $\mathcal{M}=\partial\Omega$, while the sign indicates whether the point is outside (positive) or inside (negative) the surface. Formally,

where the indicator function $\mathbf{1}_{\Omega}(\textbf{p})=1$ if $\textbf{p}\in\Omega$ and 0 otherwise. This indicates that the SDF considers every surface around the point. To apply SDF to volume rendering, Yariv et al. [46] suggest to derive the volume density $\sigma^{\text{SDF}}$ from the signed distance as follows:

in which $\alpha,\beta$ are learnable parameters. ${\rm{\Psi}}_{\beta}$ is the Cumulative Distribution Function (CDF) of the Laplace distribution with zero mean and $\beta$ scale. Then, $\sigma^{\text{SDF}}(\textbf{p})$ is applied to Eq. 1 and as the density function $\sigma$.

We illustrate the issues with volume rendering SDF by a toy example in Fig. 1. Indoor scenes typically include multiple objects. Along the camera ray $\mathbf{r}$, the SDF may be influenced by surfaces not intersecting the ray, resulting in several ambiguous local maxima of false high volume density and weight. This implies that the 2D observations may have a strong correlation to these distant points, contradicting the fact that the points closer to the surface along the camera ray may carry more significance.

To this end, we propose volume rendering scene density function from SRDF, the distance field local to the ray direction. In contrast to the SDF in Eq. 2, the SRDF $\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}(\mathbf{p},\mathbf{r})$ [52, 30] computes the shortest distance from the point to the surface along the sampled ray $\mathbf{r}$, i.e. the SRDF is ray-dependent. Formally, the SRDF $\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}(\mathbf{p},\mathbf{r})$ can be represented by

where $\mathbf{1}_{\Omega}(\textbf{p})=1$ if $\textbf{p}\in\Omega$ and 0 otherwise. In the following sections, $d_{\Omega}$ from Eq. 2 and $\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}$ are used to denote SDF and SRDF for brevity.

In Fig. 1, it can be seen that the weight computed from SRDF in the volume rendering only reaches a local peak around the surface point, aligning more consistently with the nature of the 2D observations. Therefore, this paper proposes to employ the ray-specific SRDF to yield the volume density instead of the SDF. This work predicts both SDF and SRDF, where SDF is mainly used to model the surface. As depicted in Fig. 2, for a sampled point p, a geometry MLP $f_{g}$ is applied on the input (including the encoded position or features) to predict SDF $d_{\Omega}$ and geometry feature $\textbf{F}_{g}$. By definition, SRDF is view-dependent. Hence, the viewing direction $\mathbf{r}$, geometry feature $\textbf{F}_{g}$, and point position p are concatenated and then passed through a SRDF MLP $f_{s}$ to yield the SRDF. On the other hand, the viewing direction $\mathbf{r}$, geometry feature $\textbf{F}_{g}$, point position p, and 3D unit normal $\textbf{n}=\nabla{d_{\Omega}}$ are used as input to a color MLP $f_{c}$ to output color c.

For volume rendering, our method adopts the predicted SRDF $\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}$ to derive the volume density $\sigma$. This is defined by:

Then, $\sigma^{\text{SRDF}}(\textbf{p},\mathbf{r})$ is applied to the volume rendering in Eq. 1 as a ray-conditioned density function to obtain the color value ${\textbf{C}}^{\text{SRDF}}({\mathbf{r}})$. Besides, during training, we also render color ${\textbf{C}}^{\text{SDF}}({\mathbf{r}})$ with density functions from SDF to obtain gradient signals to optimize the prediction of SDF, in which the volume rendering processes use the same rays and points as those for ${\textbf{C}}^{\text{SRDF}}({\mathbf{r}})$.

For the scene geometry, after the network is trained with NeRF optimization targets, the geometric surface $\mathcal{M}=\partial\Omega$ is extracted as watertight meshes by the marching cubes algorithm [19, 28] from the SDF results $d_{\Omega}$, which only depends on the query location ${\mathbf{p}}$.

Following the definitions in Eq. 4, despite SRDF and SDF being defined differently, they share the same sign, which indicates the spatial occupancy — positive outside the surface and negative inside the surface. However, in this paper, where SDF and SRDF are predicted from separate branches, sign consistency is not guaranteed. To deal with this issue, a SRDF-SDF consistency loss is proposed to enforce this constraint.

Although the sign function effectively computes the output’s sign, it cannot be differentiated during back-propagation. To a certain degree, the values derived from the sigmoid function approximate those calculated by the sign function. Thus, in our SRDF-SDF consistency loss, the sigmoid function is first employed to generate the sign of SRDF and SDF as the approximated sign function $\varsigma(\cdot)$, after which a $\ell_{2}$ loss is adopted to quantify the sign difference, as defined by:

in which $k$ is a hyperparameter, controlling the slope of the sigmoid function. The indicator function $[\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}\cdot{d}_{\Omega}<0]$ is leveraged to identify whether the generated SDF and SRDF have opposite signs. Therefore, the consistency loss imposes penalties when there is a difference between the signs of SRDF and SDF. ${N_{r}}$ represents the number of sampled points in a minibatch that satisfy $[\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}\cdot{d}_{\Omega}<0]$.

Taking the derivative of $\mathcal{L}_{con}$ with respect to SDF ${d}_{\Omega}$ (as an example), is computed as follows:

The derivative in Eq. 7 shows that our consistency loss has two advantages: (1) Unlike the sign function, which assigns the same penalty for outputs with different signs of SRDF and SDF, our consistency loss can adjust the penalty according to the extent of the sign discrepancy between SRDF and SDF, referring to $||\varsigma(\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}})-\varsigma({d}_{\Omega})||_{1}$. (2) The derivative of the sigmoid function $\frac{\partial{\varsigma({d}_{\Omega})}}{\partial{d}_{\Omega}}$ peaks at zero and decreases from 0 to $+\infty/-\infty$. Consequently, predictions in proximity to zero (indicating the surface) may exhibit higher absolute gradient values, providing effective supervision for points near the surface.

Theoretically, with visibility attributes of 3D points, surface location becomes straightforward. Therefore, to enhance 3D geometry prediction, we propose a self-supervised visibility task in this paper. Along the trajectory of a camera ray, points sampled prior to reaching the first surface point are categorized as visible, whereas points sampled beyond the first surface point are classified as occluded. To find the first surface point along the ray, using SRDF as an example, the multiplication of SRDF between adjacent sampled points is computed. When $\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}^{i}\cdot\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}^{i+1}\leq 0$ and ${\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}^{m}\cdot\stackrel{{\scriptstyle\sim}}{{\smash{d_{\Omega}}\rule{0.0pt}{4.73611pt}}}^{m+1}>0,m\in[1,i-1]}$, it indicates that the first surface is within the interval $[\textbf{o}+z_{i}\mathbf{r},\textbf{o}+z_{i+1}\mathbf{r}]$. Consequently, the points between $[\textbf{p}_{1},\textbf{p}_{i}]$ are considered physically visible while points between $[\textbf{p}_{i+1},\textbf{p}_{N}]$ are regarded as occluded. However, the SRDF or SDF predicted by the network may be inaccurate at some points, introducing potential noise in the visibility task. To improve the accuracy of the visibility labeling, both SDF and SRDF are utilized in determining visibility. Points are labeled as visible or occluded only when there is agreement between SRDF and SDF on their classification. If the classifications from SRDF and SDF diverge, suggesting difficulties in accurately categorizing these points for the network, such points are omitted from the visibility task. The method for assigning visibility labels is defined as follows

where $V^{\text{SRDF}}/V^{\text{SDF}}$ are the visibility labels determined based on the prediction of SRDF/SDF. $1/0$ means label for visible/occluded points. The formation of the visibility label can be inferred as self-supervised, thus eliminating the necessity for multi-view geometry and additional annotations. Additionally, it incorporates information from both SRDF and SDF.

From the analysis above, similar to SRDF, the visibility task also relies on the view direction. Therefore, a visibility probability $V_{pred}$ is predicted from the SRDF branch. It is a binary classification task, i.e. visible or occluded, so the binary cross-entropy loss is employed as visibility loss $\mathcal{L}_{vis}$ during optimization.

The visibility task can discern the point’s visibility along the camera ray, thereby influencing the learning of SRDF. Moreover, the optimization process integrates knowledge of ray-related visibility across the entire space, further impacting the learning of SDF. Notably, unlike 2D supervision, the visibility task provides priors in 3D space, imparting a more direct influence on 3D geometry.

Tab. 5 presents an ablation study to evaluate the effectiveness of our design on ScanNet with MLP representation. The ablation experiments consist of: (a) Baseline: MonoSDF. (b) RS. Den.: this structure models the volume density using the ray-specific SRDF, without the SRDF-SDF consistency loss and self-supervised visibility task. (c) RS. Den. + Con. L.: Building upon the structure in (b), this setup adopts the proposed SRDF-SDF consistency loss. (d) RS. Den. + Con. L. + Vis.(SDF): based on (c), this configuration adds the self-supervised visibility task, in which only SDF is used to compute the visibility label. (e) RS. Den. + Con. L. + Vis.(SRDF): Different from structure (d), this method relies solely on SRDF to compute the visibility label. (f) RS. Den. + Con. L. + Vis.: this structure represents the comprehensive model of our method, wherein the visibility labels integrate priors from both SRDF and SDF predictions.

The comparison between configurations (a) and (b) highlights that our design using SRDF to model the density, without any additional constraints, results in a better performance compared to MonoSDF. The SRDF-SDF consistency loss in (c) surpasses the structure in (b) by 3.1% in F-score. This shows the significance of sign consistency. Compared to (c), the self-supervised visibility task in (d)-(f) enhances the reconstruction performance. In particular, structure (f) obtains a better F-score than structures (d) and (e). These explain the effectiveness of our self-supervised visibility task and the computation of visibility labels integrating both SRDF and SDF priors. The visualization of the ablation study is given in the supplementary material.

Furthermore, Fig. 5 compares the weight distribution for the red and yellow sampled points generated by MonoSDF and our method. (1) The analysis of the yellow sampled point demonstrates the benefits of our method for reconstruction: MonoSDF generates high weights and negative SDF values when sampled rays are near the white wall, resulting in false surfaces that significantly deviate from the ground truth. In contrast, our method yields more accurate surfaces and thus enhances both completeness and recall. (2) The analysis of the red sampled point explains the reason why our method can generate accurate views: MonoSDF produces weights with two local maxima, causing an inaccurate RGB value. This occurs because the SDF along the camera ray fluctuates, resulting in varying density and weight. In comparison, our method produces uni-modal weights that peak only around the surface, resulting in more accurate views. The quantitative PSNR in Fig. 5b and Fig. 5d confirms that our rendered image is more accurate compared to MonoSDF. These findings align with our motivation to achieve a more consistent and accurate weight distribution, thereby reconstructing better surfaces and generating high-quality views. This underscores the effectiveness of using ray-specific SRDF to model volume density in our approach.