Evaluate Geometry of Radiance Fields with Low-frequency Color Prior

A radiance field has been popular for representing 3D objects or scenes since the Neural Radiance Field (NeRF) (Mildenhall et al. 2020) demonstrated that the performance of novel-view synthesis could be dramatically improved benefiting from it. From this path-breaking work, many works aimed at improving NeRF from several aspects, such as various challenging scenarios (Zhang et al. 2020; Martin-Brualla et al. 2020; Peng et al. 2021; Barron et al. 2021; Mildenhall et al. 2022), inference speed (Yu et al. 2021a; Hedman et al. 2021; Reiser et al. 2021; Garbin et al. 2021; Chen et al. 2022b), training efficiency (Deng, Barron, and Srinivasan 2020; Hu et al. 2022; Yu et al. 2022; Sun, Sun, and Chen 2022a, b; Müller et al. 2022; Chen et al. 2022a; Wang et al. 2023b, a), generalization ability (Yu et al. 2021b; Wang et al. 2021b; Chen et al. 2021), and so on. Some other works proposed restoring the surfaces via reconstructing the latent radiance field by inverse volume rendering (Wang et al. 2021a; Long et al. 2022; Oechsle, Peng, and Geiger 2021; Sun et al. 2022; Zhang et al. 2021; Wang et al. 2022; Wu et al. 2022a, 2023).

As the ground-truth of a radiance field is hard to obtain, we could not directly evaluate the reconstructed result. Alternatively, novel-view synthesis methods usually measure the similarity between the rendered image and its observed counterpart using image quality metrics such as peak signal-to-noise ratio (PSNR). As shown in the top two rows of Fig. 1, this metric may be enough to evaluate the synthetic images but not appropriate to evaluate the geometry.

Besides novel-view synthesis methods, some algorithms aim at improving the reconstructed geometry, e.g., distortion loss (Barron et al. 2022). And a lot of works exploit radiance fields for 3D reconstruction (Wang et al. 2021a; Long et al. 2022; Oechsle, Peng, and Geiger 2021; Sun et al. 2022; Zhang et al. 2021, 2022; Wu et al. 2022b; Wang et al. 2022; Wu et al. 2022a, 2023). These methods need a proper metric to quantitatively evaluate the geometry result. To this end, surface reconstruction methods, e.g., (Wang et al. 2021a; Long et al. 2022), usually transform the density field into a point-cloud using the marching cubes algorithm and compute Chamfer Distance (CD) with the scanned ground-truth. However, the scanning process along with the equipment is expensive and complicated. Therefore, few datasets have this ground-truth, and many widely-used ones do not.

To alleviate difficulties above, our key observation is that the color of any point in an ideal radiance field tends to be low-frequency if it is on the ground-truth surface of an object. This phenomenon is named as low-frequency color prior. Besides this, we adopt a closed-form method to compute the color field given the observation images and density field of a scene. We name the result as the computed color field to distinguish it from the reconstructed one given by the radiance field reconstruction method. Via exploring the computed color field, we find that the low-frequency color prior may be invalid for the points with inaccurate density values. Therefore, the density field could be evaluated by the mean frequency of a computed color field.

However, it is difficult to directly evaluate the color frequency even for a single point, not to mention the whole field. To this end, we approximate the color with low-frequency spherical harmonics and compute the residual color. Then, smaller mean residual color (MRC) implies lower color frequency, which indicates better geometry. Moreover, as the MRC is usually very small, it is not convenient for presentation and comparison. Thus, we transform it into decibel (dB) as PSNR does and name it as inverse mean residual color (IMRC). Then the higher the IMRC, the better the geometry. As demonstrated in Fig. 1, our IMRC metric could quantitatively evaluate the geometry correctly.

Our main contributions are concluded as follows: 1) We present the low-frequency color prior via analysing the ideal radiance field. We further find that this prior may be invalid for the computed color field if the density field is inaccurate. 2) To quantitatively evaluate the prior, we propose to approximate the color with low-frequency spherical harmonics and design the inverse mean residual color as a new metric. Qualitative and quantitative experimental results verify its effectiveness. 3) We further benchmark several state-of-the-art radiance field reconstruction methods using inverse mean residual color to promote future related research.

We firstly review two types of radiance field reconstruction works, and then discuss the geometry metrics.

Novel View Synthesis. Mildenhall et al. (Mildenhall et al. 2020) proposed NeRF to synthesize novel-view images from posed images and dramatically improved the performance. They adopted a multilayer perceptron (MLP) to present the radiance field, which contains two components, i.e., the density field and color field. Each point in the density field has a scalar controlling how much the color is accumulated. Each point in the color field encodes the view-dependent color. The image can be rendered using the volume rendering algorithm (Max 1995). From this path-breaking work, there have been many works aiming at improving NeRF from several aspects, such as various challenging scenarios (Zhang et al. 2020; Martin-Brualla et al. 2020; Barron et al. 2021; Peng et al. 2021; Jain, Tancik, and Abbeel 2021; Mildenhall et al. 2022; Tancik et al. 2022; Weng et al. 2022; Zhao et al. 2022; Shao et al. 2022; Niemeyer et al. 2022), inference speed (Yu et al. 2021a; Hedman et al. 2021; Reiser et al. 2021; Garbin et al. 2021; Chen et al. 2022b), training efficiency (Deng, Barron, and Srinivasan 2020; Hu et al. 2022; Yu et al. 2022; Sun, Sun, and Chen 2022a, b; Müller et al. 2022; Chen et al. 2022a; Wang et al. 2023b, a), generalization ability (Yu et al. 2021b; Wang et al. 2021b; Chen et al. 2021), and so on.

Since these works focused on novel view synthesis tasks, to evaluate the results, they usually adopted image quality metrics, such as PSNR, SSIM, and LPIPS. These metrics are suitable for the task. However, it remains questionable how to evaluate the radiance field itself and whether these metrics are adequate. Evaluating the radiance field directly is challenging due to the difficulty in obtaining its ground-truth. Existing works have not well explored this problem.

Surface Reconstruction. Some other works proposed to restore the surfaces via reconstructing the latent radiance field by inverse volume rendering (Wang et al. 2021a; Oechsle, Peng, and Geiger 2021; Sun et al. 2022; Wu et al. 2022b; Zhang et al. 2022; Wu et al. 2022a, 2023). These methods usually obtain the surfaces via transforming the latent density field into an occupancy field or signed distance field (SDF). Then the Chamfer Distance between the points on the surfaces and the ground-truth can be computed to evaluate the results. However, it is expensive and complicated to set up the hardware environment and scan the ground-truth point-cloud. Moreover, during the transformation process, some information would be discarded as the transformed result only contains the iso-surface. Therefore, the CD metric may not be suitable to evaluate a density field. In contrast, IMRC is not suitable to evaluate the surfaces, but it could evaluate the density field.

Geometry Metrics. Due to the unavailable density field’s ground-truth, we can not use direct evaluation metrics, such as mean squared error or mean absolute error. Therefore, only indirect metrics could be considered. In previous Structure-from-Motion (SfM) systems, e.g., (Hartley and Zisserman 2004; Snavely, Seitz, and Szeliski 2006, 2008; Agarwal et al. 2009), the mean re-projection error is well-known and widely adopted as a metric and optimization objective to evaluate the reconstructed structures and motions. The mean re-projection error is defined as the mean distance between each observed image feature point and the re-projection point of its reconstructed 3D point. In other words, it could evaluate the reconstructed geometry without the corresponding ground-truth. Inspired by this, we argue that the geometry of a radiance field also could be evaluated only with observation images.

According to NeRF (Mildenhall et al. 2020), a radiance field $\mathcal{F}$ is defined on a 3D space $\mathbf{V}$, which has two components, i.e., the density field $\mathcal{F}^{\sigma}$ and the color field $\mathcal{F}^{c}$. For a 3D point $\mathbf{v}$ in the space $\mathbf{V}$, the density $\mathcal{F}^{\sigma}_{\mathbf{v}}$ is a scalar that controls how much color of this point is accumulated. And the color $\mathcal{F}^{c}_{\mathbf{v}}$ encodes all view-dependent color information. For a specific 3D direction $\mathbf{d}$, the color vector is denoted as $\mathcal{F}^{c}_{\mathbf{v},\mathbf{d}}$. To render the color observed from a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ with near and far bounds $t^{n},t^{f}$, we can follow the function

$${C}(\mathbf{r})=\int_{t^{n}}^{t^{f}}T(t)\mathcal{F}^{\sigma}_{\mathbf{r}(t)}\mathcal{F}^{c}_{\mathbf{r}(t),\mathbf{d}}~{}dt,$$

(1)

where $\mathbf{o}$ is the camera original point, $\mathbf{d}$ is the direction from $\mathbf{o}$ to the corresponding pixel center, and the transmittance

$$T(t)=\exp\left(\int_{t^{n}}^{t}-\mathcal{F}^{\sigma}_{\mathbf{r}(s)}~{}ds\right).$$

(2)

To numerically estimate this continuous integral, NeRF resorts to quadrature (Max 1995). Then rendering equation is

$$\hat{{C}}(\mathbf{r})=\sum_{i=1}^{N}\hat{T}(t_{i})\left(1-\exp(-\mathcal{F}^{\sigma}_{\mathbf{r}(t_{i})}\delta_{\textbf{r}(t_{i})})\right)\mathcal{F}^{c}_{\mathbf{r}(t_{i}),\mathbf{d}},$$

(3)

where $N$ is the number of sample points along the ray,

$$\hat{T}(t_{i})=\exp\left(-\sum_{j=1}^{i-1}\mathcal{F}^{\sigma}_{\mathbf{r}(t_{j})}\delta_{\textbf{r}(t_{j})}\right),$$

(4)

and $\delta_{\textbf{r}(t_{i})}$ is the distance between the point $\textbf{r}(t_{i})$ and its previous neighbour $\textbf{r}(t_{i-1})$. Specifically, $\delta_{\textbf{r}(t_{1})}=0$.

To illustrated the low-frequency color prior, we construct an example scene in Fig. 2 (a), which consists of a cube. For better understanding, we also refer to the well-known related conception, i.e., light field, which describes the amount of light flowing in every direction through every point in space (Ng 2005). The difference between a light field and the color field of a radiance field is that, if a point has a zero density, its color in the color field could be arbitrary, but its color must be determined in the light field. If a point has a non-zero density, the colors in these two fields would be identical. Besides, a light field has no density information. As illustrated in Fig. 2 (b), the color frequency of a point tends to be lower when it approaches a surface. As the ground-truth density field should be the surface of the cube, and we do not care about the density values inside the surface as they do not affect the shape, a straight-forward hypothesis is that, for an ideal radiance field, the transmittance-density-weighted mean of color frequency should be small. This prior widely exists for most types of surface whether it is rough or glossy.

Moreover, we find that a wrong shape will break the low-frequency color prior. To illustrate this, we present an example scene in Fig. 3 (a), its reconstructed density field in Fig. 3 (b), and combine the scene and the reconstructed density field in Fig. 3 (c). Intuitively, as demonstrated in Fig. 3 (c), the observation color of a point from a ray should be identical with the color at the intersection of this ray and ground-truth surface. The color frequency of the points on wrong surfaces that lie inside ($\textbf{v}_{1}$) or outside ($\textbf{v}_{2}$) the ground-truth tends to be higher than that of the ground-truth surface ($\textbf{v}_{0}$). Note that we use a 2D graph here for simplicity. All analyses and conclusions are applicable for 3D cases.

From these observations, if we have the color frequency field computed from a density field and observation images, the transmittance-density-weighted mean color frequency should be small for a good reconstructed density field. However, it is not easy to figure out the color frequency of even a single point, not to mention the frequency field. To alleviate this difficulty, we resort to the conjugate problem. According to the frequency domain transformation theory, if a signal is low-frequency, we could well approximate the signal with a group of low-frequency basis, and the residual would be quite small. Otherwise, the residual would be large. Therefore, the residual could be adopted to replace the frequency. Suppose that the residual color of a point $\mathbf{v}$ at a direction $\mathbf{d}$ is $\mathcal{F}^{r}_{\mathbf{v},\mathbf{d}}$, and the transmittance of $\mathbf{v}$ along $\mathbf{d}$ is $T_{\mathbf{v},\mathbf{d}}$, the transmittance-density-weighted mean residual color

$${r}_{\mathbf{V}}=\frac{\int_{\mathbf{v}}\int_{\mathbf{d}}T_{\mathbf{v},\mathbf{d}}\mathcal{F}^{\sigma}_{\mathbf{v}}\mathcal{F}^{r}_{\mathbf{v},\mathbf{d}}~{}d\mathbf{v}d\mathbf{d}}{\int_{\mathbf{v}}\int_{\mathbf{d}}T_{\mathbf{v},\mathbf{d}}\mathcal{F}^{\sigma}_{\mathbf{v}}~{}d\mathbf{v}d\mathbf{d}}$$

(5)

should be small for a good reconstructed density field. We use the product of transmittance and density, i.e., $T_{\mathbf{v},\mathbf{d}}\mathcal{F}^{\sigma}_{\mathbf{v}}$, as the residual color’s weight, which is the same as the color weight used in the rendering equation (1). In this way, only the points near the reconstructed surface are of the interest. The high color frequency of a point inside the reconstructed surface is not considered, as it does not affect the shape.

In this section, we present the algorithm to compute the inverse mean residual color. The difficulty is that we could not use the color field reconstructed by the radiance field reconstruction method. If so, the geometry is not evaluated individually. To overcome this, we propose to utilize the observation images. For a 3D point, its projection points on all images are its observations. We can approximate its observations with a group of low-frequency basis, and the residual color could be obtained. We illustrate the whole pipeline in Fig. 4 and first introduce the observation acquisition process, then transform them into the low-frequency domain, and compute the IMRC at last.

In this section, we first verify the effectiveness of IMRC metric, then explore the influence of different settings. Finally, we benchmark several state-of-the-art methods.

We first discuss the limitations of IMRC and then conclude this paper in the following.

Limitations. There are also some degrade situations where the proposed IMRC metric may fail. For example, the low-frequency color prior no longer holds for surfaces that are purely specular or exhibit high levels of specular reflection, such as mirrors, leading to the failure of IMRC. Another degrade situation is that, if the whole density field is empty except for a few points that have been observed only by several cameras, the IMRC metric will be high. However, this density field is really bad. This type of degrade situations also exist for other geometry metrics, such as the re-projection error. Therefore, when using the proposed IMRC metric, it should be combined with other metrics, such as PSNR and SSIM. Only in this way, we can evaluate the results with less bias.

Conclusion. In this paper, we aim at evaluating the geometry information of a radiance field without ground-truth. This problem is important as the radiance field has been widely used not only on novel view synthesis but also 3D reconstruction tasks. For 3D reconstruction tasks, the geometry information is essential. However, due to the unavailable ground-truth and unsuitable metrics, there is no proper metric to quantitatively evaluate the geometry. To alleviate this dilemma, we propose the Inverse Mean Residual Color (IMRC) metric based on our insights on the properties of the radiance field. Qualitative and quantitative experimental results verify the effectiveness of the IMRC metric. We also benchmark $4$ state-of-the-art methods on $3$ datasets and hope this work could promote future related research.

This work was supported in part by the National Key Research and Development Program of China under Grant 2021YFB3301504, in part by the National Natural Science Foundation of China under Grant U19B2029, Grant U1909204, and Grant 92267103, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2021B1515140034, and in part by the CAS STS Dongguan Joint Project under Grant 20211600200022.