EfficientNeRF: Efficient Neural Radiance Fields

NeRF employs implicit functions to inference the sampled points’ 4D attributes when inputting 5D spatial information, formulated as

$$(r,g,b,\sigma)=f(x,y,z,\theta,\phi),$$

(1)

where $\textbf{x}=(x,y,z)$ and $\textbf{d}=(\theta,\phi)$ denote the point location and direction in the world coordinate. The color and density attributes are respectively represented by $\textbf{c}=(r,g,b)$ and $\sigma$. $f$ is a mapping function, usually implemented by a MLP network.

For each pixel in the synthesized image, to calculate its color, NeRF first samples $N$ points $\textbf{x}_{i}(i=1,2,...,N)$ along ray r. It then calculates corresponding density $\sigma_{i}$ and color $\textbf{c}_{i}$ by Eq. (1). The final predicted color $\hat{C}(r)$ is rendered by $\alpha$-compositing [15] as

	$\displaystyle\hat{C}(\textbf{r})$	$\displaystyle=\sum_{i=1}^{N}w_{i}c_{i},$		(2)
	$\displaystyle w_{i}$	$\displaystyle=T_{i}\alpha_{i},$
	$\displaystyle T_{i}$	$\displaystyle=exp(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}),$
	$\displaystyle\alpha_{i}$	$\displaystyle=1-exp(-\sigma_{i}\delta_{i}),$

where $\delta_{i}$ denotes interval of samples along ray r.

The training objective $\mathcal{L}$ of NeRF is the mean square error between each ground-truth pixel color $C(\textbf{r})$ and the rendering color $\hat{C}(\textbf{r})$ as

$$\mathcal{L}=\sum_{\textbf{r}\in\mathcal{R}}\parallel C(\textbf{r})-\hat{C}(\textbf{r})\parallel_{2}^{2},$$

(3)

where $\mathcal{R}$ is the set of all rays shooting from the camera center to image pixels.

We define the point with location $\textbf{x}_{v}$ with density $\sigma_{v}>0$ as a valid sample, as shown in Fig. 3. For the $N$ points along ray r, suppose a point with location $\textbf{x}_{i}$ has density $\sigma_{i}=0$. Because $T_{i}=exp(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j})$ belongs to the range of $[0,1]$ and $\alpha_{i}=1-exp(-\sigma_{i}\delta_{i})=0$, we calculate its contribution $w_{i}$ to the ray color $\hat{C}(r)$ by

$$w_{i}=T_{i}\cdot\alpha_{i}=0.$$

(4)

It means the point is an invalid sample and makes no difference to the final rendering result. Therefore, it can be skipped once we know the locations. The proportion of valid samples is represented as

$$V=\frac{N_{v}}{N_{c}}.$$

(5)

We measure the percentage of area by trained NeRF that is valid over common scenes and show the numbers in Table 1. It is surprising to note that only a small portion (around $10\%$ - $20\%$) of the samples are valid. From this analysis, we conclude that it is feasible and necessary to adopt sparse and valid sampling to achieve efficient scene representation.

For a specific scene, the density of any world coordinate $\textbf{x}\in\mathbb{R}^{3}$ is shared by all rays. Thus, we construct momentum density voxels $V_{\sigma}$ with resolution $D\times D\times D$ to memorize the latest global value of density over the target scene during training.

Since each point’s density $\sigma\geq 0$, we initialize the default density value in $V_{\sigma}$ as a positive number $\varepsilon$. It means that all points in $V_{\sigma}$ are valid samples in the beginning.

For a sampled point with location $\textbf{x}\in\mathbb{R}^{3}$, we infer its coarse density $\sigma_{c}(\textbf{x})$ by the coarse MLP. Then we update the density voxels $V_{\sigma}$ by $\sigma_{c}(\textbf{x})$. We add a momentum to stabilize the values. Specifically, we first transfer x to 3D Voxels index $\textbf{i}\in\mathbb{R}^{3}$ as

i

$\displaystyle=\dfrac{\textbf{x}-\textbf{x}_{min}}{\textbf{x}_{max}-\textbf{x}_{min}}\cdot D,$

(6)

where $\textbf{x}_{min},\textbf{x}_{max}\in\mathbb{R}^{3}$ denote the minimal and maximal world coordinate borders of the scene.

Next, for every training iteration, we update the global density $\sigma$ at index i of $V_{\sigma}$ through

$$V_{\sigma}[\textbf{i}]\leftarrow(1-\beta)\cdot V_{\sigma}[\textbf{i}]+\beta\cdot\sigma_{c}(\textbf{x}).$$

(7)

Where $\beta\in[0,1]$ controls the updating rate.

Our momentum density Voxels $V_{\sigma}$ reflect the latest density distribution over the whole scene. Thus, we directly obtain the density attribute at coordinate $x$ through query rather than calculating through a MLP module. It primarily reduces the inference time and is utilized to guide a dynamic sampling process.

During training, for each ray r whose starting point is $r_{o}\in\mathbb{R}^{3}$ and normalized direction is $r_{d}\in\mathbb{R}^{3}$, the original NeRF adopts a uniform sampling strategy to obtain the sampled points as

$$\textbf{x}_{i}=r_{o}+i\delta_{c}r_{d},$$

(8)

where $i\in Z$ and $\forall i\in[1,N_{c}]$. $\delta_{c}$ is the interval between the nearest coarse sampled points along ray r.

We propose Valid Sampling to pay attention to valid samples. Specifically, instead of directly inferring all these samples, we first query the latest density from $V_{\sigma}$, and only input $\textbf{x}_{i}$ with global densities

$$V_{\sigma}[\textbf{i}]>0$$

(9)

to the coarse MLP.

Inferring a single point by a coarse MLP takes times $T_{m}$, and querying a single point from voxels takes $T_{q}$. For all sampled points along ray r, predicting their densities through a coarse MLP consumes time $N_{c}T_{m}$. Our method takes time $(N_{v}T_{m}+(N_{c}-N_{v})T_{q})$. Considering time of voxels query $T_{q}\ll T_{m}$ [35, 4], we calculate the acceleration ratio $A_{c}$ of the coarse stage by

$$A_{c}=\frac{N_{c}T_{m}}{N_{v}T_{m}+(N_{c}-N_{v})T_{q}}\approx\frac{N_{c}}{N_{v}}=\dfrac{1}{V}.$$

(10)

As illustrated in Table 1, if the proportion of valid samples $V=10\%$, the coarse stage can be accelerated by $10$ times in theory.

We define the point with location $\textbf{x}_{p}$ whose weight $w_{p}>\epsilon$ as a pivotal sample, where $\epsilon$ is a tiny threashold, as illustrated in Fig. 3.

$w_{i}$ represents the contribution of $\textbf{x}_{i}$ to the ray r’s color. The nearby area of the pivotal samples is focused to infer more detailed densities and colors. Apart from $x_{p}$, we uniformly sample $N_{s}$ points near $\textbf{x}_{p}$ along each ray r as

$$\textbf{x}_{p,j}=\textbf{x}_{p}+j\delta_{f}r_{d},$$

(11)

where $j\in Z$ and $\forall j\in[-\frac{N_{s}}{2},\frac{N_{s}}{2}]$. $\delta_{f}$ is the interval at the fine stage. Suppose there are $N_{p}$ pivotal points, the proportion of the pivotal samples can be represented by

$$P=\frac{N_{p}}{N_{c}}.$$

(12)

Similar to the coarse stage, we calculate the acceleration ratio $A_{f}$ of the fine stage as

$$A_{f}=\dfrac{N_{f}}{N_{p}N_{s}}=\dfrac{2N_{c}}{N_{p}N_{s}}=\dfrac{2}{PN_{s}}.$$

(13)

In our experiments with results listed in Table 1, if $N_{s}=5$ and $P=5\%$, our pivotal sampling strategy can accelerate the fine stage by $8$ times.

We first introduce the common high-resolution Novel View Synthesis datasets, including the Realistic Synthetic dataset [17] and the Real Forward-Facing dataset [16]. The Realistic Synthetic dataset [17] contains 8 synthetic scenes. Each scene contains 100 training images and 200 testing images, all at $800\times 800$ resolution. The Real Forward-Facing dataset [16] consists of 8 complex and real-world scenes, each has 20 to 62 images at $1,008\times 756$ resolution. We follow the same training and testing dataset split as the original NeRF [17].

We evaluate the accuracy of synthesized images via metrics including PSNR / SSIM (the higher the better), and LPIPS [37] (the lower the better), following recent methods [17, 12, 35, 14]. Moreover, we measure the training speed by total training time in terms of hour and the rendering speed by Frame Per Second (FPS). For fair comparison, we re-train their source code on the same machine and skip the evaluation time during training.

During training, for the Momentum Density Voxels $V_{\sigma}$, its resolution is set as $384\times 384\times 384$, the initial density $\varepsilon=10.0$, and the updating rate $\beta=0.1$. The degree of Spherical Harmonics is $3$, which means the output dimension of MLPs is $49$ [35].The pivotal threshold $\epsilon$ is $1\times 10^{-4}$. The learning rate is initialized to $5\times 10^{-4}$ with Adam [9] optimizer and exponentially decays by $0.1$ for every $500$K iteration when the batch size is $1024$. Finally, the total number of training iteration is $1\times 10^{6}$.

During testing, we set $D_{c}=384$ and $D_{f}=4$, which means the coarse voxels have resolution $384\times 384\times 384$, and local fine sparse voxels have resolution $4\times 4\times 4$. The quantification from MLP’s continuous coordinates to discrete ones usually weakens the performance [35]. We avoid it by directly inputting the converted discrete coordinate to the MLP during training or using linear interpolation. All our experiments are performed on a server with one RTX-3090 GPU. Please refer to the supplementary material for more experimental results.

To balance the synthesized accuracy and inference speed, we provide four versions of EfficientNeRF ($N1$-$N4$) according to the number of coarse and fine sampling parameters $N_{c}$ and $N_{s}$ in Table 3 and plot the PSNR-Speed curves in Fig. 1. Our work achieves a better rendering speed than other state-of-the-art methods like PlenOctree [35].

We explore the influence of the different sizes of the coarse and fine networks, as shown in Table 4. The baseline combination is two identical standard MLPs, which come from the original NeRF [17]. The accuracy is the best under the longest running time. The combination of two lightweight MLPs yields opposite performance, which indicates the effect of standard MLPs.

It is found that lightweight coarse MLP plus standard fine MLP yields nearly the same accuracy and fast rendering speed. It reveals that the size of fine MLP mainly determines the final synthesis quality. We thus adopt the final combination as the network of our EfficientNeRF.

As shown in Table 5, we evaluate performance of the proposed efficient modules. First, representing the color in different directions by Spherical Harmonic (SH) [35] is in favor of the performance. Second, our lightweight coarse MLP and coarse valid sampling accelerate the training process. Third, our fine pivotal sampling further reduces the training time and improves synthesis accuracy. It performs better than the original probabilistic sampling of original NeRF [17].

We compare our efficient sampling with the common uniform sampling and original sampling of NeRF. The result is presented in Table 6. First, uniform sampling is adopted by IBRNet [32] and MVSNeRF [2]. It is faster than NeRF sampling while achieving lower accuracy. Second, NeRF [17] and PlenOctree [35] adopt NeRF sampling and achieve high performance. However, the training time is very long. Finally, our efficient sampling successfully accelerates all these methods while achieving comparable or even better accuracy.

Before applying NerfTree, other data structures, such as dense voxels, sparse tensor [3], and Octree [35], can also be used to cache the trained scene for fast synthesis of novel views. To compare their efficiency, we calculate memory consumption and running time when storing and querying a whole scene with resolution $1,024\times 1,024\times 1,024$ and $20\%$ of valid space. Each voxel has a 4D feature $(r,g,b,\sigma)$ with the same data type.

The results are shown in Table 7, Dense voxels representation spends the least time in caching and querying while requiring $16.0$ GB memory. The required memory by Sparse Tensor (Minkowski Engine [3]) is the smallest. But as it adopts hash structure making caching and query longer process. PlenOctree [35] balances memory consumption and query time. However, its caching time is long because of its internal optimization. Our NerfTree representation performs the best with all these three aspects. It does not require much storage memory, and cache and query 3D points at a very fast speed.