X-NeRF: Explicit Neural Radiance Field for
Multi-Scene 360^∘ Insufficient RGB-D Views

Existing NeRF-like methods are all essentially implicit, though some such as [39, 48, 23] take use of explicit structures. In spite of their promising performance in many situations, implicit models struggle with three challenges. The first is that when there are insufficient, i.e. extremely sparse and almost non-overlapping, seen views, they tend to overfit to a trivial solution since they have no constraints nor priors on the scene structures. The second is that it is hard for them to naturally process multiple scenes using one model as they directly encode the scene information in model parameters. A few existing NeRF-based methods that support multi-scene learning either need reference views [44, 50] whose number and density have impact on the rendering effect, or employ independent explicit structures before a shared MLP [23] whose space and time costs linearly increase with the rise of scene number. The third is that they usually lack the ability to fuse RGB with depth images which are increasingly popular and available at present.

To this end, we propose a fully explicit approach that can tackle the challenging problem of novel view synthesis from multi-scene 360^∘ insufficient seen views, and can naturally fuse RGB and depth modals. We consider the problem as an explicit completion task motivated by the fact that given a few RGB-D views around, we can easily get a large part of the location and color information of points in a specific space, and what we need to do is to complete the whole space, i.e. the explicit neural radiance field. Therefore, our network can be modeled as a completion mapping function:

Here $f_{\theta}$ represents our neural network with learnable parameters $\theta$, which is a completion mapping from $N$ input RGB-D points consisting of input point cloud coordinates $\{{\mathbf{x}_{\mathrm{in}}}_{i}\in\mathbb{R}^{3}\}_{i=1}^{i=N}$ and colors $\{{\mathbf{c}_{\mathrm{in}}}_{i}\in\mathbb{R}^{3}\}_{i=1}^{i=N}$, to $M$ explicit neural radiance field points where each $\sigma\in\mathbb{R}$ denotes a scalar opacity while we apply $\mathbf{k}$, a vector of spherical harmonic (SH) coefficients, to express output color information similar to [48, 49]. Each $k_{\ell}^{m}\in\mathbb{R}^{3}$ is a set of 3 coefficients for RGB channels. It has been discussed in [48] that spherical harmonics of degree 2 is enough, which requires 9 coefficients per color channel for a total of 27 harmonic coefficients per voxel, and we follow their setting. The SHs enable the output colors ${\mathbf{c}_{\mathrm{out}}}$ to be view-dependent by querying the SH functions $Y_{\ell}^{m}:\mathbb{S}^{2}\mapsto\mathbb{R}$ given its corresponding view direction $\mathbf{d}$:

We note here that SHs are vital to novel view synthesis under insufficient-view condition. More detailed discussions are illustrated in Sec. 4.5 and Fig. 5.

Processing RGB-D data is a classic multi-modal problem. In this paper, we convert multi-view RGB-D inputs to colorful point clouds, and voxelize them into Minkowski sparse tensors [4] so that we can directly apply operations like convolution and transposed convolution on them.

As shown in Fig. 2, we apply an encoder-decoder architecture with skip connections, similar to the structure of UNet [34]. The sparse tensor encoder is mainly used for extracting spatial and local features, which is composed of some convolution and residual blocks, while the generative sparse tensor decoder consisting of generative transposed convolution, pruning and residual blocks mainly plays a role of up-sampling and generating new points. The decoder is designed to have multi-stage outputs with increasing resolutions. The outputs of every stage participate in loss computation, which is inspired by the coarse-to-fine design in most of NeRF-related methods.

The encoder only operates on known coordinates. In other words, no new coordinates are generated during encoding. When decoding, we apply generative transposed convolutional layers followed by pruning layers to up-sample and complete the whole radiance field. A simple low-dimension schematic diagram of the encoding-decoding pipeline is illustrated in Fig 3. As we can see, the function of pruning is to remove redundant points to save computational resources as well as make the output more accurate. Details for generative transposed convolution layer can be found in [4, 13].

The multi-stage design of decoder is also beneficial to decide which points to keep or to prune. Specifically, in each stage, we prune a point $P_{i}$ if its corresponding output termination probability $\alpha_{i}$ is too small and it is too far away from the input point set:

where $\tau_{\alpha}$ and $\tau_{\mathrm{dist}}$ are two hyper-parameters; $\mathcal{C}_{\mathrm{in}}$ denotes the coordinate set of input point cloud; $\operatorname{dist}(\cdot,\cdot)$ represents the operation that computes the Euclidean distance between two points. The intention of the second distance term is that points too far away are not likely to belong to the complete scene. In the practice, we discover that the pruning operation can reduce memory consumption and has little impact on performance, but it may slightly affect the execution speed, so we leave it as an optional choice.

Given a few RGB-D images, we can project and voxelize them into a sparse tensor $\mathscr{T}_{\mathrm{in}}$ consisting of a set of coordinates $\mathcal{C}_{\mathrm{in}}$ and the corresponding features $\mathcal{F}_{\mathrm{in}}$:

where ${\mathbf{c}_{\mathrm{in}}}_{i}\in\mathbb{R}^{3}$ represents the 3-channel RGB colors.

Let’s denote our X-NeRF model as $f_{\theta}$, then we can get an output expanded sparse tensor that contains the information of the completed scene, as discussed in Eq. LABEL:eq:modeling:

where $\ddot{\sigma}_{j}\in\mathbb{R}$ denotes the raw density before activation.

Sun et al. [39] has shown that post-activation, i.e. implementing activation function after interpolation operation, is the best choice for volumetric rendering and we follow this setting. As discussed in Sec. 3, given view directions $\mathbf{d}$ and shooting rays $\mathbf{r}$ corresponding to specific 2D pixels, we first interpolate $\mathscr{T}_{\mathrm{out}}$ on $\mathbf{r}$ to obtain the SHs $\mathbf{k}$ and raw densities $\ddot{\mathbf{\sigma_{r}}}$ on them, and then use the shifted $\operatorname{Softplus}$ mentioned in Mip-NeRF [1] to acquire the corresponding densities:

where the shift $b$ is a hyper-parameter. The RGB colors on each ray point in radiance field can be gotten through Eq. 4. After that, we can apply Eq. 1 and Eq. 2 to render the 2D pixel colors $\hat{C}(\mathbf{r})$ and depths $\hat{D}(\mathbf{r})$.

Since all the operations in our pipeline is differentiable, we can optimize X-NeRF through gradient decent. To combat the overfitting issue arising from insufficient views, loss function composed of the following parts is applied.

Rendering Loss. Given a set of RGB-D inputs and camera poses $\mathbf{P}$, the rendering loss is given by the mean squared error (MSE) between ground-truth and rendered outputs:

where $\mathcal{R}(\mathbf{P})$ is the set of rays of $\mathbf{P}$ while $\mathcal{R}_{\mathrm{VD}}(\mathbf{P})\subseteq\mathcal{R}(\mathbf{P})$ contains rays with valid depths. $\lambda_{D}$ is a hyper-parameter.

Perceptual Loss. Besides the vanilla MSE loss, we further add a perceptual loss considering per-pixel MSE error contains no global or high level context information, which may not guide the model to the right road. In practice, lower MSE does not necessarily mean better human perceptual quality. We display intuitive and simple examples in Fig 4 for the above two cases. To this end, motivated by many image generation works, we adopt the perceptual loss [52] to avoid falling into a trivial solution:

where $\hat{y}^{l},\hat{y}_{0}^{l}\in\mathbb{R}^{H_{l}\times W_{l}\times C_{l}}$ are the channel-dimension unit-normalized $l$-th layer feature stack of rendered and reference image patches, which is extracted from $L$ layers of a fixed pre-trained neural network such as VGG [37]. Note that to use the perceptual loss we need to sample rays in an image patch level.

Combining the above, we can get an overall loss:

where $\lambda_{\mathrm{percep}}$ is a weighting hyper-parameter.

Lastly, as mentioned in Sec. 4.2, we have multi-stage outputs, so the final total loss is:

where $\lambda_{\mathrm{stage}}^{s}$ is the weight coefficient of stage $s$.

We utilize spherical harmonics to make rendered colors view-independent (see Sec. 4.1). Nevertheless, due to the sparsity and small quantity of training views, we observe in the experiments that the SH coefficients are not fitted well on novel view directions. To handle this limitation, we apply random rotation augmentation on input point clouds to manually simulate unseen view directions. We find that this simple operation can significantly improve the quality of rendered unseen view images. Fig. 5 shows the effectiveness of implementing SH, perception loss and view augmentation through random rotation on point clouds.

Most fully implicit coordinate-based NeRF methods struggle with the rendering efficiency because they have to run networks repeatedly on each position on each ray of given camera poses. However, X-NeRF can just save the explicit scene representations $\mathscr{T}_{\mathrm{out}}$ so that only rendering operations such as interpolation and integral that are highly parallelizable needed to be done during inference. The inference complexity is thus reduced a lot since no neural network is needed to be run.

Since the setting of multi-scene insufficient 360^∘ RGB-D views has never been discussed before, we collect a new dataset for this challenging task. We use 7 RGB-D cameras to capture 6 seen and 4 novel scenes, in which a robot arm is doing different tasks in different environments. The views are extremely sparse with large angle transformations. The overlapping among views is less than 10% to 20%. One example is shown in Fig. 1. Among 7 views, one is left for testing while the other 6 are training views. Please refer to the supplementary material for more details.

All experiments are conducted on a single NVIDIA A100 GPU. We apply PyTorch [31] and Minkowski Engine [4] to build our sparse network. Among all single scene experiments and the multi-scene experiment, we keep the same hyper-parameters. A simple ResNet14 [14] of 3D sparse version is employed as our backbone. When voxelizing the input point clouds, we set the voxel size as $4\times 10^{-3}$. We choose AdamW [25] as our optimizer. In each batch, we sample 2 random image patch of size $40\times 40$ for all 6 training views, which is equivalent to a total ray batch size of $6\times 2\times 40\times 40=19200$. We train our models for $240$ epochs with an initial learning rate of $10^{-3}$, and the learning rate is divided by $10$ at $120$th and $200$th epoch. See supplementary material for detailed hyper-parameter setups.

In this section, we compare proposed X-NeRF with state-of-the-art implicit NeRF-related work on our extremely challenging 360^∘ insufficient RGB-D view dataset. Note that besides common metrics for RGB novel view synthesis, we also evaluate depth error since in our setting of insufficient RGB-D input views, the rendering quality of depth counts for much. We adopt mean squared error in meters in valid areas as depth metric since low-cost depth cameras may have some invalid values.

Single Scene Comparisons. Considering our data and models are RGB-D, we first compare with DS-NeRF [7], a state-of-the-art implicit NeRF-based method that also allows depth inputs. Furthermore, we also compare with DVGO [39], a state-of-the-art NeRF-based approach that utilizes explicit voxel grid structures. Unfortunately, the above two methods do not support multi-scene training, so we compare with them on single scene. DVGO [39] originally does not support depth supervision. For fair comparison, we also add depth supervision to it. The quantitative metrics on novel view can be found in Tab. 1. We can see that X-NeRF significantly out-performs the two methods on each single scene especially on depth error, which means that X-NeRF succeeds in avoiding to overfit to a trivial solution. Moreover, DVGO [39] does not have a significant improvement on novel view after adding depth supervision, indicating that the key factor is not the depth loss but the modeling methodology. The qualitative results of scene 1-3 are displayed in Fig. 6 and all results can be found in supplementary material. It is obvious that implicit methods fails to generalize well on novel view facing insufficient training views and large view gaps.

Multi-Scene and Cross-Scene Comparisons. As mentioned above, X-NeRF is able to deal with multi-scene representation due to our explicit completion modeling. There is a little work that can handle multi-scene task. PixelNeRF [50] combines image features from 2D CNNs with NeRF so that it can train on multi-scene. Therefore, we re-train pixelNeRF [50] concurrently on 6 scenes for $3000$ epochs to compare our work with it on the multi-scene performance. IBRNet [44] is an image-based rendering approach which applies an MLP and a ray transformer to learn a generic view interpolation function. We finetune IBRNet [44] using its pre-trained weights for 60000 iterations. Since the two methods both need reference views when rendering, we use all the 6 seen views as reference images during evaluation. From the quantitative results in Tab. 2 and the qualitative results of scene 3-6 in Fig. 8 (all results can be found in supplementary material), we can see that pixelNeRF [50] completely overfit to a trivial solution, and X-NeRF again beat both the two methods. If we compare Tab. 2 with Tab. 1, the multi-scene version X-NeRF is even better than the single-scene version, indicating that X-NeRF has a powerful generalization capacity. We also do cross-scene comparisons on novel view of two novel scenes. The results indicate that X-NeRF has a powerful cross-scene performance, which proves that X-NeRF learns a completion mapping with good generalization ability.

Complexity Comparisons. We further compare the inference time per image, training time and the model size among the multi-scene methods, which is shown in Tab. 3. We can find that X-NeRF requires less training time. X-NeRF also has a comparable inference time and model size with IBRNet [44], much better than pixelNeRF [50].

We further study X-NeRF’s completion ability. The result in Fig. 7 clearly illustrates that X-NeRF are robust enough to complete the missing areas and correct the wrong values caused by low-cost depth cameras.