GeCoNeRF: Few-Shot Neural Radiance Fields via Geometric Consistency

Rendering a full image is computationally heavy and extremely timetaking, requiring tens of seconds for a single iteration. To overcome the computational bottleneck of full image rendering and warping, rays are sampled on a strided grid to make the patch with stride $s$, which we have set as $2$. After the rays undergo volumetric rendering, we upsample the low-resolution depth map back to original resolution with bilinear interpolation. This full-resolution depth map is used for the inverse warping. This way, detailed warped patches of full-resolution can be generated with only a fraction of computational cost that would be required when rendering the original sized ray batch.

To overcome these issues, we propose masked feature-level regularization loss that encourages structural consistency while ignoring view-dependent radiance effects, as illustrated in Figure 2.

Given an image $I$ as an input, we use a convolutional network to extract multi-level feature maps such that $f_{\phi,l}(I)\in\mathbb{R}^{H_{l}\times W_{l}\times C_{l}}$, with channel depth $C_{l}$ for $l$-th layer. To measure feature-level consistency between warped image $I_{i\rightarrow j}$ and rendered image $I_{j}$, we extract their features maps from $L$ layers and compute difference within each feature map pairs that are extracted from the same layer.

In accordance with the idea of using the warped image $I_{i\rightarrow j}$ as pseudo ground truths, we allow a gradient backpropagation to pass only through the rendered image and block it for the warped image. By applying the consistency loss at multiple levels of feature maps, we cause $I_{j}$ to model after $I_{i\rightarrow j}$ both on semantic and structural level.

Formally written, the consistency loss $\mathcal{L}_{\mathrm{cons}}$ is defined as such that

For this loss function $\mathcal{L}_{\mathrm{cons}}$, we find $l$-1 distance function most suited for our task and utilize it to measure consistency across feature difference maps. Empirically, we have discovered that VGG-19 network (Simonyan & Zisserman, 2014) yields best performance in modeling consistencies, likely due to the absence of normalization layers (Johnson et al., 2016) that scale down absolute values of feature differences. Therefore, we employ VGG19 network as our feature extractor network $f_{\phi}$ throughout all of our models.

It should be noted that our loss function differs from that of DietNeRF (Jain et al., 2021) in that while DietNeRF’s consistency loss is limited to regularizing the radiance field in a globally semantic level, our loss combined with the warping module is also able to give the network highly rich information on a local, structural level as well. In other words, contrary to DietNeRF giving only high-level feature consistency, our method of using multiple levels of convolutional network for feature difference calculation can be interpreted as enforcing a mixture of all levels, from high-level semantic consistency to low-level structural consistency.

In order to prevent imperfect and distorted warpings caused by erroneous geometry from influencing the model, which degrades overall reconstruction quality, we construct consistency mask $M_{l}$ to let NeRF ignore regions with geometric inconsistencies, as demonstrated in Figure 3. Instead of applying masks to the images before inputting them into the feature extractor network, we apply resized masks $M_{l}$ directly to the feature maps, after using nearest-neighbor down-sampling to make them match the dimensions of $l$-th layer outputs.

We generate $M$ by measuring consistency between rendered depth values from the target viewpoint and source viewpoint such that

where $[\cdot]$ is Iverson bracket, and $p_{j\rightarrow i}$ refers to the corresponding pixel in source viewpoint $i$ for reprojected target pixel $p_{j}$ of $j$-th viewpoint. Here we measure euclidean distance between depth points rendered from target and source viewpoints as a criterion for a threshold masking. As illustrated in Figure 4, if distance between two points are greater than given threshold value $\tau$, we determine two rays as rendering depths of separate surfaces and mask out the corresponding pixel in viewpoint $I_{j}$. The process takes place over every pixel in viewpoint $I_{j}$ to generate a mask $M$ the same size as rendered pixels. Through this technique, we filter out problematic solutions at feature level and regularize NeRF with only high-confidence image features.

Based on this, the consistency loss $\mathcal{L}_{\mathrm{cons}}$ is extended as such that

where $m_{l}$ is the sum of non-zero values.

Since our method is dependent upon the quality of depth rendered by NeRF, we directly impose additional regularization on rendered depth to facilitate optimization. We further encourage local depth smoothness on rendered scenes by imposing $l$-1 penalty on disparity gradient within randomly sampled patches of input views. In addition, inspired by (Godard et al., 2017), we take into account the fact that depth discontinuities in depth maps are likely to be aligned to gradients of its color image, and introduce an edge-aware term with image gradients $\partial I$ to weight the disparity values. Specifically, following (Godard et al., 2017), we regularize for edge-aware depth smoothness such that

where $D^{*}_{i}=D_{i}/\overline{D_{i}}$ is the mean-normalized inverse depth from (Godard et al., 2017) to discourage shrinking of the estimated depth.

We optimize our model with a combined final loss of original NeRF’s pixel-wise reconstruction loss $\mathcal{L}_{\mathrm{obs}}$ and two types of regularization loss, $\mathcal{L}^{M}_{\mathrm{cons}}$ for unobserved view consistency modeling and $\mathcal{L}_{\mathrm{reg}}$ for disparity regularization.

Difficulty of of accurate warping increases the further target view is from the source view, which means that sampling far camera poses straight from the beginning of training may have negative effects on our model. Therefore, we first generate camera poses near source views, then progressively further as training proceeds. We sample noise value uniformly between an interval of [-$\beta$, +$\beta$] and add it to the original Euler rotation angles of input view poses, with parameter $\beta$ growing linearly from 3 to 9 degrees throughout the course of optimization. This design choice can be intuitively understood as stabilizing locations near observed viewpoints at start and propagating this regularization to further locations, where warping becomes progressingly more difficult.

We find that most of the artifacts occurring are high-frequency occlusions that fill the space between scene and camera. This behaviour can be effectively suppressed by constraining the order of fourier positional encoding (Tancik et al., 2020) to low dimensions. Due to this reason, we adopt coarse-to-fine frequency annealing strategy previously used by (Park et al., 2021) to regularize our optimization. This strategy forces our network to primarily optimize from coarse, low-frequency details where self-occlusions and fine features are minimized, easing the difficulty of warping process in the beginning stages of training. Following (Park et al., 2021), the annealing equation is $\alpha(t)=mt/K$, with $m$ as the number of encoding frequencies, $t$ as iteration step, and we set hyper-parameter $K$ as $15k$.

We use mip-NeRF (Barron et al., 2021) as our backbone. We give our comparisons to the baseline and several state-of-the-art models for few-shot NeRF: InfoNeRF (Kim et al., 2022), DietNeRF (Jain et al., 2021), and RegNeRF (Niemeyer et al., 2022). We provide implementation details in the appendix.

We evaluate our model on NeRF-Synthetic (Mildenhall et al., 2020) and LLFF (Mildenhall et al., 2019). NeRF-Synthetic is a realistically rendered 360^∘ synthetic dataset comprised of 8 scenes. We randomly sample 3 viewpoints out of 100 training images in each scene, with 200 testing images for evaluation. We also conduct experiments on LLFF benchmark dataset, which consists of real-life forward facing scenes. Following RegNeRF (Niemeyer et al., 2022), we apply standard settings by selecting test set evenly from list of every 8th image and selecting 3 reference views from remaining images. We quantify novel view synthesis quality using PSNR, Structural Similarity Index Measure (SSIM) (Wang et al., 2004), LPIPS perceptual metric (Zhang et al., 2018) and an average error metric introduced in (Barron et al., 2021) to report the mean value of metrics for all scenes in each dataset.

Qualitative comparison results in Figure 5 and 6 demonstrate that our model shows superior performance to baseline mip-NeRF (Barron et al., 2021) and previous state-of-the-art model, RegNeRF (Niemeyer et al., 2022), in 3-view settings. We observe that our warping-based consistency enables GeCoNeRF to capture fine details that mip-NeRF and RegNeRF struggle to capture in same sparse view scenarios, as demonstrated with the mic scene. Our method also displays higher stability in rendering smooth surfaces and reducing artifacts in background in comparison to previous models, as shown in the results of the materials scene. We argue that these results demonstrate how our method, through generation of warped pseudo ground truth patches, is able to give the model local, scene-specific regularization that aids recovery of fine details, which previous few-shot NeRF models with their global, generalized priors were unable to accomplish.

Comparisons in Table 1 show our model’s competitive results in LLFF dataset, whose PSNR results show large increase in comparison to mip-NeRF baseline and competitive compared to RegNeRF. We see that our warping-based consistency modeling successfully prevents overfitting and artifacts, which allows our model to perform better quantitatively.

We observe that without the consistency loss $\mathcal{L}_{\mathrm{cons}}$, our model suffers both quantitative and qualitative decrease in reconstruction fidelity, verified by incoherent geometry in image (a) of Figure 7. Absence of unseen view consistency modeling destabilizes the model, resulting divergent behaviours.

We observe that addition of occlusion mask $M$ improves overall appearance as well as geometry, as shown in image (c) of Figure 7. Its absence results broken geometry throughout the overall scene, as demonstrated in (b). Erroneous artifacts pertaining to projections from different viewpoints were detected in multiple scenes, resulting lower quantitative values.

In Table 3, we justify our progressive training strategies with additional experiments on NeRF-Synthetic dataset, while in the main ablation we conduct an ablation with progressive annealing only. For pose generation, we sample pose angle from large interval in the beginning, instead of slowly growing the interval. For positional encoding, we replace progressive annealing with naïve positional encoding used in NeRF. We observe that their absence causes destabilization of the model and degradation in appearance, respectively.

We observe that inclusion of edge-aware disparity regularization $\mathcal{L}_{\mathrm{reg}}$ refines given geometry, as shown in image (e) of Figure 7. By applying $\mathcal{L}_{\mathrm{reg}}$, we see increased smoothness in geometry throughout the overall scene. This loss contributes to removal of erroneous artifacts, which achieves better results both qualitatively and quantitatively, as shown in Table 2.

In Table 4, we conduct a quantitative ablation comparisons between feature-level consistency loss $\mathcal{L}^{M}_{\mathrm{cons}}$ and pixel-level photometric consistency loss $\mathcal{L}^{M}_{\mathrm{pix}}$, both with occlusion masking. As shown in Figure 8, naïvely applying pixel-level loss for consistency modeling leads to broken geometry. This phenomenon can be attributed to $\mathcal{L}_{\mathrm{pix}}$ being agnostic to view-dependent specular effects, which the network tries to model by altering or erasing altogether non-Lambertian surfaces.