Spatial Annealing for Efficient Few-shot Neural Rendering

We start by examining the relationship between frequency regularization (see Eq. 1) and the pre-filtering strategy based on integrated positional encoding (IPE). We consider a multivariate Gaussian distribution in 3D space, depicted as

$$G(\mathbf{x})=e^{-\frac{1}{2}(\mathbf{x}-\boldsymbol{{\mu}})^{T}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{{\mu}})},$$

(4)

where $\boldsymbol{\mu}$ represents the position, and $\boldsymbol{\Sigma}$ denotes the covariance matrix. For straightforward comparison, we model the Gaussian as isotropic, characterized by a diagonal $\boldsymbol{\Sigma}$ with uniform elements $\sigma_{f}^{2}$, reflecting the sample space size. Following MipNeRF (Barron et al. 2021), we calculate the Gaussian’s integrated positional encoding $\gamma_{G}$ as

$$\gamma_{G}(\boldsymbol{\mu})=\gamma^{\prime}_{G}(\boldsymbol{\mu})\odot\mathbf{M}_{L}$$

(5)

with

$$\begin{split}\mathbf{M}_{L}&=[e^{-\frac{1}{2}\sigma_{f}^{2}},e^{-\frac{1}{2}\sigma_{f}^{2}},...,e^{-2^{2L-3}\sigma_{f}^{2}},e^{-2^{2L-3}\sigma_{f}^{2}}]\\ \gamma^{\prime}_{G}(\boldsymbol{\mu})&=[\sin(\boldsymbol{\mu}),\cos(\boldsymbol{\mu}),...,\sin(2^{L-1}\boldsymbol{\mu}),\cos(2^{L-1}\boldsymbol{\mu})],\end{split}$$

(6)

where $\gamma^{\prime}_{G}(\boldsymbol{\mu})$ represents the frequency positional encoding of $\boldsymbol{\mu}$, and $\mathbf{M}_{L}$ denotes a low-pass mask applied to this encoding, with $L$ controlling the maximum encoded frequency. The mask’s structure aligns with a Gaussian distribution, featuring a covariance $\sigma_{i}^{2}=1/\sigma_{f}^{2}$, where $\sigma_{i}^{2}$ regulates the frequency bandwidth.

Upon comparing Eq. 6 with Eq. 1, we observe a notable similarity between the IPE and frequency regularization: both apply a low-pass mask to the original positional encoding, with the mask’s form being the primary difference. Drawing inspiration from FreeNeRF (Yang, Pavone, and Wang 2023), which linearly expands the frequency mask (effectively exponentially increasing the frequency bandwidth), we adopt an exponential growth model for $\sigma_{i}^{2}$, defined as $\sigma_{i}^{2}=f_{s}2^{x}$, where $x$ is the step increment corresponding to the iteration count and $f_{s}$ controls the initial frequency bandwidth. Concurrently, the spatial Gaussian’s covariance $\sigma_{f}^{2}$ decreases exponentially, represented as

$$\sigma_{f}^{2}=\frac{1}{f_{s}2^{x}}.$$

(7)

Consequently, we deduce that the frequency regularization introduced by FreeNeRF (Yang, Pavone, and Wang 2023) can be executed by inversely adjusting the spatial sample space within the pre-filtering strategy, as detailed in Eq. 7. This insight guides the exploration of the form of spatial annealing strategy in hybrid representation.

Based on recent research, the band limit of an optimized field can be determined by the sampling kernel under specific conditions (Shabanov et al. 2024). We analyze the sampling process of this optimized neural field directly. General grid-based models (Zhao et al. 2024; Shabanov et al. 2024) reconstruct the optimized continues field $\mathbf{f}(\mathbf{x})$ from discrete feature points by applying a convolution process with a kernel function $\kappa$:

$$\mathbf{\hat{f}}(\mathbf{x})=\hat{\kappa}_{\omega}(\mathbf{x})*\sum_{n}\mathbf{f}(\mathbf{x})\cdot\delta(\mathbf{x}-\mathbf{x}_{n}),$$

(8)

where $\delta$ is the impulse function, $\omega$ is the frequency bandwidth of the kernel, and $x_{n}\in\mathbb{R}^{3}$ are discrete grid points. In current hybrid representation fields, the kernel function can be implemented using simple linear interpolation, equivalent to a conical kernel $\Lambda_{\omega}(\mathbf{x})$. For regular grids with period $T$, the frequency bandwidth of the kernel is approximated by $\omega=\frac{2\pi}{T}$, as the conical kernel is not an ideal low-pass filter.

In TriMipRF, the reconstructed signal $\mathbf{\hat{f}}^{m}(\mathbf{x},[l])$ for each plane $m$ at a specific level $[l]$ can be represented as

$$\mathbf{\hat{f}}^{m}(\mathbf{x},[l])=\Lambda^{m}_{\omega_{[l]}}(\mathbf{x})*\sum_{n}\mathbf{f}^{m}(\mathbf{x},[l])\cdot\delta(\mathbf{x}-\mathbf{x}_{n}^{[l]}),$$

(9)

where $m\in\{xy,xz,yz\}$ represents the label of each plane, $\Lambda^{m}_{\omega_{[l]}}(\mathbf{x})$ represents a two-dimensional conical kernel, and $\{\mathbf{x}_{n}^{[l]}\}$ is the uniform lattice with level $[l]$. The bandwidth of each plane at a specific level $[l]$ can be represented as $\omega_{[l]}=\frac{2\pi}{2^{[l]}T_{0}}$, where $T_{0}=\frac{B_{max}-B_{min}}{r_{0}}$ represents the period of the grid with maximum resolution $r_{0}$ and bounding box size $B_{max}-B_{min}$. We label the bandwidth $\omega$ corresponding to the level $l$ as $\omega_{l}$. TriMipRF utilizes multi-resolution tri-planes to model varying levels of detail (LOD), it queries features from adjacent integral levels of $l$ corresponding to the size of the sphere (refer to Eq. 3) as

$$\mathbf{\tilde{f}}^{m}(\mathbf{x},l)=\Lambda_{\omega_{c}}(l)*\sum_{[l]}\mathbf{\hat{f}}^{m}(\mathbf{x},l)\cdot\delta(l-[l]),$$

(10)

where $\Lambda_{\omega_{c}}(l)$ is a conical kernel with constant bandwidth $\omega_{c}$, and $[\cdot]$ represents discrete integral values. TriMipRF utilizes a pre-filtering strategy across the level dimension to filter out fields with higher-frequency bandwidths that cannot be well-sampled by discrete rays, thereby addressing aliasing artifacts. This approach constructs a continuous field along the level dimension within a continuous range of sampling spaces. Equation 10 can also be rewritten in the form of linear interpolation across the level dimension:

$$\mathbf{\tilde{f}}^{m}(\mathbf{x},l)=\frac{\lceil l\rceil-l}{\lceil l\rceil-\lfloor l\rfloor}\mathbf{\hat{f}}^{m}(\mathbf{x},\lfloor l\rfloor)+\frac{l-\lfloor l\rfloor}{\lceil l\rceil-\lfloor l\rfloor}\mathbf{\hat{f}}^{m}(\mathbf{x},\lceil l\rceil).$$

(11)

The continuous field represented by each feature plane can be considered a linear combination of the two fields at specific levels $\lceil l\rceil$ and $\lfloor l\rfloor$. Therefore, the bandwidth $\omega_{m}$ of the combined field lies within the range $(\omega_{\lceil l\rceil},\omega_{\lfloor l\rfloor})$ as

$$\omega_{\lceil l\rceil}\leq\omega_{m}\leq\omega_{\lfloor l\rfloor}.$$

(12)

The final feature field can be represented as the concatenation of components corresponding to each plane. Since the sampling process operates independently based on the same kernel and identical querying level $l$, the bandwidth $\omega$ of the final field is determined by $\omega_{m}$.

Fig. 1 provides an overview of our method. We have developed a coarse-to-fine training strategy based on TriMipRF (Hu et al. 2023). Initially, we implement blurring in the rendering process by increasing the radius $r$ of the sample sphere, as depicted in the bottom-left corner of Fig. 1. At the beginning of training, this approach aids in optimizing the reconstruction of global geometric structures, crucial for mitigating overfitting issues common in few-shot scenarios, as illustrated in Fig. 2. The radius of the sample area is systematically reduced following an exponential decay described in Eq. 14. This gradual reduction aims to progressively shift the training focus towards enhancing local geometric and textural details. The specific steps of this annealing process are outlined as

$$r_{i}=\begin{cases}\tau+\dfrac{f_{s}}{2^{\upsilon x}},i<T\\ \tau,i\geq T\end{cases},x=\left\lfloor\dfrac{iN_{split}}{T}\right\rfloor,$$

(15)

where $N_{split}$ denotes the total number of decrement steps, $i$ represents the current iteration count, and $T$ indicates the stopping point of annealing. $f_{s}$ determines the initial size of the sphere, while $\vartheta$ governs the rate of decrement.

where $\boldsymbol{w}^{\lfloor l\rfloor}_{n}$ and $\boldsymbol{w}^{\lceil l\rceil}_{n}$ are features stored in grid points at levels $\lfloor l\rfloor$ and $\lceil l\rceil$, respectively, and $\varphi$ represents the interpolation weights. As the GTK of a grid-based model remains stationary during training (Zhao et al. 2024), the GTK during training can be computed by the optimized neural field:

$$[\boldsymbol{G}_{g}(t)]_{i,j}=\left\langle\frac{\partial\mathbf{\tilde{f}}^{m}(\mathbf{x}_{i},l)}{\partial\boldsymbol{w}},\frac{\partial\mathbf{\tilde{f}}^{m}(\mathbf{x}_{j},l)}{\partial\boldsymbol{w}}\right\rangle,$$

(17)

where $\boldsymbol{w}$ denotes features stored in grid points. We numerically analyze the 1D spatial GTK of the TriMip-represented field and its spectrum, as shown in Fig. 3. The results indicate that with a larger sampling sphere, the spectrum becomes narrower, which biases training more towards low-frequency information, especially low-frequency geometry (Zhao et al. 2024). Such spectral bias performs exceptionally well when handling extreme scenarios, such as sparse views that specifically lack multi-view geometry consistency. As the training iteration progresses, the size of the sample sphere exponentially decreases, corresponding to a wider GTK spectrum, which shifts the training focus towards high-frequency details. Although current research has reached this insight for implicit representation (Yang, Pavone, and Wang 2023), we expand this insight to grid-based hybrid representation utilizing recently introduced GTK analysis.

We conduct experiments to validate this insight. We vary the sampling sphere size $r$ from a small value (0.001) to a relatively larger value (0.07), aligning with the setting of the GTK analysis, and evaluate the depth mean square error (D-MSE) between the rendered depth results and the ground truth (GT) depth maps, as shown in Fig. 4. We observe that the D-MSE decreases with increasing sampling sphere size. Additionally, qualitative results indicate that a larger sampling area with blurred rendered results corresponds to more complete geometry.

Fig. 5 and Tab. 1 show the qualitative and quantitative comparisons on the Blender dataset (Mildenhall et al. 2021), respectively. The qualitative analysis reveals that the TriMipRF (Hu et al. 2023) produces noticeable distorted rendered results in the “chair” and “hotdog” scenes. In contrast, our approach, which requires only a few additional lines of code, effectively addresses these issues. When compared with the latest few-shot baseline, FreeNeRF (Yang, Pavone, and Wang 2023), our method shows superior performance, particularly in detail-rich areas highlighted in red boxes. Specific examples include the texture in the “chair” scene and the edge of the plate in the “hotdog” scene.

Our method demonstrates significant advancements in quantitative results, outperforming TriMipRF (Hu et al. 2023) by nearly 3 dB in PSNR with similar 35 seconds reconstruction time. Furthermore, the evaluated training durations demonstrate that our method achieves faster reconstruction speed. Notably, while the FreeNeRF (Yang, Pavone, and Wang 2023) necessitates 7.5 hours for training, and DietNeRF (Jain, Tancik, and Abbeel 2021), with its patch-based semantic regularizer, requires an extensive 26 hours, our approach achieves a remarkable 700$\times$ reconstruction speed compared to FreeNeRF. In addition to its efficiency, our method also delivers superior quantitative outcomes, exceeding FreeNeRF by 0.27 dB in PSNR. While these results are impressive, extending the training duration has the potential to further enhance performance. To this end, we increase the training iterations to $5K$, while keeping all other settings unchanged. As shown in Tab. 1, our method not only surpasses FreeNeRF with a 0.55 dB gain in PSNR but also delivers a substantial 350$\times$ acceleration.

In previous section, we provide a theoretical analysis that clarifies the relationship between the pre-filtering strategy and frequency regularization based on integrated positional encoding. To demonstrate the strategy’s validity and to evaluate our method’s performance across various base architectures, we apply it using the JAX MipNeRF framework (Barron et al. 2021). In this setup, we adjust the initial sphere size to 0.2, establish the decrease rate $\vartheta$ at 1, and set the total number of decrease steps $N_{split}$ to 33.

Fig. 6 and Tab. 2 present qualitative and quantitative comparisons of our method with FreeNeRF (Yang, Pavone, and Wang 2023). Quantitatively, our method matches or slightly outperforms FreeNeRF, showing up to a 0.05 dB and 0.1 dB increase in PSNR on the LLFF dataset (Mildenhall et al. 2019) and the DTU dataset (Jensen et al. 2014), respectively. Qualitatively, our method provides improved novel views and depth maps. For instance, whereas FreeNeRF struggles with accurately reconstructing the leaf edges in the first row and the ceiling’s middle section in the second row, our method maintains geometric consistency and offers more detailed depth maps. This advantage likely stems from our method’s greater adaptability. Unlike frequency regularization, constrained by a fixed frequency range, our method dynamically adjusts the sample space to a wider range, thereby enhancing global geometry reconstruction.

We validate our method alongside the TriMipRF (Hu et al. 2023) using varying numbers of input views to assess our method’s resilience to view sparsity, as shown in Fig. 7. We use the first $n$ images from the Blender’s training set as input, where $n$ is varied among 8, 20, 40, 60, 80, and 100, with the dataset containing a total of 100 images. We fix the total iteration steps at 10,000 and the spatial annealing strategy’s endpoint at 2,000. The curve of Fig. 7 demonstrates that our method outperforms TriMipRF under different input view conditions. Notably, it shows marked improvements in scenarios with sparse input views, achieving a 2.50 dB and 1.22 dB higher PSNR with 8 and 20 input views, respectively.

To thoroughly assess the efficacy of our proposed method, we conduct qualitative and quantitative ablation studies using the Blender dataset (Mildenhall et al. 2021) with 8 input views, as detailed in Fig. 8 and Tab. 3. We evaluate the contributions of the newly introduced spatial annealing strategy (SA) and the spherical harmonic mask (SHM) separately. The findings reveal that the spatial annealing strategy significantly enhances performance in a few-shot setting, achieving a 1.2 dB increase in PSNR on the Blender dataset with SA alone. Regarding qualitative results, the rendered RGB images and depth maps demonstrate that our spatial annealing strategy primarily enhances the reconstruction of low-frequency geometry but does not effectively address color distortions. Conversely, the SHM effectively mitigates color distortions but fails to resolve geometric underfitting. The combination of both strategies results in well-established geometry and photorealistic rendering. These results underscore the substantial improvements achieved by our method.