NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field

All possible light rays in a space can be described by a 5D plenoptic function. Since radiance along a ray is constant if viewed from outside of the convex hull of the scene, this 5D function can be reduced to a 4D light field [16, 8]. The most common parameterization is a two-plane model shown in Fig. 3. Each ray from the camera to the scene will intersect with the two planes. Thus, we can represent each ray using the coordinates of the intersections, $(u,v)$ and $(s,t)$, or simply a 4D coordinate $(u,v,s,t)$. Using this representation, all rays from the object to one side of the two planes can be uniquely determined by a 4D coordinate.

Based on this representation, rendering a novel view can be done by querying all the rays from the center of projection to every pixel on the camera’s image plane. We denote them as $\{R_{1},R_{2},...,R_{N}\}$, where $N$ is the total number of pixels. Then, for the $i$-th ray $R_{i}$, we can obtain its 4D coordinate $(u_{i},v_{i},s_{i},t_{i})$ by computing its intersections with the two planes. If a function $f$ maps the continuous 4D coordinates to color values, we can obtain the color of $R_{i}$ by evaluating the function $f_{c}(u_{i},v_{i},s_{i},t_{i})$. In the next section, we will introduce Neural 4D Light Field (NeuLF) for reconstructing this mapping function $f$.

We formulate the mapping function $f_{c}$ as a Multilayer Perceptron (MLP). The input of this MLP is a 4D coordinate and the output is RGB color. As shown in Fig. 2. The goal of the network is to learn the mapping function from training data.

Training Data: for a given scene, the training data comes from a set of captured images $\{I_{1},I_{2},...,I_{M}\}$, where $M$ is the total number of images. Assuming the camera pose for each image is known or obtainable, for each image $I_{k}(k=1,...,M)$, we can traverse its pixels and generate all corresponding rays $\{R_{1}^{k},R_{2}^{k},...,R_{N_{k}}^{k}\}$, where $N_{k}$ is the total number of pixels in the $k$-th image. Based on the 4D light field representation, all 4D coordinates $\left\{\left(u_{1}^{k},v_{1}^{k},s_{1}^{k},t_{1}^{k}\right),...,\left(u_{N_{k}}^{k},v_{N_{k}}^{k},s_{N_{k}}^{k},t_{N_{k}}^{k}\right)\right\},(k=1,...,M)$, can be obtained. On the other hand, the color for each pixel is known from the input images. To this end, we have constructed a collection of sample mappings from 4D coordinates to color values $\left(u_{i}^{k},v_{i}^{k},s_{i}^{k},t_{i}^{k}\right)\rightarrow c_{i}^{k},(k=1...M,i=1...N_{k})$, where $c_{i}^{k}$ is the color of the $i$-th pixel on the $k$-th image. By feeding this training data to the MLP network, the parameters $\Theta$ can be learned by minimizing the following photometric loss $\mathcal{L}_{p}$:

In Fig. 2, we demonstrate an example of capturing images to train our neural 4D light field representation with a set of unstructured front-faced camera views. In this example, the cameras are placed on one side of two light slabs.

Rendering: Given a viewpoint $\mathcal{V}$, we can render a novel view $\mathcal{R}\left(\mathcal{V}\right)$ by evaluating the learned mapping function $f$. With the camera pose and the desired rendering resolution $\{W^{\mathcal{V}},H^{\mathcal{V}}\}$, we sample all rays $\{R_{1}^{\mathcal{V}},R_{2}^{\mathcal{V}},...,R_{N_{\mathcal{V}}}^{\mathcal{V}}\}$, where $N_{\mathcal{V}}=W^{\mathcal{V}}\times H^{\mathcal{V}}$ is the number of pixels to be rendered. We can further calculate the 4D coordinates $\left\{\left(u_{i}^{\mathcal{V}},v_{i}^{\mathcal{V}},s_{i}^{\mathcal{V}},t_{i}^{\mathcal{V}}\right)\right\}$ for each ray $R_{i}^{\mathcal{V}},(i=1...N_{\mathcal{V}})$. We then formulate the rendering process $\mathcal{R}$ as $N_{\mathcal{V}}$ evaluations of the mapping function $f_{c}$:

To simulate variable focus and variable depth-of-field using light field, previous work [12] dynamically reparameterizes the light field by manually moving the focal surface as a plane. To optimally and automatically select a focal plane given a pixel location, we aim to solve the focal surface manifold geometry that conforms to the scene geometry under Lambertian scene assumption.

We let the same MLP predict per-ray depth as shown in Fig. 2. As in Fig. 4, for a 4D query ray $(u_{i}^{k},v_{i}^{k},s_{i}^{k},t_{i}^{k})$ from the camera $C_{k}$, the predicted scene depth is $Z_{i}^{k}=f_{d}(u_{i}^{k},v_{i}^{k},s_{i}^{k},t_{i}^{k}\>|\>\Theta)$ where $f_{d}$ is the network mapping function from 4D coordinates to scene depth. We compute ray-surface intersection $(X_{i}^{k},Y_{i}^{k},Z_{i}^{k})$. Then, we self-supervise $Z_{i}^{k}$ by applying multi-view consistency cues. To do this, we trace the rays from $(X_{i}^{k},Y_{i}^{k},Z_{i}^{k})$ back to $C_{k}$’s K-nearest data cameras $\{C_{j}\},(j=1,...,K)$ (K=5 in our experiments). Those rays intersect uv-st plane and can be parameterized as:

Ray-plane intersection is differentiable. We propose two loss functions $\mathcal{L}_{mp}$ and $\mathcal{L}_{d}$ to minimize the multi-view photometric error and depth differences respectively as follows:

where $\omega(i,j)$ are the normalized weights $\omega(i,j)=\frac{\frac{1}{d^{2}_{ij}}}{\sum_{j}\frac{1}{d^{2}_{ij}}}$ with the Euclidean distance between the camera $i$ and its neighbor camera $j$ as $d_{ij}$. $\mathcal{C}_{i}^{k}(Z^{k}_{i})$ and $\mathcal{D}_{i}^{k}(Z^{k}_{i})$ are the supervise color and depth sum over weighted nearest data cameras with network output $Z^{k}_{i}$ respectively. Training with both $\mathcal{L}_{mp}$ and $\mathcal{L}_{d}$ will encourage the MLP to learn the depth representation of the scene.

With ray-based depth, we can enable efficient auto-refocus effect by adopting a dynamic parameterization of the light field  [12]. More specifically, we simulate the depth-of-field effect by combining rays within an given aperture size. Fig. 5 shows the case of two rays $r_{1}$ and $r_{2}$. The two rays intersect camera image plane at pixels $p_{1}$ and $p_{2}$, intersect the scene geometry at $P_{1}$ and $P_{2}$, and intersect a given focal plane $F_{2}$ at $P_{1}$ and $P^{\prime}_{2}$. To reconstruct the final color of the pixels $p_{1}$ and $p_{2}$, we collect a cone of sample rays originated from $P_{1}$ and $P^{\prime}_{2}$ on the focal plane $F_{2}$ within an aperture $A$. We then query the network $f_{c}$ to obtain the ray colors. These ray colors are weighted-averaged to produce the final pixel color. In this case, $P_{1}$ is on the surface of the object, while $P^{\prime}_{2}$ is not. Thus, the image pixel $p_{1}$ appears in focus, while pixel $p_{2}$ is blurred since it combines colors from a small area around surface point $P_{2}$.

To auto-refocus at pixel location $p_{2}$, we extract its depth by query the depth network, and set a new focal plane $F_{1}$ at pixel $p_{2}$’s depth. Rendering the NeuLF with this new focal plane will make pixel $p_{2}$ in focus while blur pixel $p1$.The results are shown in Fig. 6.

We train the MLP on the following overall loss function:

where the weighting coefficients are $\lambda_{s}=0.5$ and $\lambda_{r}=0.1$. These parameters are fine-tuned by mixing the manual tuning and grid search tuning. This set of parameters works best for most of the scenes we tested. For scenes with strong view-dependent effects such as specularities, Lambertian assumption is no longer valid. Therefore, we disable the multi-view photometric error and depth difference terms in the loss for such scenes. Grid search can be used to automatically find optimal parameters for a specific scene but will lead to a longer training time.

To extract camera rays from input photos, we calibrate the camera poses and intrinsic parameters using a structure-from-motion tool from COLMAP [30]. During training, we randomly select a batch of camera rays from the training set at each iteration. By passing them to the MLP to predict the color of each ray, we calculate $\mathcal{L}_{p}$ and back-propagate the error.

The input 4D coordinate $(u,v,s,t)$ (normalized to $[-1,1]$) is passed through 20 fully-connected ReLU layers, each with 256 channels. We developed a structure that includes a skip connection that concatenates the input 4D coordinate to every 4 layers start with the fifth layer. An additional layer outputs 256-dimensional feature vector. This feature vector is split into the color branch and depth branch. Each branch is followed with an additional fully-connected ReLU layer with 128 channels, and output 3 channel RGB radiance with sigmoid activation and 1 channel scene depth with sigmoid activation, respectively. For model training, we set the ray batch size in each iteration to $8192$. We train the MLP for 1500 epochs using the Adam optimizer [15]. The initial learning rate is $5\times 10^{-4}$ and decays by $0.995$ every epoch. To train the NeuLF on a scene with 30 input images with a resolution of $567\times 1008$, it takes 5 hours using 1 Nvidia RTX 3090 card. For testing on the same situation, rendering an image costs about 70ms while NeRF takes 51 seconds.

In this section, we demonstrate the qualitative results of novel view synthesis and compare them with current top-performing approaches: NeRF [25], NeX [41] and the baseline light field rendering method [16]. We evaluate the models on the shinny dataset [41]. In this dataset, each scene is captured by a handheld smartphone in a forward-facing manner with a resolution $567\times 1008$ or $756\times 1008$. This is a challenging dataset that contains complex scene geometry and various challenging view-dependent effects. For example, refraction through the test tubes (filled with liquid) and magnifier, rainbow effect emits by a CD disk and sharp specular highlights from silverware and thick glasses.

We hold out $\frac{1}{8}$ of each scene as the test set and the rest of them as the training set. The qualitative results are shown in Figure. 7. The leftmost column shows our results on the test view of three challenging scenes (Lab, CD, and Tools). We have zoomed in on the part of the image areas for comparison with other methods. Our method is superior when a scene contains detailed refraction and reflection. Our result has less noise on glass sparkles. In the CD scene, our result shows more sharp and vivid detail on the rainbow, plastic cup reflections, and less noise on the liquid bottle than NeX and NeRF. In the Tools scene, although our result is not as sharp as the ground truth, it contains more overall details than NeX and NeRF and is able to recover metallic reflection with less noise than others. Our method essentially relies on ray interpolation rather than volumetric rendering like NeRF/NeX. We believe the traditional ray interpolation handles refraction and reflection better than volumetric representation.

The baseline Light Field rendering (last column) exhibits good results when rays are sufficiently sampled (magnifier). However, its method exhibits aliasing and misalignment artifacts in the low ray sampling area (metallic, tube).

We report three metrics: PSNR (Peak Signal-to-Noise Ratio, higher is better), SSIM (Structural Similarity Index Measure, higher is better), and LPIPS (learned perceptual Image Patch Similarity, lower is better) to evaluate our test results. In Table 2, we report the three metrics for the 8 scenes in the shinny dataset. We use the NeX and NeRF scores originally report from the NeX paper. For each scene, we calculate the scores by averaging across the views in the test split. Our method produces the highest score across all three metrics on CD and Lab scenes which contain challenging refraction and reflection. For the rest scenes, while NeX has the highest score by producing high-frequency details in the richly-textured area, we generate comparable scores with NeRF. Note that our rendering speed is $1000\times$ than NeRF.

To demonstrate the effectiveness of our proposed multiview and depth regularization terms, we use two mostly Lambertian scenes the $columns$ and the $tribe$, which contain complex local details, for ablation studies. We demonstrate how the multi-view and depth loss $\mathcal{L}_{mp}$ and $\mathcal{L}_{depth}$ affect the quality of synthesized views. We show the results of two different scenes. We compared the model trained with and without our proposed regularization terms (MPVL). As shown in Table. 8 and Fig. 3, our model with multi-view and depth regularization improves the result both qualitatively and quantitatively in both scenes, of which more details can be reconstructed. For Lambertian scenes, the network trained with multi-view and depth consistency constraints takes the scene geometry into account. Hence the results in unseen views are more reasonable and detail preserved.

To understand how the number of input views affect the novel view synthesis result, we train our model on fewer images. As shown in Figure 9, we use 75%, 50%, and 25% of the original data for the experiment. Although the input images are dramatically decreased, our method still generates high-quality results. Our results still retain the rainbow and background reflections, and the refraction details through the test tube are also well retained. In Table  4, note that even with less input images for CD and Lab scenes, our results are still comparable with or even better than NeRF and NeX with full number of inputs in the above challenging scenes.

As applications of NeuLF, we show results of depth estimation and automatic refocusing.

In Figure 10, we show an example of the depth estimation and refocusing effect on our own captured scene Tribe. Note the detailed depth of the house and statue are successfully recovered. With free-viewpoint scene depth, we can automatically select a focal plane given an image pixel location. We show two synthesized novel views rendered from our own captured scene. Then, we show the auto-refocus result given two positions on the image, one focuses on the near object (red cross), and another on the far object (green cross). This is enabled naturally by the dynamic 4D light field representation [12].

Our method is based on a 4D 2PP (two-plane parameterization) light field representation. Since each 3D world position corresponds to multiple discontinued 4D coordinates, NeuLF representation is difficult to learn. Therefore, we observe that our model cannot fully recover the high-frequency details in the scene as shown in Figure. 11. The seaweed texture (first row) and object’s Hollow-out structure (second row) are over-smoothed. In addition, different exposure and lighting change across the frames can lead to flickering artifacts in the results. Recent works show that using a high-dimensional embedding [38], or using periodic activation functions [32] can help recover fine details. However, we found that the above method is causing over-fitting in the training views and a lack of accuracy in the test views on our NeuLF representation. Learning how to recover the fine details of the 2PP light field representation can be an interesting direction in the future.