Adjustable Visual Appearance for Generalizable Novel View Synthesis

NeRFs [25] synthesize consistent and photo-realistic novel views of a scene, by representing each scene as a continuous 5D radiance field parameterized by an MLP mapping 3D positions and 2D viewing directions to volume densities and view-dependent emitted radiances. Views are synthesized by querying points along camera rays and and utilizing volumetric rendering to aggregate the output colors and densities into RGB values. There have been several works improving NeRFs further, e.g. to improve the efficiency [26, 21] and handling few input views [46, 27].

The original NeRF methodology is constrained to training a neural network for representing a single scene, optimizing from scratch for each new scene, without leveraging any prior knowledge. Methods for generalizable neural rendering address this limitation by training on multiple scenes, enabling the learning of a general understanding of how to utilize source observations to synthesize novel views. Earlier methods such as [46, 5] use a multilayer perceptron (MLP) conditioned on feature vectors extracted from the source images to predict color and radiance values which are aggregated with volumetric rendering. To enhance generalization capabilities and rendering quality, recent approaches have incorporated transformer-based architectures [39, 9] for feature aggregation from the source images [23, 42], computing densities along the camera ray [43], and even for the entire rendering pipeline [34, 35, 33, 11]. While these methods have demonstrated impressive rendering quality, they are currently incapable of modifying the appearance of the rendered views.

The success of Generative Adversarial Networks (GANs) [13] has largely driven advances in 2D style transfer. Methods such as Pix2Pix [20], Pix2Pix-HD [44], and BicycleGAN [50] utilize paired training data, which consists of corresponding images in the source and target conditions. CycleGAN [49] and CyEDA [2] employ cycle-consistency constraints to learn from unpaired data. NICE-GAN [6] reuses the discriminator for encoding the images of the target domain. In addition to GANs, the style-attentional network (SANet) [28] can synthesize a content image with the style of another image. Diffusion models [8, 17] have recently achieved superior results in image generation. Palette [32] introduced the first diffusion-based paired image-translation model, and DiffuseIT [22] recently presented a diffusion-based unsupervised image translation method. Instruct-Pix2Pix [3] re-trains a latent diffusion model [30] using paired images generated by prompt-to-prompt [16] and massive instructions generated by GPT-3 [4] to facilitate instruction-based style transfer. While the images translated by these 2D style transfer methods can individually appear realistic, they do not ensure temporal consistency. In contrast, our method inherently ensures 3D consistency. We experimentally compare our results with 2D style transfer methods applied frame by frame on rendered views.

Prior work to enable changing the visual appearance of a NeRF model [24, 36] typically assign an appearance embedding vector to each image which affect the appearance but not geometry, and is optimized alongside the NeRF model parameters. In [24], low dimensional embeddings allows for smooth interpolation between lighting conditions. One limitation of this approach is that it requires access to images of the scene at both lighting conditions as input. In contrast, our method is a generalizable method that does not require images at both lighting conditions as input when rendering novel views with changed visual appearance. Another line of research is to edit the style of a NeRF model based on a given style prompt [18, 14, 19, 47] typically given as a reference image. More recent works [40, 41] use the joint language-image embedding space of CLIP [29] to enable specifying the desired style using a text prompt. These methods focus on artistic style changes and have thus not been specifically trained and evaluated on realistic appearance changes such as differences in weather or lighting, in contrast to our method. The recent method Instruct-NeRF2NeRF [15] enables editing a NeRF model based on a text-prompt, by iteratively updating dataset images using a pretrained 2D editing model [3]. This method allows for a variety of appearance changes since they utilize pre-trained diffusion models to perform the editing, but they require training per-scene NeRF models and additional per-scene optimization, in contrast to our method that does not require per-scene training.

The used dataset is generated using the autonomous driving simulator CARLA [10], which enables the generation of synthetic images within a simulated city environment along with their ground truth camera poses. Additionally, weather and lighting conditions can easily be changed.

For our experiments, four conditions were defined, corresponding to night, day, rain and evening. A scene was defined as a sequence of 10 observations taken along a road. With four different conditions, this led to a total of 40 images per scene. All generated images are $800\times 600$ pixels. The CARLA map was split into two regions, one to generate 145 training scenes, and the other to generate 38 evaluation scenes, ensuring separation between training and evaluation scenes. This dataset alongside the code will be made publicly available. We also show qualitative examples evaluating our trained model on scenes from the Spaces dataset [12] to show that our method can generalize to real images.

The model was initialized with weights from a GNT network pretrained on a combination of synthetic and real data [35]. The model was trained to perform visual appearance change using the 145 training scenes from the introduced CARLA dataset, including the proposed appearance change loss term (3). The training was performed on a single A100 GPU, taking approximately 8 hours, and the method was then able to generalize to scenes not seen during training. Note that the model was trained for all training scenes at the same time, and there is no scene-specific training for the test scenes. When we test the model, we only use images of the test scene in the source condition $\mathbf{I}_{s,c}$ and not any images of the test scene in the target condition $c^{\prime}\neq c$.

The GAN-based methods, such as Pix2Pix-HD [44], BicycleGAN [50], NICE-GAN[6], and CyEDA [2], along with the diffusion model-based method Palette [32], have been retrained using our synthetic dataset. The reference-based methods, DiffuseIT [22] and SANet [28], are capable of performing image translation using a reference images at the target condition from the synthetic dataset. Instruct-Pix2Pix [3] is pre-trained on editing images based on text prompts and was not retrained on our synthetic dataset.

Furthermore, we compared our method with the Instruct-NeRF2NeRF [15] model, utilizing the official implementation that employs the Nerfstudio [37] Nerfacto NeRF model. Due to the unsatisfactory quality of the Nerfacto models when using 10 images, we increased the number of images in the sequence to 25 images. More details can be found in appendix 0.C.

Our model is evaluated on the 38 evaluation scenes not seen during training. The method is capable of synthesizing novel views using only a set of images with corresponding camera poses. Furthermore, it is able to adapt the visual appearance of the scene to specified weather and lighting conditions, without having access to observations of the scene under those target conditions. We show several qualitative examples of this. Fig. 3 shows that our method is able to change the visual appearance of images to match a target weather and lighting condition, and Fig. 4 shows a comparison with other methods.

It also becomes possible to interpolate between two latent variables corresponding to different conditions by defining $\mathbf{z}_{\alpha}=\alpha\mathbf{z_{c}}+(1-\alpha)\mathbf{z_{c^{\prime}}}$ for $\alpha\in[0,1]$. In Fig. 5, we observe that this enables getting realistic intermediate visual appearances that are not included in the original images. The model trained on appearance change of synthetic scenes can also be applied to change appearance of real scenes [12], as seen in Fig. 6 where we can see realistic appearance changes even though the model is not trained on any scenes from that dataset.

We now show quantitative rendering quality results. We show PSNR, SSIM [45] and LPIPS [48], where the images with changed appearance are evaluated against the corresponding ground truth images for the target weather and lighting conditions. In Table 1, we show how our method performs on all possible combinations of source and target conditions. Using the same source and target conditions corresponds to novel view synthesis without appearance change, which, as expected, gives better metrics, but the gap is small for some combinations, e.g. comparing Day into Evening with Evening into Evening. In Table 2, we compare our method with several 2D style transfer methods. We see that our method outperforms the 2D methods on the performance metrics for all combinations. We observe that performance varies for the different conditions and that adapting images from another condition into day is the most challenging, while transforming from day gives significantly higher performance for all methods. A comparison against Instruct-NeRF2NeRF [15] was also performed, but results varied largely for different scenes and prompts. Further details are included in appendix 0.C.

We show two consistency metrics [7] in Table 3. If $(x_{1},\ldots,x_{n})$ and $(y_{1},\ldots,y_{n})$ are two image sequences rendered from the same pose sequences, we define $\text{tOF}=\|\text{OF}(y_{t+1},y_{t})-\text{OF}(x_{t+1},x_{t})\|_{1}$, where OF is the optical flow computed via RAFT [38] and $\text{tLP}=\|\text{LPIPS}(y_{t+1},y_{t})-\text{LPIPS}(x_{t+1},x_{t})\|_{1}$. The metrics are low if the reference images and the rendered images yield similar optical flow and similar changes in LPIPS, which is assumed to correspond to a consistent rendering. We observe that our method significantly outperforms most of the 2D style transfer methods. Notably, Instruct-NeRF2NeRF exhibits poorer consistency results than anticipated, primarily stemming from two key factors. Firstly, the NeRF models generate low-quality novel views for some scenes. Secondly, there are inconsistent appearance changes in response to certain prompts, which results in unrealistic alterations that do not clearly preserve the scene content. CyEDA gives comparable consistency metrics for some scenarios, but gives less realistic rendered views as seen in Fig. 4 and Table 2.

We compare two different ways of learning latent appearance variables $\mathbf{z}_{c}\in\mathbb{R}^{d}$. One approach is to initialize a random $d$-dimensional vector with no enforced structure for each condition as a learnable parameter that is optimized jointly with the rest of the model. Another approach is to enforce structure by representing each condition as a fixed 2D-coordinate, placing them such that the evening condition is in between day and night, based on the assumption that one should pass through evening when going from day to night. These fixed 2D coordinates are then fed through a small learned fully-connected network to generate $\mathbf{z}_{c}$. Comparing the performance metrics for these two approaches, as can be seen in Table 4, shows that both approaches give similar performance. However, enforcing a structure on the latent space leads to more realistic lighting effects when interpolating, as can be seen in Fig. 5, giving the appearance of a sunset. Based on this, it was decided to use the latent appearance variable with the enforced structure for our experiments. The choice of dimension $d=136$, for the latent appearance variable, was made by observing that a higher dimension leads to a better ability to handle local appearance changes, such as turning on street lamps and removing shadows. More details and qualitative examples can be found in appendix 0.B.