AI-Driven Stylization of 3D Environments

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction and novel view synthesis, but pose significant challenges in editing. Early initiatives like NeRF-Editing (Yuan et al., 2022) were limited to simpler deformations. NeuMesh (Yang et al., 2022) advanced this by enabling more complex edits such as texture modifications and geometry deformation. Despite progress, current NeRF editing tools still lack the functionality of traditional 3D software and require significant manual input to achieve high-quality results.

The advent of generative NeRF editing, combining text-to-3D generation with NeRF modifications, brought new methodologies like Set-the-Scene (Cohen-Bar et al., 2023) and Compositional 3D (Po and Wetzstein, 2023). These methods allow for more controlled scene generation using proxy objects. More recently, Instruct-NeRF2NeRF (Haque et al., 2023) introduced an Iterative Dataset Update (IDU) strategy, leveraging InstructPix2Pix (Brooks et al., 2023) to edit NeRF datasets for more refined control over the edits.

Building on these innovations, our method uses SIGNeRF (Dihlmann et al., 2024), which introduces a system that integrates edits and object generation directly within an existing NeRF scene. Using a reference sheet image grid, SIGNeRF maintains multi-view coherence and enhances control over the generation process.

Immersive 3D modeling in Virtual Reality (VR) has transcended the spatial constraints of traditional design methods (Wolfartsberger, 2019). VR platforms like Dreams (Dreams, 2020), Figmin XR(XR, 2022), and Horizon Worlds (Worlds, 2020) have expanded user-interaction models, demonstrating VR’s capability to facilitate complex design tasks through intuitive interfaces. Han et al. (Han et al., 2023) further explore HCI advances in VR, focusing on enhanced gesture recognition to enable more natural and effective user interactions within virtual spaces.

To stylize the set of primitives according to the user’s prompt while maintaining their overall structure, we employ InstructPix2Pix (Brooks et al., 2023), a state-of-the-art text-guided image editing model. InstructPix2Pix is particularly suitable for this task due to its ability to follow natural language instructions to modify specific parts of an image while preserving its overall structure and unedited regions. By leveraging the power of large-scale pre-training on a diverse set of image editing instructions, InstructPix2Pix enables us to generate high-quality stylized images that align with the user’s intent while retaining the structural integrity of the primitives, as shown in Figure.2. The model’s flexibility in handling a wide range of editing tasks and its capacity to generate realistic and consistent results make it an ideal choice for our primitives stylizer component.

After obtaining a stylized single-view image from the primitives stylizer, we aim to generate a textured mesh that maintains view consistency, minimizes photometric loss under the supervision of the given view image, and closely matches the shape of real-world objects. The underlying representation can either be surface representations (SDF, mesh) or volumetric representations (3D Gaussian blob, NeRF). In this work, we consider two state-of-the-art models for mesh generation: the Convolutional Reconstruction Model (CRM)(Wang et al., 2024) and the Gaussian Reconstruction Model (GRM)(Xu et al., 2024).

We use SIGNeRF (Dihlmann et al., 2024) to integrate the newly generated mesh into the scene. This approach provides a technique of utilizing the ControlNet (Zhang et al., 2023) to consistently augment the existing scene across different view angles. As NeRF is trained from 2D images, the network essentially provides a mapping from stacked 2D images into the 3D environment, hence enabling the powerful 2D image editing techniques to also take effect in the 3D environment. In the SIGNeRF (Dihlmann et al., 2024) pipeline, a set of reference camera positions are first chosen to consistently generate reference image grids, encoding color, control mask, and depth information. Then, utilizing the one slot intentionally left blank in the image grid, the SIGNeRF provides a way to iteratively update the original images in the NeRF dataset. Notice that the updating process takes the current style of NeRF scene into consideration as the pipeline utilizes ControlNet.

This approach, therefore, provides two ways to insert new objects into a NeRF scene with their styles aligned: 1) directly adding new images to the NeRF dataset and 2) modifying existing images from the NeRF dataset. However, limitations of such methods were discovered during our experiments. For the first method, the resulting NeRF dataset exhibits a spatial inconsistency across different view angles, primarily because of the lack of modification on the original dataset. On the other hand, only modifying existing images in the NeRF scene may have results highly dependent on the position chosen to insert the new object, as the original NeRF scene might not have enough camera view angles to capture sufficient information about the new objects. Another concern of utilizing the NeRF occurs when we iteratively add new objects to the scene. Since image editing happens on all camera view images, including both original NeRF data and the view captures generated for previously inserted objects (via approach 1), the generated view captures surrounding objects inserted earlier will accumulate higher noise compared to the augmented view captures of the latest object being inserted.

We evaluated our pipeline on an empty apartment scene with three objects of different shapes and sizes that are commonly found in apartments: (1) a sofa, (2) a lamp, and (3) a bed. These three objects are iteratively added into the apartment scene at different locations, simulating how users would like to iteratively add their objects to their target scene. After adding each object, SIGNeRF retrains the scene to stylize the object according to the scene background.

We capture the empty apartment scene with an iPhone and train an initial NeRF scene with Nerfstudio (Tancik et al., 2023). The collected NeRF dataset has a total of 303 images and is trained in 15 mins. This scene will be used as the base for our pipeline to add objects into. Then, we utilized our proposed pipeline to add the three objects iteratively into the scene in the order of: (1) a sofa, (2) a lamp, and (3) a bed. These objects are added using the first way to add objects in the scene mentioned in section 3.3, which is to directly add new images of the object into the original NeRF dataset.

We show the before-after comparison of the apartment scene in Figure 4. In addition, we show the generated results for each of the three objects for each step of the entire pipeline in Figures 5 and 6. As seen in Figure 4, the sofa and lamp object appears translucent. This happens because they are the first two objects added into the scene. Each time we iteratively add an object into the scene, the conditioning signals such as depth and mask information of prior objects are lost. Moreover, since we utilized the first way to add objects into a scene with SIGNeRF, only the newly added images to the NeRF dataset contains the new object. The original images from the NeRF dataset are not updated, even at locations where the new object is supposed to be located. Thus, these factors contribute to the blurry and translucent results generated.

From Figure 6 we can observe that the objects are not consistent across different view angles, especially for the bed and for certain angles of the sofa. Upon further investigation, we discover that this is due to the inconsistencies of the generated dataset by SIGNeRF across different view angles. This can be seen from Figure 7, where the bedsheets look white from certain angles but brown with a wood-like texture from other angles. As for the sofa, it is green from most view angles but the side, which looks gray. This shows that even by conditioning the Control-Net stylization through reference grids, it is not robust enough to produce view-consistent results.

Given the unsatisfactory blurriness and view-inconsistencies observed in the outputs from the previous pipeline, an alternative approach was implemented. The model generated by the GRM was integrated into the Unity scene, replacing the primitive forms, as depicted in Figure 3. The resulting meshes are robust, allowing for user interactions such as moving and scaling. However, the material and color attributes of the objects, determined by the ControlNet results, do not correspond with their environmental context. This contrasts with our earlier pipeline, which stylized objects in accordance with the surrounding environment.