EucliDreamer: Fast and High-Quality Texturing for 3D Models with Depth-Conditioned Stable Diffusion

Figure 3 shows some qualitative results. Overall, the generated textures are of superior quality in terms of the realism, the level of detail, cross-view consistency and aesthetics. Notice on the dumpster model, EucliDreamer even accurately captured the specular pattern of the metallic material, a special trait resembling texture baking. For the compass model, not only the coloring is realistic, it also accurately recovers the details such as the characters and the small decorations on the rim.

We invited 28 participants for the user study who are professionals in game industry and 3D content creation, including engineers, researchers and artist. Most of them are experienced with the 3D modeling workflow and downstream applications, thus we expect them to be decent critics of generation quality.

In the study, we present four sets of textured objects: an ambulance, a dumpster, a compass, and a backpack, via online questionnaires. Each set contains four textures generated by CLIPMesh, Latent-Paint, Text2Tex [3], and our method. The participants are asked to rank the results in terms of generation quality, taking into account correctness, fidelity, level of detail and overall aesthetics.

The results are shown in Figure 5 and  5. Most participants selected their top 2 choices from either SOTA Text2Tex [3] or ours. For the ambulance and dumpster objects, the votes are unanimously toward our generated textures. For the compass, about two-thirds voted for ours. The votes for the backpack are diverse for the second-best quality object.

Overall, the study indicates a high acceptance of the textures generated by our model.

Figure 6 shows the SDS loss throughout the training steps. In general, it takes EucliDreamer around 4300 steps to converge the loss. Note that for all 4 objects being tested, most progress is already made before 2500 steps. In contrast, under identical experimental conditions and using the same set of hyperparameters, DreamFusion [22] requires over 10,000 steps to converge. Thus, the runtime of EucliDreamer is about half of that of DreamFusion. This comparison highlights the superior convergence speed and computational efficiency of our method.

Figure 7 compares the texturing results on 4 models with and without depth-conditioning. We observe that depth-conditioning is especially helpful for ensuring correctness. For instance, without depth information, windows can show up on the roof of the house, and food texture can incorrectly appear on the wall of the food cart container. In addition, depth-conditioning seems to improve the detail quality and texture sharpness. This might due to that it removes a lot of ambiguities and prevents many modes from mixing together.

Our second set of experiments focuses on the diversity of the generated textures. By providing input text prompts of the model content and styles, one can use EucliDreamer to generate 3D textures for a given 3D mesh in certain styles. For example, a prompt of ”a building, damaged, dirty, 3D rendering, high-quality, realistic” will give the building model a rusty and pale feeling, as shown in Figure 8 in Supplementary Materials. Through simply alternating the text prompts, EucliDreamer can provide diverse combinations of color tones, brush strokes, lighting and shadows in the textures, which highlights its flexibility and ”creativity”.

Our method, like many other SDS-based methods, requires that all surfaces of an object to be visible from some angles. Therefore, it may encounter issues with extremely complex structures with holes or obstructions. This might be mitigated by splitting the model into parts.

Also, as our method relies on text-to-image diffusion models that were trained to produce realistic natural images, which may produce ”baked-in” effects into the texture, such as shadows and highlights. This may be undesirable for use cases that requires re-lightable 3D assets. As a future work, a possible solution is to fine-tune a Stable Diffusion variant that produces images with flat lighting style.

We present a simple and effective solution for automatically generating textures for 3D meshes. High-quality texturing is one of the labor-intensive stages in 3D modeling workflow, which can take an experienced artist many days to complete for a single object. We hope our method can reduce the cost for preparing 3D assets in applications like gaming, animation, virtual reality [33, 15, 9] or even synthetic training data [29, 23, 12] for computer vision models [13, 28, 11, 6], and inspire further research in this direction.

For the main experiment, we selected the following parameters that performed the best in  2. The Adam optimizer has a learning rate of 0.01.

The prompt keywords below are added in addition to the original phrase of the object (e.g. “a compass”).

prompt:
  "animal crossing style, a house,
  cute, Cartoon, 3D rendering, red
  tile roof, cobblestone exterior"
negative_prompt:
  "shadow, greenshadow, blue shadow,
  purple shadow, yellow shadow"

The texture resolution is set to 512*512.

ThreeStudio

(https://github.com/threestudio-project/threestudio)

is a unified framework for 3D modeling and texturing from various input formats including text prompts, images, and 3D meshes. It provides a Gradio backend with easy-to-use frontend with input configurations and output demonstration.

We forked the framework and added our approach with depth conditioning. A depth mask is added to encode the depth information.

with torch.no_grad():
    depth_mask =
      self.pipe.prepare_depth_map(
        torch.zeros((batch_size, 3,
          self.cfg.resolution,
          self.cfg.resolution)),
        depth_map,
        batch_size=batch_size,
        do_classifier_free_guidance=True,
        dtype=rgb_BCHW.dtype,
        device=rgb_BCHW.device,
    )

We used a Nvidia RTX 4090 GPU with 24GB memory. This allows us to run experiments with a batch size up to 8 and dimension 512*512. In theory, a GPU with more memory will allow us to generate textures with higher definition and in better quality.

SDS [22] is notoriously known for its lack of diversity and overly smoothed texture. This is because SDS is mean-seeking. Intuitively, a single prompt has multiple plausible generation results. SDS tends to create an average of them (mean seeking). Variational Score Distillation (VSD) is proposed in ProlificDreamer [32] to mitigate this problem by introducing a variational score that encourages diversity. However, in depth-conditioned texturing, there are fewer plausible generation results given a single prompt and depth. This leads to the diversity and details in our approach (mode seeking), even though VSD is not used. Through experiments shown in Figure 9, our approach produces better results without VSD. Thus VSD in ProlificDreamer [32] is not required when Stable Diffusion depth is used.

The most straightforward way of depth-conditioned SDS is to use ControlNet [35]. However, empirically we found that the SDS gradient from ControlNet [35] is much noisier than SD-depth. SD-depth concatenates depth with latents. The SDS gradients flow to depth and latents separately, which is less noisy. As shown in Figure 9, we demonstrate better generation results with Stable Diffusion depth than ControlNet depth.

Elevation range defines camera positions and angles at the view generation step. When it is set between 0-90 degrees, cameras face down and have an overhead view of the object. We find that fixed-angle cameras may have limitations and may miss certain angles, resulting in blurry color chunks at the surface of the object. One way to leverage the parameter is to randomize camera positions so all angles are covered to a fair level.

The sampling min and max timesteps refer to the percentage range (min and max value) of the random timesteps to add noise and denoise during SDS process. The minimum timestep sets the minimum noise scale. It affects the level of detail of the generated texture. The maximum timestep affects how drastic the texture change on each iteration or how fast it converges. While their values may affect quality and convergence time, no apparent differences were observed between different value pairs of sampling steps. In general, 0 and 1 should be avoided for the parameters.

In the context of 3D texture generation, a large learning rate usually leads to faster convergence. A small learning rate creates fine-grained details. We found learning rate of 0.01 with the Adam optimizer is good for 3D texture generation.

We set batch sizes to 1, 2, 4, and 8 respectively. Results show that a larger batch size leads to more visual details and reduces excessive light reflections and shadows. More importantly, large batch size increases view consistency. This is because gradients from multiple views are averaged together and the texture is updated once. A batch size of 8 is enough from our observation. A batch size of more than 8 does not lead to a significant reduction in the number of iteration steps.

Intuitively gradient clipping can avoid some abnormal updates and mitigate the Janus problem, shown in Debiased Score Distillation Sampling (D-SDS) [14]. In the context of texture generation, we do not observe Janus problem due to the fact that the mesh is fixed. In our experiments, there’s no noticeable benefit of gradient clipping.

The guidance scale specifies how close the texture should be generated based on input text prompts. We find that the guidance scale affects diversity and saturation. Even with the depth conditional diffusion model, a high guidance scale like 100 is still required.

Negative prompts in Stable Diffusion can fix some artifacts caused by SDS. Adding negative prompts like ”shadow, green shadow, blue shadow, purple shadow, yellow shadow” will help with excessive shadows.

We found that adjusting the camera distance can improve both resolution and generation quality. We experimented with three different ranges of camera distance, shown in Figure 10. The distance range of [1.5, 2.0] produces the optimal outcomes regarding the quality of textures and colors. This can be attributed to its ability to simulate viewpoints most representative of conventional photography, similar to the usual perspectives from which an ambulance is observed.

Dreambooth3D [24] was initially proposed as an extension of DreamFusion [22] to enable image-to-3D shape generation. We partially finetuned Stable Diffusion depth with the user-provided image(s), generated textures with finetuned stable-diffusion-depth from step 1, fine-tuned Stable Diffusion depth again with outputs from step 2, and then generated final results with finetuned stable-diffusion-depth from step 3. Shown in Figure 11, combined with Dreambooth3D [24], our approach generates satisfactory textures given a 3D mesh and a single image as inputs.