Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation

With the rise of large pre-trained 2D models (Radford et al. [2021], Saharia et al. [2022], Rombach et al. [2021], DeepFloyd [2023]), text-to-3D generation has experienced a remarkable leap in progress. Existing text-to-3D methods can be grouped into two main approaches. The first group of works (Jun and Nichol [2023], Nichol et al. [2022], Gupta et al. [2023]) adopts only captioned 3D data for the text-to-3d generation task. However, due to the fact that captioned 3D data is limited in scale and diversity, these methods are very dependent on access to large-scale 3D data and can only generate objects from a limited set of categories.

A second well-used paradigm is to tackle 2D supervision to supervise a 3D representation model such as NeRF (Mildenhall et al. [2020]). Early works use CLIP (Radford et al. [2021]) to optimize NeRF by projecting the text and images of the 3D generation into a shared latent space and aligning them with the CLIP loss (Jain et al. [2022], Mohammad Khalid et al. [2022], Xu et al. [2023b], Lee and Chang [2022], Michel et al. [2022], Wei et al. [2023]). Because CLIP is not a generative model, these methods tend to produce generations that lack geometric fidelity. Improving upon previous works, Poole et al. [2022], Wang et al. [2023a] are the pioneer works that leverage 2D pre-trained diffusion models for text-to-3D generation. They propose score distillation sampling (SDS) that supervises 3D representation models, such as NeRF (Mildenhall et al. [2020]) or DMTet (Shen et al. [2021]), by guiding their corresponding rendered images towards areas of high probability density that are contingent on the text. Expanding upon the aforementioned research based on SDS, existing works explore different supervision methods (Wang et al. [2023b], Yu et al. [2023b]) and various 3D representations such as Gaussian Splatting (Vilesov et al. [2023], Tang et al. [2023a], Yi et al. [2023], Chen et al. [2023d]).

Building upon these seminal works, a large collection of 2D-lifting methods (Huang et al. [2023], Wang et al. [2023b], Yu et al. [2023b], Seo et al. [2023a], Tsalicoglou et al. [2023]) focus on improving fidelity and ameliorating existing issues, such as over-saturation (generations tend to appear cartoon-like). In addition, prior works also tackle the poor view-consistency issue and the Janus problem (Shi et al. [2023b], Zhao et al. [2023], Zhu and Zhuang [2023], Yu et al. [2023a], Ma et al. [2023], Li et al. [2023b], Seo et al. [2023b], Hong et al. [2023]). Moreover, efficiency-centric methods emphasize reducing the generation time through parallel sampling (Zhou et al. [2023]), distilling a multi-view 2D diffusion model (Chen et al. [2023b]), and splitting the task into sparse view generation and reconstruction (Li et al. [2023a]). Other methods (Lin et al. [2023a], Chen et al. [2023a]) focus on generating high-resolution 3D contents using a coarse-to-fine approach and exploring efficient 3D representations, as well as producing generations with fine details (Qiu et al. [2023]). Several other works improve generation diversity (Huang et al. [2023], Wang et al. [2023b], Seo et al. [2023a], Lu et al. [2023]).

NeRF (Mildenhall et al. [2020]), with its ability to take a collection of images from different viewpoints and reconstruct the underlying 3D scene, has shown tremendous potential in progressing the image-to-3D research direction. However, NeRF is greatly constrained by the large demand for dense viewpoint data with significant viewpoint overlaps. Subsequent works focus on relaxing this constraint under the single-image-to-3D paradigm, such as performing generation through progressively deforming the mesh (Wang et al. [2018]), leveraging NeRF with learned scene prior (Yu et al. [2021]), adding semantic/geometric regularization (Xu et al. [2022]), or leveraging differentiable surface modeling and differentiable rendering (Gao et al. [2022]). However, these methods are limited by the lack of large-scale 3D data as well as suffer from numerous issues such as inconsistency and long generation times.

Later works leverage 2D models as priors in the 3D generation process (Deng et al. [2023]) and take a coarse-to-fine approach (Tang et al. [2023b], Seo et al. [2023a], Qian et al. [2023]. Subsequent methods likewise adopt the 2D diffusion prior to the novel view synthesis task (Watson et al. [2022], Chan et al. [2023], Guangcong et al. [2023], Liu et al. [2023c], Lin et al. [2023b]). Expanding upon established foundations, the current image-to-3D task features several orthogonal research focuses, include enabling 360-degree reconstruction of objects (Melas-Kyriazi et al. [2023], Xu et al. [2023a]) and scenes (Sargent et al. [2023]), improving generation efficiency (Liu et al. [2023b, a]), and ameliorating 3D inconsistencies (Ye et al. [2023], Lin et al. [2023c], Weng et al. [2023], Long et al. [2023], Liu et al. [2023d]).

One major approach to ameliorate the multi-view consistency issue in 2D-lifting methods is through leveraging multi-view diffusion models, which requires modeling the joint distribution of an object’s information from multiple views. The multi-view information can take the form of latent intermediary features (Tang et al. [2023c]), epipolar lines (Tseng et al. [2023]), or whole images (Shi et al. [2023b]). Information-sharing is achieved with a variety of techniques, such as placing different viewpoint images into a tiling layout (Shi et al. [2023a], Tsalicoglou et al. [2023], Li et al. [2023a]), denoising several image views with different noises and then sharing information across images with an attention mechanism during each denoising step (Shi et al. [2023b], Liu et al. [2023d], Tang et al. [2023c], Xu et al. [2023c]), or adopting a coarse-to-fine approach by incorporating an orthogonal-view diffusion prior (Zhao et al. [2023]).

NeRF (Mildenhall et al. [2020]) is a neural inverse rendering approach based on volume rendering. In formulae, for each point in space and viewing direction unit vector in $\mathbb{R}^{3}$, NeRF is a differentiable volumetric renderer parameterized by a neural network $\phi$ that returns the density $\sigma$ and RGB color $\mathbf{c}$. In particular, NeRF renders each pixel via volume ray casting. Given a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ with ray origin $\mathbf{o}$ the camera center, it approximates the following integral via numerical quadrature:

where $T(t)$ is the transmittance function that predicts the probability that the ray travels from $t_{n}$ and terminates at $t_{f}$. NeRF is trained in an end-to-end manner on a set of posed images to minimize the reconstruction loss between the rendered color and the ground-truth color of the posed images. For simplicity, denote $x=g(\phi,c)$ a rendered view at some desired camera pose $c$, which is obtained by applying Eq. 1 on every pixel.

DDPM (Ho et al. [2020]) is described as two Markov processes yielding latent variables $\{z_{t}\}_{t=1}^{T}$ from a data distribution $x\sim p_{\text{data}}$. Specifically, it defines a forward process $q(z_{t}\mid z_{t-1})$ and a reverse process $p(z_{t-1}\mid z_{t})$, both being Gaussians. While $q$ is hand-crafted, we learn $p$ using a deep neural network (DNN) with parameters $\theta$, which we denote $p_{\theta}$. In formulae, given $x$ the diffusion process $q$ generates latent variables $\{z_{t}\}_{t=1}^{T}$ with decreasing signal-to-noise ratio such that:

where $\alpha=1-\beta_{t}$, $\bar{\alpha}_{t}=\prod_{i=0}^{t}\alpha_{i}$, and $\beta_{t}$ is a hyper-parameter that controls the noise level at $t$. Since $q(z_{t})$ converges towards an isotropic Gaussian distribution as $t\rightarrow\infty$, we define the prior of the reverse process $p_{\theta}$ as an isotropic Gaussian distribution, $p_{\theta}(z_{T})=\mathcal{N}(0,I)$. Given a noising schedule $\sigma^{2}_{t}\propto\beta_{t}$, the reverse process runs from $t=T$ to $t=1$ and at each step we sample from $\mathcal{N}(\mu_{\theta}(z_{t},t),\sigma_{t}^{2}I)$ where:

In particular, the DNN $x_{\theta}(z_{t},t)$ predicts the signal $x$ such that it minimizes the following loss $J(\theta)$ defined as

where $w(t)$ is a positive weighting function that depends on $t$ and $z_{t}$ is obtained using Eq. 2. Additionally, one can use classifier-free guidance to decrease the diversity of the samples while increasing the quality of each individual sample w.r.t to some condition $y$ (e.g., text). Notably, the neural network’s output is modified as follows

where $\gamma$ is a scalar coefficient that balances the aforementioned trade-off.

Neural inverse rendering techniques such as NeRF optimize a 3D scene representation based on image observations. Thus, one can use a text-to-image generative model for optimizing a 3D scene representation. Prior work refers to this as score distillation sampling (Poole et al. [2022]), for the sampling process of this generative model is done by distilling the learned score function of a 2D diffusion model into NeRF. Leveraging text-to-image generative models has the following advantages. First, compared to 3D data, there is abundant 2D data. Second, it allows leveraging other abilities in 2D generative models, such as editing or controllability (Li et al. [2023c], Haque et al. [2023], Chen et al. [2023c]). Finally, it can also be applied in a similar manner to video generation (Singer et al. [2023]). In practice, we generate multi-view images using a text-to-image diffusion model and minimize the score distillation loss on these images by optimizing NeRF.

Score Distillation (Poole et al. [2022]) encourages NeRF to render an image $x$ such that it belongs to the distribution of plausible images as evaluated by a diffusion model $\hat{x}_{\theta}(\cdot)$. In practice, it consists of optimizing $\phi$ to minimize the following loss function:

where $z_{t}$ is sampled from the forward process and $y$ is some condition (e.g., text). In practice, the gradient of the diffusion model is detached, yielding the following gradient for NeRF’s parameters $\phi$:

Eq. 1 shows that the optimization process in NeRF is applied in a pixel-wise manner. Thus, pixel-wise view consistency is the signal for learning the density and color. Additionally, the gradient signal in Eq. 7 must maintain multi-view consistency to match the condition $y$. In other words, accurate text-to-3D asset generation requires view consistency w.r.t $y$ and between views at the pixel level.

Our score distillation uses two different scores for supervision: a text-conditioned multi-view diffusion model and an image-conditioned novel-view diffusion model. These two learning signals are complementary; one maintains multi-view consistency with text, while the other maintains consistency between views. In this paper, we denote a view as $x$ and a set of views (i.e., multi-view) as $\mathbf{x}$ where we index each view as $\mathbf{x}^{i}$.

Text-conditioned generation. Denote with $i$ the index of views generated from the text condition. Following prior work (Shi et al. [2023b]), we start by sampling a set of camera poses $\mathbf{c}$ and render these views $\mathbf{x}=g(\phi,\mathbf{c})$, which we call reference views. In particular, the views $\mathbf{x}$ are chosen such that they are orthogonal to each other. For each view, we sample a timestep $t$ and compute the forward process of the diffusion process $\mathbf{z}_{t}^{i}$. Given the text $y$ and the set of noised views $\mathbf{z}_{t}$ rendered from NeRF, the text-conditioned diffusion model $\hat{x}_{\theta_{1}}(\mathbf{z}_{t};y,\mathbf{c},t)$ computes score function w.r.t to $\mathbf{z}_{t}$, yielding an update direction towards higher density regions. Additionally, we modify the score function to include classifier-free guidance as in Eq. 5 and compute score distillation to obtain gradient update as in Eq. 7.

Image-conditioned generation. Prior work utilized image-conditioned diffusion to generate 3D content from a single 2D reference image, while our approach differs in how the model is leveraged. Instead of providing the reference as an additional input to the model, we innovatively employ it as extra supervision to guide different views and ensure fine-grained multi-view consistency. Denote with $j$ the index of novel views generated from images. We render extra views $\mathbf{x}^{j}$ at camera extrinsic $\mathbf{c}^{j}$. Denote the relative camera extrinsic $\mathbf{c}^{(j\rightarrow i)}$ from camera position $i$ to $j$. In formulae, the image-conditioned diffusion model takes rendered image $\mathbf{x}^{j}$ along with the relative camera extrinsic $\mathbf{c}^{(j\rightarrow i)}$ as conditioning. In a similar manner, we sample $t$ from the uniform distribution. The model was trained to compute the score function for novel views $\mathbf{z}^{i}_{t}$, denoted as $\hat{x}_{\theta_{2}}(\mathbf{z}_{t}^{i};\mathbf{x}^{j},\mathbf{c}^{(j\rightarrow i)},t)$. Following Liu et al. [2023c], we compute the score distillation at $\mathbf{z}^{i}_{t}$, which was previously rendered. The resulting score distillation encourages NeRF to render consistent views $\mathbf{x}^{j}$, such that the pre-trained diffusion models can predict accurate novel views $\mathbf{x}^{i}$.

Text and image score distillation. We modify Eq. 6, such that the gradient signal becomes:

where $\lambda_{t}$ and $\lambda_{i}$ are scale factors of the text and image diffusion model, respectively. The score distillation process adds a score toward view consistency and another score toward text multi-view consistency. This optimization process provides NeRF with enough constraints to predict accurate densities and colors as shown in Figure 4.

Models and Representations. We utilize MVDream’s (Shi et al. [2023b]) pre-trained model as our multi-view diffusion model, and Zero123-xl, provided by Zero-1-to-3 (Liu et al. [2023c]), as our novel view image-conditioned diffusion model. For 3D representation, we implement ThreeStudio’s (Guo et al. [2023a]) implicit volume approach, consisting of a multi-resolution hash grid and an MLP network for predicting voxel density and RGB values.

View Selection. For each camera view to render, we first randomly select cameras with a field-of-view (fov) between [15, 60] and an elevation between [0, 30] for the multi-view diffusion model. The camera distance is set as the object size (0.5) multiplied by the NDC focal length and a random scaling factor ranging from [0.8, 1.0]. We then randomly select views from the above set as reference views for the novel view diffusion model. For each reference view, we choose an additional random camera with the same fov and an elevation between [-30, 80] before applying the novel view image-conditioned diffusion model. The batch size starts at 8 and 12 for the multi-view and novel view models, respectively, and then decreases to 4 and 4 after 5,000 iterations.

Optimization. The 3D model is optimized for 10,000 steps using an AdamW (Kingma and Ba [2014]) optimizer. The learning rate for the hash-grid and MLP components is set to 0.01 and 0.001, respectively. Score distillation sampling is applied, with the maximum and minimum time steps decreasing from 0.98 to 0.5 and 0.02 over the first 8,000 steps, respectively. Both the loss scale factors $\lambda_{t}$ and $\lambda_{i}$ are set to 1.0. The rendering resolution begins at 64$\times$64 and is increased to 256$\times$256 after 5,000 steps. Guidance scales of 50.0 and 3.0 are used for the multi-view and novel-view models, respectively.

We evaluate our method on the T³Bench (He et al. [2023]), a comprehensive text-to-3D benchmark containing diverse text prompts across three categories: Single Object, Single Object with Surroundings, and Multiple Objects, with 100 distinct prompts in each category. We also leverage the two automatic metrics proposed in T³Bench (He et al. [2023]) for evaluating the quality and text alignment of generated 3D scenes. The quality metric captures multi-view scene images and utilizes text-image scoring models and a regional convolution mechanism to measure overall quality and detect view inconsistency issues. The alignment metric employs multi-view scene captioning with BLIP (Li et al. [2022]) and aggregates the captions with a large language model, which measures how well the caption covers information in the original text prompt.

Apart from the analysis of 3D content, we also conduct a text-2D consistency analysis by applying the CLIP (Radford et al. [2021]) cosine similarity between the original text prompt and the image captured at a fixed position and orientation. In practice, we first normalize the 3D mesh into a cube with a $[-1,1]$ range, then capture an image at the front of the 3D mesh, with a distance of $2.2$ and a focal of $3.0$.

For extracting the mesh model, we adopt the Marching Cubes (Lorensen and Cline [1998]) algorithm for our method and the original mesh extracting algorithm in T³Bench (He et al. [2023]) for other methods. We also follow the settings of T³Bench to apply mesh geometry simplification to a maximum of 40,000 faces before texture extraction.

We compared our method with the seven advanced text-to-3D methods, including Dreamfusion (Poole et al. [2022]), Magic3D (Lin et al. [2023a]), LatentNeRF (Metzer et al. [2023]), Fantasia3D (Chen et al. [2023a]), SJC (Wang et al. [2023a]), ProlificDreamer (Wang et al. [2023b]), and MVDream (Shi et al. [2023b]). For all the methods, we adopt the original implementation of ThreeStudio (Guo et al. [2023b]) to generate 3D contents, and the original codebase of T³Bench to extract 3D meshes and evaluate metrics.

Our method achieves the best performance among all the text-to-3D methods on the average T³Bench score across all the prompt sets, as shown in Table 1. On the one hand, compared with the performance of a single object, our precision of reconstruction is significantly higher than other methods, proving our superiority. On the other hand, for the more difficult multiple objects task, our method shows a strong result, which is attributed to the co-supervision of text and image features.

As illustrated in Figure 3, the 3D-generated content from most existing methods lacks texture details and clear geometry, making it difficult to faithfully reproduce all information from textual prompts. DreamFusion produces renderings lacking in fine texture details and suffers from a high failure rate. Magic3D and LatentNeRF generate improved texture details but still demonstrate poor geometric quality. SJC exhibits a tendency to output less compact geometry, an attribute unfavorable to the final shape of the 3D content. By contrast, Fantasia3D typically outputs compact and well-defined geometry alongside richer textures. However, its performance declines when processing complex textual prompts, often yielding completely erroneous geometry. While ProlificDreamer utilizes LoRA (Hu et al. [2021]) to finetune the diffusion model during optimization and variational score distillation, which benefits the generation of rich details, the geometry quality is often poor with excessively incorrect shapes. MVDream performs much better geometrically, but the overall generation quality and alignment to the original text prompt need further improvement. Our proposed method demonstrates superior performance across all aspects; it can generate detailed 3D content with accurate, high-quality geometry that effectively reflects the textual description, even on challenging prompt sets.

To further validate the effectiveness, we also construct the CLIP cosine similarities comparison test between the front view image and the original text prompt, as shown in Table 2. The results also support that our method retains better consistency with the original text prompt compared to other methods.

To demonstrate the effectiveness of the image-conditioned diffusion module, we first conducted quantitative experiments, adjusting only the inclusion of the module while keeping all other experimental conditions constant. As shown in Table 3, the addition of the image-conditioned module effectively improved response quality and alignment with the original text prompt.

To show how the image-conditioned module effectively utilizes consistency modeling within a robust, plug-in architecture, we conducted a case study. The first row of Figure 4 demonstrates how viewpoint inconsistencies in text-conditioned diffusion models can lead to instability when rendering special materials like glass during optimization, often resulting in a loss of density. However, by continuing to constrain cross-view consistency during training, the addition of the image-conditioned module ameliorates such density collapse. Furthermore, when faced with certain prompts (e.g. the second row of Figure 4), the textual inconsistency of diffusion models gradually accumulates erroneous excess density. As these erroneous densities tend not to match the 3D prior in the image-conditioned module, they can be suppressed by our extra module during optimization. Additionally, the third row of Figure 4 illustrates the color drift and blending issues that emerge with 2D diffusion guidance. Our module, which facilitates cross-view consistency, effectively addresses these challenges as well.