DivCon: Divide and Conquer for Progressive Text-to-Image Generation

Given a text prompt $C$ describing an image, an LLM is adopted to predict a set of strings $O=\{s_{j}|j=1,2,...,n\}$ to describe the objects in the prompt. If the prompt primarily focuses on counting relationships among objects, each string $s_{j}$ includes both the name and quantity information of each object category $j$. If the prompt mainly describes spatial relationships, each string $s_{j}$ contains the object name and its spatial position in the image. After numerical and spatial reasoning, the numerical information and spatial relationships can be explicitly parsed from the complicated text prompt as intermedium conditions.

After numerical and spatial reasoning, both the meta prompt $C$ and intermedium conditions $O$ are fed to the LLM again to predict a list of name-box pairs $P=\{p_{i}|i=1,2,...,n\}$. Each pair $p_{i}=\{o_{i}:(l_{i},t_{i},r_{i},b_{i})|i=1,2,...,n\}$ consists of the object name $o_{i}$ and its bounding box’s coordinates $(l_{i},t_{i},r_{i},b_{i})$. Meanwhile, an in-context exemplars including a process of correcting the features of generated bounding boxes is provided to the LLM to improve the quality of the predicted layouts.

We employ a layout-to-image diffusion model with sampling optimization (Phung, Ge, and Huang 2024) as the generative model to generate an image conditioned on the text prompt and the predicted layout. During the denoising process, random noise $z_{T}$ is fed to the diffusion model to gradually denoise it to an image $z_{0}$ after $T$ time steps.

In the generated image $z_{0}$, each object is cropped using the bounding boxes in the layout and denoted as $x_{i}\in\mathbb{R}^{4\times h\times w}$, where $h\times w$ represents the height and width of the bounding box. Then, the CLIP similarity $S_{x_{i},o_{i}}\in(-1,1)$ is employed to evaluate the consistency between the cropped object and its name $o_{i}$ in the text prompt. The higher the similarity is, the higher the fidelity of the corresponding object is. Consequently, the objects with similarities higher than a specified threshold can be considered as easy samples that are ”good enough”. In contrast, other objects with low similarities are considered as hard samples that are more difficult to synthesize (e.g. ”sheep” and ”horse” in Fig. 3). Next, a binary mask $M$ is produced with bounding boxes of easy objects set to 1 and the rest regions set to 0.

The objective of second-round refinement is to preserve the “good enough” objects while reproducing the remaining ones to ease the difficulty of the diffusion model. To this end, a straightforward approach is to retrieve the latent representation of the objects in the first-round denoising process. Nevertheless, directly freezing the object regions during the denoising process produces inharmonious artifacts around the boundaries. To address this issue, we retrieve the latent representation $z_{T}$ of all time steps during the denoising process of the first-round generation. Then, the binary mask $M$ is adopted to crop high-fidelity regions in the latent representation. At each time step $T$ of the second-round forward pass, these cropped representations are employed to replace the corresponding regions in the noises:

$$z^{\prime}_{T}=z_{T}\circ M+z^{\prime}_{T}\circ(1-M)$$

(1)

where $\circ$ is element-wise multiplication and $z^{\prime}_{T}$ is the noise at $T$ step of the second-round forward pass. With the latent representation of high-fidelity objects (masked by $M$) being retrieved, these objects can be well preserved to encourage the model to focus on reconstructing the remaining ones. Meanwhile, the bounding boxes of “bad samples” $P^{\prime}$ are passed to the diffusion model as the layout conditions, enabling the model to concentrate more on refining $P^{\prime}$ to further improve the overall quality of the generated image.

To evaluate the performance of spatial and numerical reasoning in the text-to-image generation tasks, we utilize benchmarks HRS (Bakr et al. 2023) and NSR-1K (Feng et al. 2023b). The HRS dataset contains 3,000 prompts (labeled with objects’ names and specific counts) for numerical relationships and 1,002 prompts (labeled with objects’ names and corresponding positions) for spatial relationships. The NSR-1K dataset consists of 762 prompts for numerical relationships and 283 prompts for spatial relationships, all labeled with names, counts, bounding boxes, and ground truth images. Compared to NSR-1K, HRS exclusively employs natural language prompts, alongside a larger quantity of objects per sample and more intricate spatial relations. For these benchmarks, Stable Diffusion (Rombach et al. 2022), Attend-and-Excite (Chefer et al. 2023), Divide-and-Bind (Li et al. 2023a), GLIGEN (Li et al. 2023b), Layout-Guidance (Chen, Laina, and Vedaldi 2023b), and Attention-Refocusing (Phung, Ge, and Huang 2024) are included for comparison. In addition, we also include the dataset in (Lian et al. 2023) for evaluation, which includes four tasks: negation, generative numeracy, attribute binding, and spatial reasoning. As our method focuses on numerical and spatial reasoning, we select 100 prompts from these two tasks to construct a benchmark termed LLM-Grounded.

Fidelity. To evaluate the fidelity of the generated images, grounding accuracy between layouts/images and the text prompt is calculated. Particularly, YOLOv8 (Jocher, Chaurasia, and Qiu 2023) is employed to obtain the bounding boxes of the objects in the generated images. For counting, the number of bounding boxes is compared to the ground truths to calculate the precision, recall, F1 score, and accuracy (El-Nouby et al. 2019; Fu et al. 2020). For spatial relationships, the relative position between two bounding boxes is adopted following PaintSkills (Cho, Zala, and Bansal 2022) and LayoutGPT (Feng et al. 2023b). Specifically, if the horizontal distance between two bounding boxes exceeds the vertical distance, they are denoted as a left-right relationship. Otherwise, they are denoted as a top-bottom relationship.

Efficiency. To validate the efficiency and scalability of our method, inference runtime is evaluated on an AWS g5.xlarge GPU instance.

The HRS dataset. We present the results on the HRS benchmark in Tab. 1. As we can see, our DivCon outperforms previous T2I models by over 10% in numerical accuracy and about 45% in spatial accuracy, demonstrating its superior numerical and spatial reasoning performance. In addition, DivCon surpasses Attention Refocusing (Phung, Ge, and Huang 2024) on most metrics. Specifically, our layout prediction improves numerical accuracy by 3% and spatial accuracy by 2%, respectively. For image generation, the F1 score and accuracy achieve 2% and 4% gains, while the spatial accuracy is boosted by up to 6%. By dividing the T2I task into simple subtasks, our DivCon can better understand the text prompt to achieve higher fidelity.

The NSR-1K dataset. The results on the NSR-1K benchmark are presented in Tab. 2. It can be observed that DivCon outperforms other T2I models by over 20% in numerical accuracy and about 45% in spatial accuracy. As compared to Attention Refocusing (Phung, Ge, and Huang 2024), our DivCon produces competitive results in terms of layout accuracy. Meanwhile, DivCon achieves approximately 2% gains in precision, recall, and F1 score for numerical reasoning, and surpasses by nearly 4% in terms of spatial accuracy. Quantitative results in terms of image quality are shown in Tab. 3. As we can see, our DivCon produces comparable FID scores to Stable Diffusion and outperforms attention-refocusing with notable margins. This further demonstrates that our approach is able to produce images with better visual quality while maintaining high prompt fidelity.

LLM-Grounded benchmark. Table 4 presents the accuracy achieved by our method and LLM-grounded Diffusion. It is clear that our method achieves comparable performance to LLM-Grounded Diffusion on the numerical reasoning while outperforming it by 3% on the spatial reasoning task. This also validates the superiority of our method.

Runtime comparisons are presented in Tab. 5. Among techniques that require multiple forward passes to synthesize different regions separately, our DivCon produces notable efficiency gains. Particularly, LLM-Grounded takes 134.98s to generate an image, while our method achieves a $1.87\times$ speedup. Compared with methods with a single forward pass, our method (1 Iter) produces competitive results with comparable efficiency.

Figure 5 illustrates the qualitative comparison of DivCon and baselines. In all cases, our DivCon can accurately predict the object quantities and the spatial relations from the text prompts.

Large Quantity of Objects. Given a prompt containing more than one object category (Fig. 5), Stable Diffusion (Rombach et al. 2022) and Attend-and-Excite (Chefer et al. 2023) often generate an insufficient quantity of objects, while Attention-Refocusing (Phung, Ge, and Huang 2024) sometimes omits one category of objects or generates an excessive number of objects. For instance, the prompt in Fig. 5(b) contains five spoons and four oranges. However, Stable Diffusion and Attend-and-Excite cannot synthesize sufficient spoons or oranges. Meanwhile, Attention-Refocusing generates an excess number of objects, with spoons appearing incomplete due to missing parts. As compared to these methods, our DivCon achieves superior numerical accuracy.

Complicated Spatial Relation. Given a prompt containing complex spatial relationships among more than two object categories, previous T2I models struggle to generate correct spatial relations. For example, Stable Diffusion and Attend-and-Excite miss certain object categories (Fig. 5(d)). Meanwhile, Attention-Refocusing tends to produce distortion or blending artifacts around the boundaries of objects (Fig. 5(c)(d)) or merges the color of the banana with the person (Fig. 5(d)). In contrast, our DivCon produces accurate object categories and spatial relationships, exhibiting superior performance in these challenging cases.

User Study. We conduct a user study to evaluate the images generated by different methods. To be specific, we recruit 17 people with different backgrounds and asked them to rank the images generated by different methods according to their fidelity to the original prompt. Lower ranks indicates better alignment. A total of 20 examples are randomly selected from the NSR-1K benchmark. As shown in Fig. 6, our method achieves the best performance in terms ofaverage rank, which demonstrates the superior perceptual quality of our results.