Detector Guidance for Multi-Object
Text-to-Image Generation

The diffusion denoising models [1, 2, 4] are a type of deep generative models that employ an iterative denoising process to generate samples. These models utilize noise-conditioned score networks, as described in [13, 14], and denoising score matching objectives, as described in [15, 16], at varying noise levels. They have demonstrated successful applications in various domains, such as text-to-image generation [6, 5, 7], natural language generation [17], time series prediction [18], audio synthesis [19, 20], 3D point cloud generation [21], and molecular conformation generation [22].

Text-to-Image Generation Diffusion models play an important role in text-to-image generation. To improve computational efficiency, diffusion models are typically trained on low-resolution images [6] or latent variables [7, 23], which are then transformed into high-resolution images through super-resolution diffusion models [24] or latent-to-image decoders [25]. The sampling process of diffusion models utilizes classifier-free guidance [26] as well as various sampling algorithms that use deterministic [3, 27, 28, 29, 30] or stochastic [31, 32, 33] iterations. In addition, several works [34, 35] retrieve additional images related to the text prompt from an external database and use them to condition generation, thereby enhancing performance.

Multi-Object Generation Multi-object generation needs text-to-image models to comprehend different objects in the generation process. As the difficulty in the alignment between text and image, prior studies have utilized bounding boxes [9], masks [10], or small patches [11] of target objects to enhance alignment. Some studies aim to improve generation results without the need for human involvement in each generation, Liu et al. [36] proposed an approach where concept conjunctions are achieved by adding multiple estimated scores for different objects. And Feng et al. [8] utilize language parsers to associate attributes solely with the corresponding objects. Balaji et al. [37] combine T5 [38] and CLIP [39, 40] text encoders to improve the alignment in the text side.

Applications The diffusion models for text-to-image have a significant impact on downstream applications. These models can be directly applied to various inverse problems, such as super-resolution [41, 42], inpainting [43, 44], and JPEG restoration [45, 46]. Text-to-image diffusion models can also be used for other semantic image editing tasks. For instance, SDEdit [47] enables editing of an existing image via colored strokes or image patches. DreamBooth [48] and Textual Inversion [49] allow for personalized model implementation by learning a subject-specific token from a few images. Similar image-editing capabilities can also be achieved by fine-tuning the model parameters [50, 51] or automatically searching for editing masks using denoisers [52]. Several text-to-video diffusion models are developed on text-to-image ones and achieve high-quality video generation results [53, 54, 55, 56, 57]. Furthermore, the fitness capabilities of diffusion models have proven beneficial for the task of out-of-distribution detection [58].

Our method is constructed on a SOTA open-source text-to-image model: Stable diffusion, which comprises an autoencoder and a diffusion model. During the training process, the pre-trained autoencoder first compresses the image distribution into a latent space, and the diffusion model attempts to fit this new distribution in the latent space. In the sampling process, the diffusion model first generates latent results based on the text prompts and then uses the autoencoder to decode the latent results and obtain the final images.

Stable Diffusion utilizes a pre-trained CLIP model to encode the text prompt and incorporates multiple cross-attention blocks to integrate text information into the target regions of images. Previous work [59] has shown that these cross-attention blocks contain rich spatial structure information and control the spatial layouts of the generated results. This observation provides us with the possibility of using such spatial information to separate different objects and correcting the following cross-attention blocks to solve the mixing problems.

The multiple cross-attention blocks in Stable Diffusion exhibit remarkable alignment abilities in text-to-image generation. In a cross-attention block, the features $z_{t}$ of the noisy data are projected to a key $K=L_{K}(z_{t})$, and the textual embedding $c$ is projected to a query $Q=L_{Q}(c)$ and a value $V=L_{V}(c)$, via learned linear projections. The cross-attention map (CAM) is then $M=KQ^{T}$, and the final cross-attention output of this block is $\text{Softmax}(M/\sqrt{d})V$. Here, $d$ is the projection dimension of the key and query. Among multiple CAMs at different resolutions, CAMs in the middle have rich spatial structure information [59]. Spatial information can still be clearly observed even when only 20% of the generation process has been completed, as shown in Figure 7.

While such alignment is effective in many cases, it lacks global coordination, leading to disorderly competition. For instance, consider a text prompt “a leopard and a tiger” and the intermediate results correctly generate two objects. Ideally, the leopard prompt should align with one region, and the tiger prompt should align with another. However, in practice, the leopard and tiger prompts may both attempt to align with two regions simultaneously, resulting in the mixing of conflicting information and the generation of leopard and tiger hybrids, as shown in Figure 7. Moreover, designing global coordination solely with a pre-trained diffusion model is challenging due to its local alignment strategy, which results in limited global comprehension abilities for distinguishing between different objects. Thus, an additional model is necessary to identify objects from the local alignment information of CAMs, which we refer to as latent object detection.

Although we incorporate an additional model, our latent object detection model can be simple for several reasons. Firstly, local alignment has already marked important areas, making it easier to extract features. Secondly, the resolution of the middle CAMs is only $16\times 16$, resulting in a relatively small scale difference between objects. Thirdly, we typically avoid images where objects overlap each other, as our objective is to present the objects mentioned in the text as comprehensively as possible. These factors make the detection process relatively straightforward.

In this paper, we use a simple YOLO [60] model for latent object detection and train it on the COCO dataset. Our training procedure is as follows: we add Gaussian noise into the images and use the labels of the original images as the prompts. We then feed the noisy images and prompts into a pre-trained diffusion model. We capture the outcomes of middle CAMs, which we subsequently employ as input to the latent object detection model. The latent object detection model infers the bounding boxes and confidence scores for each object based on the corresponding CAMs. We then use the predicted bounding boxes and the ground truth bounding boxes to calculate the loss outcomes and update the latent object detection model correspondingly. More details and analyses about our latent detection model can be found in Appendix A.1.

In the sampling procedure, we first utilize a language parser [61] to find the objects in prompts. More details about language parsers can be found in Appendix A.2. Then we generate bounding boxes for each object based on the corresponding CAMs, with the class for each bounding box being the noun corresponding to the input CAM. The next step is to assign bounding boxes to different objects within a single image. To achieve this, we first employ non-maximum suppression to eliminate redundant bounding boxes. We do not discard all information from unused bounding boxes. Rather, we incorporate the confidence score and object class of unused bounding boxes into the corresponding choice bounding boxes. As a result, each choice bounding box may pertain to different objects with varying scores. We employ the linear sum assignment algorithm from the SciPy library to assign bounding boxes to different objects and optimize the total confidence score across all objects.

Despite the relatively limited number of categories in the COCO dataset compared to larger-scale datasets like Laion-5B [62], we find that our well-trained detection model demonstrates good generalization to previously unseen categories. The use of CAMs as input has played a significant role in this. Many previously unseen categories, which may exhibit substantial visual differences at the image level, are remarkably similar to certain known categories in terms of their CAMs as long as they share similar overall structures.

After obtaining the desired bounding boxes for each object, we use three consecutive steps to correct the CAMs and a strategy to maintain continuity in the generation process.

Boundary Correction Theoretically, we can use segmentation models instead of detection models to obtain more precise object boundaries. However, we find that this is often unnecessary, as the CAMs already contain sufficient local boundary information for objects. Therefore, we first use the detection model to generate bounding boxes $\text{BBox}[*]$ and then further refine the boundaries using CAMs. The segmentation for a certain object $n$ is:

Here, the shape of CAM is $H\times W\times N$, and $N$ is the total number of tokens in a prompt. We only do detection and boundary correction on the nouns that represent objects, and the shape of a segmentation of object $n$ is $H\times W$. The threshold $\sigma$ is computed using Otsu’s method [63]. For pixels belonging to more than one object, we assign them to the smallest object.

Conflict Elimination We find that the alignment results in CAMs become meaningful after $t\leq 800$.¹¹1Visual results for the meaningful CAMs be found in Section 4.3. The transition timestep is $I=800$. Then, we can address the issue of information mixing. In each cross-attention block, we utilize the conditional prompt $c$ and the unconditional prompt $uc$ to generate two queries $Q_{c}$, $Q_{uc}$ and two CAMs $KQ_{c}^{T}$, $KQ_{uc}^{T}$, respectively. Here, we denote $\text{CAM}_{0}=KQ_{c}^{T}$. For any region that has established its correspondence with the text describing a certain object, we eliminate the influence of other conflicting texts by replacing the values of the corresponding $KQ_{c}^{T}$ with those of $KQ_{uc}^{T}$ at the same position. Conflicting relationships can be obtained through human annotations or a language parser. Therefore, the corresponding mask is:

Here, the shape of the mask is identical to that of a CAM. Other tokens in a noun phrase share the same mask as the core noun. The Conflict Elimination algorithm can be expressed as follows:

Target Enhancement While the masking operator can prevent conflicting information from being mixed in the subsequent steps, it does not guarantee that the correct information can have sufficient influence in the target region to generate the right objects. One reason is that some mistakes may have already occurred in the previous steps, which can affect the alignment between pixels and target text. To address this issue, we propose Target Enhancement that strengthens the influence of the target text in such regions. We record the maximum value of CAMs for each pixel. If the maximum decrease after Conflict Elimination, it indicates that this pixel contains a large percentage of features belonging to other objects. In this case, we increase the value of the remaining CAMs at this pixel to enhance the injection of target information. The algorithm of Target Enhancement can be written as:

Smooth Involvement Another issue with masking operators is that they can cause a sharp change in the outputs when first applied. Such discontinuous outputs are not suitable for high-order numerical acceleration methods of diffusion models, such as PNDM [28], DPM-Solver [27], and DEIS [29]. To address this, we propose a smoothing approach that gradually increases the impact of Detector Guidance. Specifically, we begin by using 25% $\text{CAM}_{2}$ and 75% original CAM and gradually increase the ratio to 50%:50%, 75%:25%, and 100%:0%.

Upon the introduction of all the steps, the whole algorithm of our Detector Guidance can be found in Algorithm 1 and 5. A more comprehensive version can be found in Appendix A.3. Notably, our CAM correction has no hyperparameters for different situations, making it easy and robust to use. We also find the bounding boxes can be cached and reused in several successive steps without affecting alignment and quality, thereby improving the computational efficiency of our method.

We evaluate Detector Guidance on pre-trained Stable Diffusion models v1.4 and v2.1. The additional latent detection model is YOLOv1 with the conv2d stride of 1 to suit a small input size. The latent detection model is trained on the COCO dataset, utilizing 2 RTX3090*days, with nearly 70% of the time allocated to CAMs computation.

Here, we evaluate the alignment of our DG method on three benchmarks. First is the Concept Conjunction (CC) benchmark from [8], which comprises about 500 prompts following the structure of “a red car and a white sheep”. This is a suitable benchmark for Stable Diffusion v1.4 but too simplistic for Stable Diffusion v2.1. Consequently, we present a novel benchmark, the multi-related object (MRO) benchmark, which uses a similar sentence pattern. However, instead of using distinct objects such as a car and a sheep, we utilize GPT4 [64] to generate 30 prompts that contain two related objects, such as a tiger and a leopard. Additionally, we generate 10 samples for each prompt instead of just 1. This benchmark evaluates the ability of generation models to solve mixing problems both at the attribute level and at the object level. The complete MRO list can be found in Appendix A.4. Moreover, the COCO validation set is a commonly used benchmark for zero-shot text-to-image generation. We utilize it to evaluate the performance under more complex prompt patterns.

Regarding evaluation metrics, we find that FID and CLIP-score are not effective indicators of the mixing problem. Therefore, we primarily rely on human evaluation. Nevertheless, we also evaluate FID and CLIP-score to ensure that our method does not lead to any performance degradation on these two metrics. In Figure 5, we also present the visual generation results on MRO and COCO.

CC Structured diffusion guidance [8] is another guidance method that focuses on the text side to address the mixing problem. This work is built on Stable Diffusion v1.4 and provides the CC benchmark. Thus, we compare it with our method on the same benchmark and model. Moreover, since the two methods address the mixing problem on the text and image sides, respectively, they can be combined. In Table 4, we find that the combination of our method and the structured diffusion guidance method can further enhance the performance. One issue with structured diffusion guidance is that it may introduce unnecessary adjustments and is more likely to result in guidance results that are worse than the original ones, about 10% worse than ours.

MRO We use the more challenging MRO benchmark to evaluate the performance of our method on Stable Diffusion v2.1. The results are shown in Table 4. We can see that our method still achieves huge improvements on Stable Diffusion v2.1. As shown in the first row of Figure 5, Detector Guidance successfully solves the attribute mixing, object mixing, and object disappearance problems.

COCO In our experiments, we randomly select 5,000 captions from all COCO captions and generate 5,000 corresponding image pairs with and without Detector Guidance. To perform latent object detection on COCO, we use a language parser to identify all the noun phrases within the captions from COCO. We plot the trade-off curves between FID and CLIP-score using different guidance scales, including $[2.0,3.0,\cdots,10.0]$. Figure 4 shows that Detector Guidance improves a bit the results in both FID and CLIP-score when the guidance scale is larger than 3. To evaluate our method under complex prompt patterns, we conducted human evaluations on 500 pairs of images with the largest L2 distance from the 5,000 pairs. The results also show a clear advantage of our method.

Detection Based on our observations, we notice that independent objects can be discerned from CAMs when $t\leq 800$. This is in strong agreement with the outcomes we obtained using a latent object detection model that was well-trained on CAMs. In Figure 7, we find that the IOU results undergo significant changes when $t\geq 800$ and gradually converge to one when $t\leq 800$. Therefore, it can be deduced that for $t\geq 800$, the latent object detection model essentially makes random guesses regarding the object, while for $t\leq 800$, the model can effectively locate the object and subsequently refine its boundaries, ultimately converging to the final result.

Our detection model is only trained on the COCO dataset. Nonetheless, we observe that it exhibits good generalization to unseen categories. Notably, most objects in the demos of this paper do not belong to any category included in COCO. Our method can still detect objects and align objects with these prompts successfully.

Correction In Figure 7, we show an example of CAMs associated with the tokens “tiger” and “leopard” under different timesteps with or without DG. Here, the whole prompt is “A striped tiger and a spotted leopard”. In the original diffusion model, the CAMs attempt to align the tokens “tiger” and “leopard” with both objects simultaneously. Consequently, the information from “tiger” and “leopard” amalgamates in the right object, resulting in an animal that resembles both a tiger and a leopard. Upon incorporating DG into the diffusion model, DG can accurately distinguish between the two objects and align each one with its corresponding prompt, resulting in a final result that is highly consistent with the conditions. What’s more, DG adheres to the principle of minimal intervention, as it successfully preserves the image elements, such as rocks, vegetation, and the tiger, except for the region where the mixing error occurred. This reduces the possibility of introducing unknown new problems through additional corrections.

Boundary Correction Since bounding boxes of different objects may overlap with each other, some areas of one object may be mistakenly assigned to other objects. Boundary Correction can address this issue, as illustrated in Figure 9. In the second image, the boundary box of the tree overlaps with the boundary box of the giraffe, leading to poor generation results. Boundary Correction can refine

the boundary of the tree and remove the overlapping area. As a result, the hand of the giraffe can be generated correctly in the third image.

Target Enhancement While Conflict Elimination cannot guarantee proper alignment of the target prompt with the assigned area, we use Target Enhancement to enhance the target object, as shown in Figure 9. In the first image, both the left and right sides of the image bear a resemblance to a cat. Without Target Enhancement, even though we remove the cat prompt on the right and improve the left cat in the second image, the dog prompt cannot effectively align the right region. In the third image, Target Enhancement resolves this issue by enhancing the target object in CAMs.

Smooth Involvement Smooth Involvement is an important theoretical issue, but we did not observe significant improvement in practice. A similar phenomenon occurs in the acceleration of diffusion models, which use lower-order methods to start the generation and still yield good generation results.