MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis

Stable diffusion [38] is one of the most popular T2I models, and it uses the CLIP [36] text encoder to project texts into sequence embedding and integrate textual conditions into the generation process via Cross-Attention [46] layers.

Attention layers [46] play key roles in the interaction between various modal features. Omitting the reshape operation, the attention layer on 2D space can be expressed as:

$$\vspace{-0.2em}\mathbf{R}=\text{Softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V},\mathbf{R}\in{\mathbb{R}^{(H,W,C)}}$$

(1)

where $\mathbf{R}$ represents the output residual, and $\mathbf{Q},\mathbf{K},\mathbf{V}$ separately represents the Query, Key, and Value in attention layers, which are projected by linear layers.

Problem Definition. In the Multi-Instance Generation (MIG), users will give generation models the global prompt $\mathcal{P}$, instance layout bounding boxes $\mathbb{B}=\{\mathbf{b}^{1},...,\mathbf{b}^{N}\}$, where $\mathbf{b}^{i}=[x_{1}^{i},y_{1}^{i},x_{2}^{i},y_{2}^{i}]$, and corresponding descriptions $\mathbb{D}=\{\mathbf{d}^{1},...,\mathbf{d}^{N}\}$. According to user-provided inputs, the model needs to generate an image $\mathcal{I}$, in which the instance within the box $\mathbf{b}^{i}$ should adhere to the instance description $\mathbf{d}^{i}$, and global alignment is ensured in all instances.

Difficulties in MIG. When dealing with Multi-Instance prompts, stable diffusion struggles with attribute leakage, i.e., 1) Textual Leakage. Due to the causal attention masks used in the CLIP encoder, the latter instance tokens may exhibit semantic confusion [14]. 2) Spatial Leakage. The Cross-Attention lacks precise position control [6], and instances will affect the generation of each others’ region.

Motivation. Divide and conquer is an ancient but wise idea. It first divides a complex task into several simpler subtasks, then conquers these subtasks respectively, and finally obtains the solution to the original task by combining the solutions of the subtasks. This idea is highly applicable to MIG. For example, MIG is a complex task for most T2I models, while Single-Instace Generation is a simpler subtask that T2I models can solve well [39, 8, 51, 32, 45]. Based on this idea, we proposed our MIGC, which extends the stable diffusion with stronger MIG ability. We will introduce the technology details by telling “how to divide,” “how to conquer,” and “how to combine.”

Instance shading subtasks in Cross-Attention space. Cross-Attention is the only way for text and image features to interact in stable diffusion, and the output determines the generated content, which looks like a shading operation on image features. In this view, the MIG task can be defined as performing correct Multi-Instance shading on image features, and $subtask_{i}$ can be defined as finding a single instance shading result $\mathbf{R}^{i}$ to satisfy the following:

$$\mathbf{R}^{i}=\arg\min_{\mathbf{R}^{i}}(\left\|\mathbf{R}^{i}-\mathbf{R}^{correct}\right\|_{2}\cdot\textbf{M}^{i}),\vspace{-3mm}$$

(2)

where $\mathbf{R}^{correct}$ represents an objectively existing correct feature, and $\mathbf{M}^{i}$ is an instance mask generated according to the box $\mathbf{b}^{i}$, with the values inside the box region are set to 1, and the rest of the positions are set to 0. That is to say, each shading instance should have the correct textual semantic in its corresponding area.

Two benefits of division in the Cross-Attention space. i.e., 1) Conquer more efficiently: For N-instance generation, MIGC conquers N subtasks solely on Cross-Attention layers instead of the entire Unet network, which will be more efficient; 2) Combine more harmoniously: Combining subtasks in the middle layer enhances the overall cohesiveness of the generated image compared to combining at the final output of the network.

Shading stage 1: shading results of Cross-Attention. The pre-trained Cross-Attention will notice regions with high attention weight and perform shading according to the textual semantics [48, 9]. As shown in Fig. 1, MIGC uses the masked Cross-Attention output as the first shading results:

$$\vspace{-0.2em}\mathbf{R}^{i}_{f}=\text{Softmax}(\frac{\mathbf{Q}\mathbf{K^{i}}^{T}}{\sqrt{d}})\mathbf{V^{i}}\cdot\textbf{M}^{i},$$

(3)

where $\mathbf{K}^{i}$ and $\mathbf{V}^{i}$ are obtained from text embedding of $\mathbf{d}^{i}$, and $\mathbf{Q}$ is obtained from the image feature map.

Two issues of Cross-Attention shading results. 1) Instance Merge. According to Eq. (3), for two instances with the same description, they will get the same $\mathbf{K}$ and $\mathbf{V}$ in the Cross-Attention layer. If their boxes are close or even overlap, the network will easily merge the two instances; 2) Instance Missing. The initial edit method [31] shows that the initial noise of SD largely determines the layout of the generated image, i.e., specific regions prefer to generate specific instances or nothing. If the initial noise does not tend to generate an instance according to description $\mathbf{d}^{i}$ in box $\mathbf{b}^{i}$, the $\mathbf{R}_{f}^{i}$ will be weak, leading to the instance missing.

Grounded phrase token for solving instance merge. To identify instances with the same description but different boxes, MIGC extends the text tokens of each instance to a combination of text and position tokens. As shown in Fig. 2(a), MIGC first projects the bounding box information to the Fourier embedding, then uses a MLP layer to get position tokens. MIGC concatenates the text tokens with position tokens to obtain the grounded phrase tokens:

$$\vspace{-0.2em}\mathbf{G}^{i}=[\text{CLIP}(\mathbf{d}^{i}),\text{MLP}(\text{Fourier}(\mathbf{b}^{i}))],$$

(4)

where $[\cdot]$ represents the concatenation.

Shading stage 2: Enhancement Attention for solving instance missing. Illustrated in Fig. 1, MIGC uses a trainable Enhancement-Attention (EA) Layer to enhance the shading result. Specifically, as shown in Fig. 2(a), after obtaining the grounded phrase token, EA uses a new trainable Cross-Attention layer to obtain an enhanced shading result and adds it to the first shading result $\mathbf{R}^{i}_{f}$:

Global prompt residual as shading background. Obtaining N-instance shading results as shading foreground, the next step of MIGC is to get the shading background. Illustrated in Fig. 1(c), MIGC utilizes global prompt $\mathcal{P}$ to obtain the shading background result $\mathbf{R}^{bg}$ in a manner similar to Eq.(3), with the background mask $\mathbf{M}^{bg}$, in which positions containing the instance are assigned a value of 0, while all other positions are marked as 1.

Layout Attention residuals as shading template. A certain gap exists between shading instances $\{\mathbf{R}_{s}^{1},\ldots,\mathbf{R}_{s}^{N}\}$ and the shading background $\mathbf{R}^{bg}$, as their shading process is relatively independent. To bridge these shading results and minimize the gap, MIGC needs to learn a shading template according to the image feature maps’ information. As shown in Fig. 1(c), a Layout Attention layer is used in MIGC to achieve the above goal. Illustrated in Fig. 2(b), Layout Attention performs similarly to the Self-Attention [42, 40] while instance masks $\mathbb{M}_{inst}=\{\mathbf{M}^{bg},\mathbf{M}^{1},\ldots,\mathbf{M}^{N}\}$ are used to construct attention masks:

$$\vspace{-0.8em}\mathbf{A}_{(a,b),(c,d)}=\begin{cases}1,\text{if}\,\exists\ \mathbf{m}\in\mathbb{M}_{inst},\mathbf{m}_{a,b}=\mathbf{m}_{c,d}=1\\ -inf,\ \ \ \ \ \ \text{otherwise}\end{cases}$$

(6)

Shading Aggregation Controller for the final fusion. To summarize, in all the above operations, MIGC can get $\mathbb{R}_{s}=\{\mathbf{R}_{s}^{1},\ldots,\mathbf{R}^{N}_{s},\mathbf{R}^{bg},\mathbf{R}_{LA}\}\in\mathbb{R}^{(N+2,C,H,W)}$ and $\mathbb{M}=\{\mathbf{M}^{1},\ldots,\mathbf{M}^{N},\mathbf{M}^{bg},\mathbf{M}_{LA}\}\in\mathbb{R}^{(N+2,1,H,W)}$, where $\mathbf{M}_{LA}$ is the all-1 guidance mask corresponding to $\mathbf{R}_{LA}$. In order to dynamically aggregate shading results at different timesteps of the generation process, we propose the Shading Aggregation Controller (SAC). As shown in Fig.2(c), SAC sequentially performs instance intra-attention and inter-attention, and aggregation weights summing to 1 are assigned to shading results on each spacial pixel through the softmax function, resulting in the final shading.

$$\mathbf{R}_{final}=SAC(\mathbb{R}_{s},\mathbb{M}),\mathbf{R}_{final}\in\mathbb{R}^{H,W,C}$$

(8)

For more details, please refer to supplementary materials.

Training Loss. We use the original denoising loss [38, 20]:

Implementation Details. We only deploy MIGC on the mid-layers (i.e., $8\times{8}$) and the lowest-resolution decoder layers (i.e., $16\times{16}$) of UNet, which greatly determine the generated image’s layout and semantic information [32, 6]. In other Cross-Attention layers, we use the global prompt for global shading. We use COCO 2014 [28] to train MIGC. To get the instance descriptions and their bounding boxes, we use stanza [35] to split the global prompt and detect the instances with the Grounding-DINO[30] model. We train our MIGC based on the pre-trained stable diffusion v1.4. We use AdamW [25] optimizer with a constant learning rate of $1e^{-4}$, and train the model for 300 epochs with batch size 320, which requires 15 hours on 40 V100 GPUs with 16GB VRAM each. For inference, we use EulerDiscreteScheduler [23] with 50 sample steps and use our MIGC in the first 25 steps. We select the CFG scale[19] as 7.5. For more details, please refer to supplementary materials.

In COCO-MIG, we pay attention to position, color, and quantity. To construct it, we randomly sampled 800 COCO images and assigned a color to each instance while keeping the original layout. Furthermore, We reconstruct the global prompts in the format of ’ a $<$attr1$>$ $<$obj1$>$ and a $<$attr2$>$ $<$obj2$>$ and a … ’, and we divide this benchmark into five levels based on the number of instances in the generated image. Each method will generate 6400 images.

In COCO-Position, we sampled 800 images, using the captions as the global prompts, labels as instance descriptions, and bounding boxes as layouts to generate 6400 images.

Drawbench is a challenging T2I benchmark. We use GPT4 [34, 15] to extract all instance descriptions and generate the layouts for each prompt. We use a total of 64 prompts, of which 25 are related to color, 19 are related to counting, and 20 are related to position, ultimately generating 512 images.

Position Evaluation. We use Grounding-DINO [30] to detect each instance and calculate the maximum IoU between the detection boxes and the Ground Truth box. If the above IoU is higher than the threshold $t$=0.5, we mark it as Position Correctly Generated.

Attribute Evaluation. For a Position Correctly Generated instance, we use the Grounded-SAM model [30, 26] to segment it and calculate the percentage of the target color in the HSV color space. If the above percentage exceeds the threshold $S$=0.2, we denote it as Fully Correctly Generated.

Metrics on COCO-MIG. We primarily measure the Instance Success Rate and mIoU. The Instance Success Rate calculates the probability that each instance is Fully Correctly Generated, and mIoU calculates the mean of the maximum IoU for all instances. Note that if the color attribute is incorrect, we set the IoU value as 0.

Metrics on COCO-Position. We use Success Rate, mIoU and Grounding-DINO AP score to measure the Spatial Accuracy. The Success Rate represents whether all instances in one image are Position Correctly Generated. Besides, we use the Fréchet Inception Distance (FID) [18] to evaluate Image Quality. To measure Image-Text Consistency, we use CLIP score and Local CLIP score[1].

Metrics on DrawBench. We evaluate the Success Rate for images related to the position and count by checking whether all instances in each image are Position Correctly Generated. For color-related images, we check whether all instances are Fully Correctly Generated. In addition to automated evaluations, a manual evaluation is conducted.

COCO-MIG. Tab.1 shows results in COCO-MIG. MIGC improves the Instance Success Rate from 32.39% to 58.43% and mIoU from 32.25 to 51.48. Improvements are consistently observed across all count-division levels, underscoring the robust control capabilities of MIGC on position, quantity, and attributes. Furthermore, MIGC runs at almost the same speed as the original stable diffusion, thanks to MIGC dividing MIG in the Cross-Attention Space, accelerating the conquering and combing of subtasks.

COCO-Position. Tab.2 shows quantitative results in COCO-Position, indicating that MIGC brings significant improvement in Spatial Accuracy: increased the Success Rate from 70.52% to 80.29%, mIoU from 71.61 to 77.38, and AP score from 40.68/68.26/42.85 to 54.69/84.17/61.71. MIGC also achieves similar FID scores compared to the stable diffusion, highlighting that MIGC can enhance position control capabilities without destroying image quality.

DrawBench. Tab.3 shows the results in drawbench. MIGC achieves the best performance in both mechanical metrics and human evaluation. Human evaluation doesn’t rely on IoU to determine position correctness.

Shading Aggregation Controller. From Tab.4, we find that using SAC improves the performance metrics(compare ⑤ with ⑥ and ① with ②), which is also reflected in the ablation experiments on COCO-MIG in Fig.6(a).

Enhancement Attention Layer. In Tab.4, the EA Layer significantly improves the Success Rate from 12.10% to 80.16%, mIoU from 29.55 to 76.63, and AP from 1.89 / 7.64 / 0.49 to 53.03 / 84.05 / 58.67 (Compare ② with ④). We also observe significant improvement in Fig.6(a).

Layout Attention Layer. The results of ④ and ⑥ in Tab.4 show that LA Layer can improve the AP. We find that SAC+LA, compared to SAC alone, has improved mIoU to some extent in Fig.6(a).

Inhibition Loss. We also conducted the ablation study on the Inhibition loss, with 0.1 and 1.0 loss weight. We show the results in Tab.5 and Fig.6(b). Tab.5 indicates that inhibition loss can significantly improve the AP metric in COCO-Position. We find that setting the loss function weight to 1.0 can further improve the AP metric, but it comes at the cost of a slight decrease in image quality (i.e., FID). So we finally choose loss weight as 0.1. Fig.6 (b) shows the comparison between w/ loss 0.1 and w/o loss on COCO-MIG, and we observe that the inhibition loss can improve the mIoU, especially when generating images with large instance quantity.

Qualitative Results. We show qualitative results in Fig.7. The first column indicates that the EA Layer can effectively alleviate instance missing. The second column illustrates that the LA Layer can significantly improve generated image quality. The third column suggests that the SAC also aids in better aggregation of shading instances. The fourth column demonstrates that inhibition loss enhances the model’s control capabilities. The fifth column demonstrates that position tokens effectively alleviate instance merging.