A Diffusion-based Method for Multi-turn Compositional Image Generation

Given an initial reference image $a^{(1)}$, which is a background canvas, and a sequence of $k$ modification texts $\{m^{(1)},\ldots,m^{(k)}\}$, which describe the desired changes to be successively made to $a^{(1)}$, M-CIG is a $k$-turn iterative process, where at each turn $i\in\{1,\ldots,k\}$, it is required to generate a target image $z^{(i)}$ by changing the current reference image $a^{(i)}$ according to $m^{(i)}$, and if $i<k$, $z^{(i)}$ will be used as the next reference image $a^{(i+1)}$.

Considering the cross-modal nature of M-CIG, we map images and texts into a joint representation space through encoding. As shown in Figure 2, we include an image encoder and a text encoder in CDD-ICM, where the former is used to encode reference images and target images, and the latter is used to encode modification texts. Since M-CIG is a vision-and-language (V&L) task, we take advantage of large-scale V&L pre-training by using CLIP, which is pre-trained on 400M image-text pairs, as the backbone of both encoders. Specifically, we use the vision part of CLIP, which is a vision transformer (ViT) [11], as the backbone of the image encoder, and use the language part of CLIP, which is a GPT-like [46] auto-regressive language model, as the backbone of the text encoder. In each encoder, we append a linear projection layer after the backbone, which finally yields the representations. For both linear projection layers, we set their output dimensionality to the same value $d$, which is a hyper-parameter denoting the dimensionality of the joint representation space.

Additionally, as shown in Figure 2, we also include a noisy image encoder in CDD-ICM, which is used to encode the noisy target images obtained in the denoising diffusion mechanism. It has the same structure as the image encoder, but holds different trainable parameters.

At each turn of M-CIG, to extract clues for the generation of the target image, we perform compositional fusion between the reference image and the modification text. As shown in Figure 2, we include a fusion module in CDD-ICM, which fuses the representation of the reference image with that of the modification text. Actually, we can interpret each turn of M-CIG from the perspective of QA. Specifically, we can regard the reference image as a context, the modification text as a relevant question, and the target image as the corresponding answer. In this way, to implement the fusion module, we borrow the following gated fusion mechanism from a QA method [67]:

$$\small\begin{split}&\mathrm{f}(u,v)=g\odot h+(1-g)\odot u\\ &g=\mathrm{sigmoid}(W_{g}[u;v;u\odot v;u-v]+b_{g})\\ &h=\mathrm{gelu}(W_{h}[u;v;u\odot v;u-v]+b_{h})\end{split}$$

(1)

where $W_{g}$ and $W_{h}$ are trainable weight matrices, $b_{g}$ and $b_{h}$ are trainable bias vectors, $\odot$ denotes element-wise multiplication, and $[;]$ denotes vector concatenation. In the fusion module, we set $u$ to the representation of the reference image, set $v$ to that of the modification text, and thereby obtain $\mathrm{f}(u,v)$ as the compositional fusion result.

To generate the target image at each turn of M-CIG, we learn conditional denoising diffusion (CDD), which is to perform the generation using a denoising diffusion mechanism conditioned on the compositional fusion result between the reference image and the modification text. As proposed by [16], the denoising diffusion mechanism consists of a forward diffusion process, which gradually injects noises to the target image, and a reverse denoising process, which gradually erases the injected noises.

With the target image denoted as $x_{0}\sim q(x_{0})$, the forward diffusion process is a pre-defined Markov chain of $T$ time steps $\langle 1,\ldots,T\rangle$, where the state transfers from $x_{0}$ all the way to $x_{T}$. Specifically, at each time step $t$, we obtain $x_{t}$ by injecting a Gaussian noise $\epsilon_{t}\sim N(0,\mathit{I})$ to $x_{t-1}$:

$$\small\begin{split}&q(x_{t}|x_{t-1})=N(\sqrt{\alpha_{t}}x_{t-1},(1-\alpha_{t})\mathit{I})\\ \Leftrightarrow\quad&x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}}\epsilon_{t}\end{split}$$

(2)

where $\alpha_{t}$ is a hyper-parameter used to control the noise scale. It is easy to derive that with $\prod_{i=1}^{t}\alpha_{i}$ denoted as $\bar{\alpha}_{t}$, we can actually obtain $x_{t}$ by directly injecting $\epsilon_{t}$ to $x_{0}$:

$$\small\begin{split}&q(x_{t}|x_{0})=N(\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathit{I})\\ \Leftrightarrow\quad&x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{t}\end{split}$$

(3)

As shown in Figure 2, we include a noise injector in CDD-ICM, which executes Equation 3 at an arbitrary time step $t$ to obtain $x_{t}$ as a noisy target image. To implement the noise injector, we set each $\alpha_{t}\in\{\alpha_{1},\ldots,\alpha_{T}\}$ in the following way proposed by [40]:

$$\small\begin{split}&\alpha_{t}=\frac{s(t)}{s(t-1)}\\ &s(t)=\mathrm{cos}^{2}(\frac{t/T+0.008}{1.008}\cdot\frac{\pi}{2})\end{split}$$

(4)

With the compositional fusion result between the reference image and the modification text denoted as $c$, the reverse denoising process is a parameterized Markov chain of $T$ time steps $\langle T,\ldots,1\rangle$, where the state transfers from $x_{T}$ all the way back to $x_{0}$ conditioned on $c$. Specifically, at each time step $t$, given $x_{t}$ and the condition $c$, we obtain $x_{t-1}$ using a neural network $\theta$:

$$\small\begin{split}&p_{\theta}(x_{t-1}|x_{t},c)=N\big{(}\mu_{\theta}(x_{t},t,c),\Sigma_{\theta}(x_{t},t,c)\big{)}\\ \Leftrightarrow\quad&x_{t-1}=\mu_{\theta}(x_{t},t,c)+\Sigma_{\theta}^{\frac{1}{2}}(x_{t},t,c)\xi_{t}\end{split}$$

(5)

where $\xi_{t}\sim N(0,\mathit{I})$. According to [16], we can learn $\theta$ by minimizing the following variational lower-bound (VLB) loss:

$$\small\begin{split}&L^{vlb}=E_{x_{0}\sim q(x_{0}),t\sim U\{1,T\}}[L^{vlb}_{t}]\\ &L^{vlb}_{t}=\begin{cases}-\mathrm{log}\ p_{\theta}(x_{t-1}|x_{t},c),&\text{if}\ t=1\\ D_{KL}\big{(}q(x_{t-1}|x_{t},x_{0})||p_{\theta}(x_{t-1}|x_{t},c)\big{)},&\text{else}\end{cases}\end{split}$$

(6)

Using Bayes’ theorem, it can be derived that with $1-\alpha_{t}$ denoted as $\beta_{t}$, $q(x_{t-1}|x_{t},x_{0})$ is the following Gaussian distribution:

$$\small\begin{split}&q(x_{t-1}|x_{t},x_{0})=N\big{(}\tilde{\mu}(x_{t},x_{0}),\tilde{\beta}_{t}\mathit{I}\big{)}\\ &\tilde{\mu}(x_{t},x_{0})=\frac{\sqrt{\bar{\alpha}_{t-1}\beta_{t}}}{1-\bar{\alpha}_{t}}x_{0}+\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}x_{t}\\ &\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}\end{split}$$

(7)

Based on Equation 3 and Equation 7, we parameterize $\mu_{\theta}(x_{t},t,c)$ in the following way proposed by [16]:

$$\small\mu_{\theta}(x_{t},t,c)=\frac{1}{\sqrt{\alpha_{t}}}\big{(}x_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t,c)\big{)}$$

(8)

where $\epsilon_{\theta}(x_{t},t,c)$ is a prediction to $\epsilon_{t}$. Besides, we also parameterize $\Sigma_{\theta}(x_{t},t,c)$ in the following way proposed by [40]:

$$\small\Sigma_{\theta}(x_{t},t,c)=\mathrm{exp}\big{(}\mathrm{log}\frac{\beta_{t}}{\tilde{\beta}_{t}}\rho_{\theta}(x_{t},t,c)+\mathrm{log}\tilde{\beta}_{t}\big{)}$$

(9)

where $\rho_{\theta}(x_{t},t,c)$ is a fraction used to interpolate between $\mathrm{log}\beta_{t}$ and $\mathrm{log}\tilde{\beta}_{t}$. As shown in Figure 2, we include a denoising U-Net in CDD-ICM, which performs the above parameterizations and thus can be seen as $\theta$. To implement the denoising U-Net, we make three changes to the U-Net structure used by [8]. First, we replace the class embedding with the condition $c$. Second, we concatenate $x_{t}$ with the reference image along the channel dimension, and thereby use the result as the input. Third, we concatenate the patch representations of the reference image with the token representations of the modification text, and thereby use the result to augment each attention layer as suggested by [39]. The output of the denoising U-Net is divided into two parts along the channel dimension, which are separately used as $\epsilon_{\theta}(x_{t},t,c)$ and $\rho_{\theta}(x_{t},t,c)$.

According to [16], to learn the above reverse denoising process, instead of minimizing $L^{vlb}$, we can actually minimize the following mean squared error (MSE) loss:

$$\small L^{mse}=E_{x_{0}\sim q(x_{0}),t\sim U\{1,T\}}[||\epsilon_{t}-\epsilon_{\theta}(x_{t},t,c)||^{2}]$$

(10)

Compared with $L^{vlb}$, $L^{mse}$ is not only simpler but also more effective. However, minimizing $L^{mse}$ cannot bring any learning signal to $\Sigma_{\theta}(x_{t},t,c)$. To benefit from learning $\Sigma_{\theta}(x_{t},t,c)$, we combine $L^{mse}$ with $L^{vlb}$ as suggested by [40]. Specifically, we minimize a CDD loss $L^{cdd}$, which is calculated as follows:

$$\small L^{cdd}=L^{mse}+\gamma L^{vlb}$$

(11)

where $\gamma$ is a hyper-parameter used to control the weight of $L^{vlb}$. Additionally, we also stop the gradients of $L^{vlb}$ from flowing to $\epsilon_{\theta}(x_{t},t,c)$.

At each turn of M-CIG, the semantic quality of the generated target image depends on the condition of the denoising diffusion mechanism, which is the compositional fusion result between the reference image and the modification text. Although learning CDD ensures that the condition is learned, this effect is implicit. To explicitly enhance the condition so that it embodies more clues about the target image, we learn image compositional matching (ICM) as an auxiliary objective of CDD, which is to align the compositional fusion result with the representation of the target image.

To learn ICM, we adopt the InfoNCE loss [41] used in the contrastive pre-training of CLIP, and apply it to individual turns constituting M-CIG samples. Specifically, given a mini-batch of $n$ (reference image, modification text, target image) triples $\{(a_{1},m_{1},z_{1}),\ldots,(a_{n},m_{n},z_{n})\}$, each of which denotes an individual turn picked from an M-CIG sample, we treat them as positive samples, and generate $n^{2}-n$ negative samples by replacing the target image $z_{i}$ in each positive sample $(a_{i},m_{i},z_{i})$ separately with the other $n-1$ target images $\{z_{1},\ldots,z_{n}\}-\{z_{i}\}$. For each of the positive and negative samples, suppose that we have already obtained the compositional fusion result between the reference image and the modification text and the representation of the target image, then we calculate the cosine similarity between them. As a result, we construct a similarity matrix $S\in\mathbb{R}^{n\times n}$, where the element at the $i$-th row and the $j$-th column corresponds to the sample $(a_{i},m_{i},z_{j})$. It is easy to see that the diagonal elements in $S$ correspond to the positive samples, while the other elements correspond to the negative samples. Based on $S$, we minimize an ICM loss $L^{icm}$, which is calculated as follows:

$$\small\begin{split}L^{icm}=&\frac{1}{n}\mathrm{tr}\Big{(}\mathrm{-log}\big{(}\mathrm{softmax}(\frac{S}{\tau})\big{)}\Big{)}+\\ &\frac{1}{n}\mathrm{tr}\Big{(}\mathrm{-log}\big{(}\mathrm{softmax}(\frac{S^{\top}}{\tau})\big{)}\Big{)}\end{split}$$

(12)

where $\tau$ is a trainable temperature scalar, $\mathrm{tr}(\cdot)$ denotes calculating matrix trace, and $\mathrm{softmax}(\cdot)$ is calculated along the row dimension. In this way, the compositional fusion result between a (reference image, modification text) pair will be close to the representation of the real target image, while apart from those of the fake ones.

Besides, to enable ICM guidance, which will be introduced later, we also learn noise-aware image compositional matching (N-ICM). Specifically, we replace the above target images with their noisy variants, which are obtained using the noise injector, and encode these noisy target images using the noisy image encoder. On this basis, we minimize an N-ICM loss $L^{n-icm}$, which is calculated in the same way as we calculate $L^{icm}$.

For the training, we disassemble M-CIG samples into individual turns and thereby apply teacher forcing. On this basis, we divide the training into three stages, where in each stage, we minimize a different loss through mini-batch gradient descent to update the corresponding trainable components of CDD-ICM. Specifically, in the first stage, we minimize $L^{icm}$ to update the image encoder, the text encoder, and the fusion module. In the second stage, we minimize the following joint loss, which is a combination of $L^{cdd}$ and $L^{icm}$, to update the image encoder, the text encoder, the fusion module, and the denoising U-Net:

$$\small L^{joint}=L^{cdd}+\delta L^{icm}$$

(13)

where $\delta$ is a hyper-parameter used to control the weight of $L^{icm}$. In the third stage, we freeze the image encoder, the text encoder, and the fusion module, and minimize $L^{n-icm}$ to update the noisy image encoder. To effectively fine-tune CLIP, we set the backbone learning rate as a product of the global learning rate and a backbone activity ratio $\eta$, which is a hyper-parameter. From the perspective of transfer learning, $\eta$ controls the trade-off between the knowledge transferred from CLIP and that embodied in the training data.

For the inference, at each turn of M-CIG, we use the image encoder to encode the reference image, use the text encoder to encode the modification text, use the fusion module to perform compositional fusion based on the encoding results, and use the denoising U-Net to iteratively execute Equation 5 from the time step $T$ until the time step $1$, where the condition $c$ is set to the compositional fusion result. From Equation 3 and Equation 4, it can be derived that if $T$ is large enough, then $q(x_{T})\approx N(0,I)$, thus we sample $x_{T}$ from $N(0,I)$ at the time step $T$. To accelerate the inference, we traverse only a part of the time steps, which are uniformly distributed among all of them, and make this process deterministic as suggested by [60]. Finally, we obtain $x_{0}$ as the generated target image at the time step $1$.

Moreover, we also perform ICM guidance and classifier-free guidance to boost performance. Our ICM guidance is similar to the CLIP guidance of [39]. Specifically, suppose that we have minimized $L^{n-icm}$ in the training, then in the inference, instead of using $\mu_{\theta}(x_{t},t,c)$ in Equation 5, we use $\hat{\mu}_{\theta}(x_{t},t,c)$, which is obtained by perturbing $\mu_{\theta}(x_{t},t,c)$ using the gradient of $L^{n-icm}$ with respect to $x_{t}$:

$$\small\hat{\mu}_{\theta}(x_{t},t,c)=\mu_{\theta}(x_{t},t,c)+\psi\Sigma_{\theta}(x_{t},t,c)\nabla_{x_{t}}L^{n-icm}$$

(14)

where $\psi$ is a hyper-parameter used to control the perturbation scale. Our classifier-free guidance is similar to that of [39]. Specifically, in the training, when calculating $\epsilon_{\theta}(x_{t},t,c)$ in Equation 10, we set the condition $c$ to $\vec{0}$ with a probability of $\lambda$, which is a hyper-parameter. On this basis, in the inference, instead of using $\epsilon_{\theta}(x_{t},t,c)$ in Equation 8, we use $\hat{\epsilon}_{\theta}(x_{t},t,c)$, which is obtained by perturbing $\epsilon_{\theta}(x_{t},t,c)$ using $\epsilon_{\theta}(x_{t},t,\vec{0})$:

$$\small\hat{\epsilon}_{\theta}(x_{t},t,c)=\phi\epsilon_{\theta}(x_{t},t,c)+(1-\phi)\epsilon_{\theta}(x_{t},t,\vec{0})$$

(15)

where $\phi$ is a hyper-parameter used to control the perturbation scale.

The goal of image manipulation is to modify specific attributes of an image while avoiding unintended changes or generating a completely new image. Existing works can be split into two main categories: image-to-image translation and text-conditioned image manipulation.

Image-to-Image Translation. The image-to-image translation aims to generate an output image only conditioning on an input image, i.e., uni-modal condition. Image inpainting and image super-resolution are two typical image-to-image translation tasks. In recent years, deep learning has achieved great success in image inpainting. Context Encoders [44] first explores to utilize conditional GANs. Multiple variants [69, 72, 73, 29] of U-Net [50] have been proposed for image inpainting. Some works explore multi-stage generation by taking object edges [38], structures [48], or semantic segmentation maps [64] as intermediate clues. In terms of super-resolution, most early works are regression-based and trained with MSE loss [9, 10, 70, 23]. Auto-regressive models [7, 42] and GAN-based methods [19, 26, 68, 35, 32] have also shown high quality results.

Text-Conditioned Image Manipulation. The text-conditioned image manipulation targets generating an output image conditioned on both the input image and text, i.e., multi-modal condition. The input text can be a caption-like description of the target image, and the editing is usually single-turn. TAGAN [37] employs word-level local discriminators to preserve text-irrelevant content. ManiGAN [28] first selects image regions and then correlates the regions with semantic words. DiffusionCLIP [22] is a robust framework that utilizes the pre-trained diffusion models and CLIP loss for image manipulation.

The input text can also be user-provided text instructions that describe desired modifications, such as adding, changing, or removing the objects in images. Generating an image based on provided instructions and an input image is dubbed as the compositional image generation (CIG) task in this paper. [1] achieve great performance on the benchmarks CSS [65] and Fashion Synthesis [75] by designing an improved image & text composition layer and a multi-modal similarity module. [74] propose a GAN-based method to locally modify image features and show remarkable results on both CSS and Abstract Scene [76]. Afterward, the M-CIG task presents a more challenging setting compared to the above single-turn CIG task. [12] first propose the M-CIG task known as Generative Neural Visual Artist (GeNeVA) task, which requires iteratively generating an image according to ongoing linguistic input. [14] introduce the self-supervised counterfactual reasoning (SSCR) framework to tackle the data scarcity problem. LatteGAN [34] improves desired object generation by introducing a Latte module and a text-conditioned U-Net discriminator. Our research work targets the M-CIG task, following [12], we conducted experiments on CoDraw [25] and i-CLEVR [12].

Diffusion models (DMs) [58], which formulate the data sampling process as an iterative denoising procedure, are closely related to a large family of methods for learning generative models as transition operators of Markov chains [3, 58, 56, 61, 27]. Many research works concentrate on improving the diffusion process of DMs. [62] propose to estimate the gradients of data distribution via score matching and produce samples via Langevin dynamics. Denoising diffusion probabilistic models (DDPMs) [16], which optimize a variational lower bound to the log-likelihood, can achieve comparable sample quality as GANs [5, 20]. Denoising diffusion implicit models (DDIMs) [59] speed up the sampling process while enabling near-perfect inversion [8]. The improved DDPM [40] introduces several modifications to achieve competitive likelihoods without sacrificing sample quality. The latent diffusion [49] model is applied in latent space instead of pixel space to enable an efficient diffusion process. Although GANs have achieved plausible results in image synthesis, they are usually difficult to train and tend to limit the diversity of the generated images [57, 66]. DMs are more stable during training and demonstrate comparable or even better performance for image synthesis [8, 63, 17].

Motivated by the progress in developing DMs, some research works explore text or image conditional diffusion mechanisms. While certain diffusion models solely utilize an input image for conditioning, such as PALETTE [51] and SR3 [53], those that condition on both an input image and text are more pertinent to our work. GLIDE [39] is a text-guided diffusion model, where classifier-free guidance yields higher-quality images than CLIP guidance. DALL-E 2 [47] initially generates a CLIP [45] image embedding given a text caption and then generates an image conditioned on the image embedding. ImageGen [52] exhibits a deep level of language understanding which enables high-fidelity image generation. Stable Diffusion [49] is an efficient latent diffusion model and has achieved superior image synthesis performance. [30] compose pre-trained text-guided diffusion models to improve structured generalization for image generation. However, in the context of this paper, it should be noted that the textual input in the aforementioned models refers to a caption-like description, rather than iterative instructions on image manipulations. Designing diffusion-based methods for the M-CIG task remains a relatively underexplored area, presenting challenges in iteratively modifying images with instructions and conditioning the denoising diffusion mechanism on multi-modalities. To address this gap in the literature, we propose a diffusion-based approach coupled with auxiliary ICM objectives to enhance the visual and semantic fidelity of generated images in the M-CIG task.

To verify the effectiveness of CDD-ICM, we conduct experiments on the following two M-CIG benchmark datasets:

CoDraw. CoDraw contains $8$K M-CIG samples for training, $1$K for validation, and $1$K for test. The number of turns per M-CIG sample varies between $1$ and $14$ with an average of $4.25$. The reference images and the target images contain $58$ classes of clip-art-style objects, such as boys, girls, and trees. The modification texts are conversations between a teller and a drawer.

i-CLEVER. i-CLEVER contains $6$K M-CIG samples for training, $2$K for validation, and $2$K for test. The number of turns per M-CIG sample is always $5$. The reference images and the target images contain $24$ classes of colored geometric objects, such as red spheres, yellow cubes, and blue cylinders. The modification texts are sentences indicating the addition of a new object.

Both datasets adopt four metrics to evaluate the semantic quality of the generated target images: precision, recall, F1 score, and relational similarity (RSIM). Specifically, each dataset comes with an object detector, which is trained on the dataset by [12]. In the evaluation, the object detector is applied to both the generated target images and the ground-truth target images. On this basis, precision, recall, and F1 score are calculated for each turn of M-CIG by comparing the object presence in the generated target image with that in the ground-truth target image:

$$\small\begin{split}&\mathrm{precision}=\frac{|O_{gen}\cap O_{gt}|}{|O_{gen}|}\\ &\mathrm{recall}=\frac{|O_{gen}\cap O_{gt}|}{|O_{gt}|}\\ &\mathrm{F1}=2\cdot\frac{\mathrm{precision}\cdot\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}\end{split}$$

(16)

where $O_{gen}$ and $O_{gt}$ denote the objects in the generated target image and the ground-truth target image, respectively. RSIM is calculated on the last turn of M-CIG by comparing the object topology in the generated target image with that in the ground-truth target image:

$$\small\begin{split}&\mathrm{RSIM}=\mathrm{recall}\cdot\frac{|E_{gen}\cap E_{gt}|}{|E_{gt}|}\end{split}$$

(17)

where $E_{gen}$ and $E_{gt}$ denote the edges interconnecting $O_{gen}\cap O_{gt}$ in the generated target image and the ground-truth target image, respectively.

We use PyTorch [43] to implement CDD-ICM, and use HuggingFace’s Transformers [71] to load CLIP. In the three CLIP-based encoders, we adopt the basic version of CLIP (i.e. CLIP-ViT-B/32) as the backbone, set the backbone activity ratio $\eta$ to $0.001$, and set the output dimensionality $d$ of all the linear projection layers to $512$. In the denoising diffusion mechanism, we use $1000$ time steps for the training (i.e. $T=1000$), and use the $250$ time steps uniformly distributed among them for the inference. For ICM guidance, we set $\psi$ to $2$. For classifier-free guidance, we set the value of $\lambda$ to $0.2$ and $\phi$ to $3$. To calculate $L^{cdd}$, we set $\gamma$ to $1.5$. To calculate $L^{icm}$ and $L^{n-icm}$, we initialize $\tau$ to $e^{-1}$. To calculate $L^{joint}$ in the second training stage, we set $\delta$ to $0.1$. For the optimization in each training stage, we apply an AdamW optimizer [31] with an initial learning rate of $0.0001$ and a weight decay factor of $0.01$. We perform the optimization on $8$ NVIDIA V100 $16$GB GPUs in parallel, and set the mini-batch size on each GPU to $32$. We calculate the average loss on the validation subset after every $10$ epochs. If the resulting number is reduced, then we save the current CDD-ICM model, otherwise we restore the CDD-ICM model to the previous saved version. We decay the learning rate by $50\%$ after each restoration, and terminate the optimization after the $5$th restoration.

On each dataset, we train a CDD-ICM model by using the training set for optimization and using the validation set for model selection. To compare CDD-ICM with the existing M-CIG methods, we use the test set for evaluation, and finally report the precision, recall, F1 score, and RSIM over all the M-CIG samples in the test set. As shown in Table 1, we achieve SOTA performance on both datasets. Specifically, on CoDraw, CDD-ICM outperforms the existing M-CIG methods by a large margin, where the advantage in precision, recall, and F1 score is relatively larger than that in RSIM. On i-CLEVR, although the existing M-CIG methods did not leave too much room for improvement in precision, recall, and F1 score, CDD-ICM is still better than them in these metrics, reaching almost perfect numbers, and also outperforms them in RSIM.

From all the M-CIG methods in Table 1, we observe two regularities. On the one hand, the performance of these methods on CoDraw is generally worse than that on i-CLEVR. By comparing CoDraw with i-CLEVR, we speculate that this is mainly because the modification texts in CoDraw are commonly longer and more complicated than those in i-CLEVR, which makes it more difficult to generate the desired target images. On the other hand, the performance of these methods in object presence, which is reflected by precision, recall, and F1 score, is generally better than that in object topology, which is reflected by RSIM. By investigating both datasets, we speculate that this is mainly because good performance in object presence just requires correctly identifying the names and attributes of objects from the modification texts, while that in object topology usually requires comprehensively understanding the modification texts.

To visually demonstrate the capability of CDD-ICM, we select several representative M-CIG samples from the test set of both datasets, and use the inference results of CDD-ICM on them as demo cases. A demo case from CoDraw and another one from i-CLEVR are shown in Figure 3, and more demo cases are available in the appendix. In the demo cases from CoDraw, the generated target images contain most of the desired objects, but there are still some extra and missing objects. Besides, it also shows that on CoDraw, CDD-ICM struggles with accurately positioning and orienting objects. In the demo cases from i-CLEVR, the generated target images are very similar to the ground-truth target images, where only a few objects are misaligned.

To probe the performance contribution from each design point of CDD-ICM, we conduct the following five ablation experiments. As shown in Table 2, in each ablation experiment, we change a corresponding design point, and report the resulting F1 score and RSIM on each dataset.

ICM. We disable ICM by skipping the first training stage and setting $\delta$ to $0$ when calculating $L^{joint}$ in the second training stage. As a result, we observe a significant drop in F1 score and RSIM. This verifies the effectiveness of learning ICM as an auxiliary objective of CDD. Besides, we also observe a significant drop in the converging speed of the second training stage. This further verifies that learning ICM is beneficial for learning CDD.

ICM Guidance. We disable ICM guidance by skipping the third training stage and setting $\psi$ to $0$. As a result, we observe a drop in F1 score and RSIM. This verifies the effectiveness of ICM guidance.

Classifier-free Guidance. We disable classifier-free guidance by setting $\lambda$ to $0$ and setting $\phi$ to $1$. As a result, we observe a drop in F1 score and RSIM. This verifies the effectiveness of classifier-free guidance.

Fine-tuning of CLIP. For the fine-tuning of CLIP, which is controlled by the backbone activity ratio $\eta$, we examine two extreme cases. On the one hand, we make CLIP frozen by setting $\eta$ to $0$. On the other hand, we make CLIP fully-trainable by setting $\eta$ to $1$. As a result, we observe a significant drop in F1 score and RSIM in both cases. This verifies the necessity of applying $\eta$.

Iterative Setting. In the evaluation, we disable the iterative setting by using the ground-truth target image at each turn of M-CIG as the reference image of the next turn, which actually downgrades M-CIG to CIG. As a result, we observe a significant rise in F1 score and RSIM. This verifies that M-CIG is more challenging than CIG.