TextureDiffusion: Target Prompt Disentangled Editing for Various Texture Transfer

Diffusion models [16, 17, 18, 19, 20] are generative models that can generate data by iterative denoising starting from Gaussian noise. It include a forward process and a reverse process. The forward process adds noise to the data sample $x_{0}$ at time step $t$ to generate the noisy sample $x_{t}$: $q(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}}_{t}x_{0},(1-\bar{\alpha}_{t})I)$, where $\bar{\alpha}_{t}=\Pi_{i=1}^{t}\alpha_{i}$, $\alpha_{i}$ denotes the predefined noise schedule. The reverse process removes the noise from the previous sample $x_{t}$ to generate a clean sample $x_{t-1}$: $p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{t}),$ where $\sigma_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$, $\beta_{t}=1-\alpha_{t}$, $\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha}_{t}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon)$. Noise $\epsilon$ can be predicted by a neural network $\epsilon_{\theta}(x_{t},t)$ trained on the objective: $L=E_{x_{0},\epsilon,t}(\|\epsilon-\epsilon_{\theta}(x_{t},t)\|).$ Additionally, when $\epsilon_{\theta}$ is conditioned on the text prompt $P$, it can be formulated as $\epsilon_{\theta}(x_{t},t,P)$. After doing so, the diffusion model can generate images that match the provided text prompt.

Our method is based on the state-of-the-art text-to-image model Stable Diffusion (SD) [21]. SD belongs to Latent Diffusion Models (LDMs) that performs the diffusion process in the latent space. SD is based on U-Net architecture [22]. The U-Net contains a series of basis blocks, each containing a residual block [23], a self-attention module, and a cross-attention module [24]. Self-attention module contains important semantic information and its output can be formulated as follows:

where $Q$, $K$, and $V$ are the query, key, and value features projected from spatial features with corresponding projection matrices.

After directly modifying the target prompt to “$<$texture$>$”, information about the content of the input image is lost. Thus the structure of the input image needs to be preserved.

As mentioned in previous work [25, 26, 27], in the self-attention module of SD U-Net, the query features control the overall layout of the generated image, while the key and value features control the semantic contents. Therefore we inject the query features in the self-attention module into the generation process of the edited image and the result is shown in Fig. 4. The structure of the input image is partially preserved after injecting the query features, but it is still insufficient and more structural information needs to be injected. Inspired by  [14], which demonstrated that features in residual blocks contain the structural information of the input image, we further inject features in residual blocks and the experimental results are shown in Fig. 4. The structure of the input image can be well maintained when query features in the self-attention module and features in residual blocks are injected at the same time.

In addition, since the generation process of the diffusion model is from the overall layout to the semantic details, structural information is injected only in the first and middle stages of the generation process. We do not inject the structural information in the later stages, which enables the texture details to be fully represented.

To localize the edit on the target object while keeping the rest unchanged, we introduce an edit localization technique.

Initially, the position of the target object must be identified. Drawing inspiration from [13], the cross-attention map contains location information of the prompt tokens. Therefore, we aggregate cross-attention maps across all heads and layers of the spatial resolution of 16×16. Subsequently, we extract the map corresponding to the target object and binarize it to derive the mask $M$.

Since the self-attention module in SD U-Net contains important semantic information, we blend the self-attention results from the source image and the edited images:

where $\odot$ represents the Hadamard product and $\bar{R}^{l}$ denotes the ultimate attention output. To further keep the remainder unchanged, we blend the intermediate latents of the source and edited images:

where $Z_{t}$ denotes the intermediate latents of the edited image. Using this edit localization technique, the edit is restricted to the target object, keeping the remainder unchanged.

We compare the proposed method to state-of-the-art baselines that can be applied to text-guided image editing tasks, including: SDEdit [28], P2P [13], PnP [14], MasaCtrl [25], FPE [29], and InfEdit [15]. We use their open-sourced codes to produce the editing results.

Qualitative Experiments  As shown in Fig. 3, we present the qualitative results of our method compared with the baselines. SDEdit edits the input image by adding noise to it and then denoising it, but this process does not preserve the structure of the input image. P2P adds an additional cross-attention map corresponding to texture, which alters the structure of the input image and changes the shape of the target object. MasaCtrl applies mutual self-attention to preserve the contents of the input image, preventing changing the texture of the target object. PnP and FPE inject structural information from the input image to maintain the structure, and InfEdit uses virtual inversion to achieve efficient image reconstruction. However, among these methods, the description of the input image in the target prompt restricts the representation of the texture, preventing the texture to be successfully transferred. In contrast, our method successfully transfer various textures to the target object while keeping the remainder unchanged.

Quantitative Experiments  The dataset is the editing type of changing material on PIE-Bench [30]. We find that some text prompts do not meet the standards for changing material, so we modify them. To demonstrate the efficiency of our method, we employ six metrics including four aspects: structure distance [31], background preservation (PSNR, LPIPS [32], MSE, and SSIM [33] outside the annotated editing mask), and edit prompt-image consistency (CLIP Similariy [34]) . Note that to evaluate whether the texture has been transferred to the target object, we set the prompt to “$<$texture$>$”  only and calculate the CLIP Similarity between the prompt and the target object region of edited image.

Tab. I shows quantitative results of our method compared with the baselines. As seen, our method outperforms the baselines by achieving highest preservation of structure, highest preservation of background and highest fidelity to the prompt.

We conduct an ablation study to validate the effectiveness of our designed core components and the results is shown in Fig. 4. As seen, the texture can be fully represented when the target prompt is directly set to “$<$texture$>$”. When both query features in self-attention module and features in residual blocks are added during the generation of the edited image, the structure of the input image is well preserved. When applying the proposed edit localization technique, the background is well retained.