Streamlining Image Editing with Layered Diffusion Brushes

Huang et al. surveyed existing DM-based image editing techniques and categorized them into three groups: training-based approaches, testing-time fine-tuning methods, and training and fine-tuning-free methods (Huang et al., 2024). Per the survey, these methods utilize various types of inputs to control the editing process, such as text, masks, segmentation maps, or dragging points. This survey also categorizes image editing methods based on the types of edits they can accomplish, including semantic editing (adding, removing, or replacing objects; background change; and emotional expression modification), stylistic editing (colour, texture, or style change), and structural editing (moving objects; changing size, position, shape, or pose of objects; and viewpoint change).

Our system requires no training or fine-tuning, putting it in the latter category of models and enabling its use with any existing pretrained model.

One of the most common group of works in training-based approaches is instructional editing via full supervision (Brooks et al., 2022; Guo and Lin, 2023; Geng et al., 2023). Many such approaches, while attempting to make localized changes to images, end up creating some degree of global change as well, which is not desirable. InstructPix2Pix (Brooks et al., 2022) is a well-known method in this category, which constructs a large dataset of instruction-image pairs using Stable Diffusion(Rombach et al., 2022) and Prompt-to-Prompt (Hertz et al., 2022), and trains an instructional image editing diffusion model. While this method is widely recognized and is one of the benchmarks in our study, it faces a challenge in expressing certain types of edits solely through text and instruction, such as changing the colour or style of a single object amongst several similar objects in an image. This limitation underscores the utility of allowing users to specify regions for editing purposes.

Another group of works, including Prompt-to-Prompt (Hertz et al., 2022) and Zero-shot Image-to-Image Translation (Parmar et al., 2023), edit images by modifying the attention maps through discerning the intrinsic principles in the attention layers and then utilizing them to modify the attention operations. One of the challenges with these works is obtaining optimal results when making multiple small and fine-tuned edits.

Some other techniques utilize inversion (Choi et al., 2021) to preserve a subject while modifying the context. Textual Inversion (Gal et al., 2022) and DreamBooth (Ruiz et al., 2023) synthesize new views of a given subject given 3–5 images of the subject and a target text.

Finally, among training and fine-tuning-free methods, modification of inversion and sampling formulas are common techniques. Through the inversion process, a real image transforms into a noisy latent space, and then the sampling process is used to generate an edited result given the target prompt. We utilize the Direct Inversion approach (Ju et al., 2023), which is fast and efficient for obtaining the noisy latent space, to support editing real images. However, we employ a different approach for editing the image that modifies the latent space during sampling.

Another common category of image manipulation is image inpainting, which involves adding new content to an image. Traditional inpainting techniques can also be utilized for object removal by providing no text prompt and guidance (Avrahami et al., 2022). While these models are effective for generating new content in images, they are not suitable for making small and targeted adjustments (Avrahami et al., 2022; Lugmayr et al., 2022; Saharia et al., 2022a). We also compare our proposed method with Stable Diffusion Inpainting (SD-Inpainting) (Rombach et al., 2022).

Similar to many other works in the literature, including DiffEdit (Couairon et al., 2022), Blended Diffusion (Avrahami et al., 2022), Inpainting (Yu et al., 2023), and Blended Latent Diffusion (Avrahami et al., 2023), we utilize masks for creating our edits. However, in contrast, our objective is to fine-tune the image and make localized adjustments to the existing image while keeping the remaining content intact. Unlike the method in (Avrahami et al., 2023), our method modifies the original latent space, which results in better preservation of the original context of the image and more natural blending of the results into the image, as well as enabling substantial performance optimizations.

In (Avrahami et al., 2023), LDM’s latent representation is obtained using a VAE, which is lossy and results in the encoded image not being reconstructed exactly, even before any noise addition. (Avrahami et al., 2023) incorporates a background reconstruction strategy to address this, which increases computation and lowers inference time. However, since one of the primary goals of this study is to maximize speed for a real-time user experience, this method is not feasible. On the other hand, we store the latent for regeneration in memory, resulting in much faster results.

Layer-based image editing has long been a cornerstone in the field of computer graphics (Porter and Duff, 1984). Recently, some works have delved into merging this concept into the evolving landscape of machine learning-driven methodologies (Bar-Tal et al., 2022; Sarukkai et al., 2024). These layered representations offer a dynamic platform for the manipulation of distinct components within an image. Often, the essence of this process lies in transforming a singular input image into a multi-layered representation.

Collage Diffusion (Sarukkai et al., 2024) presents an interesting layer-based scheme by modifying text-image cross-attention with the layers’ alpha masks to generate globally harmonized images that respect a user’s desired scene composition. They also introduce an editing framework utilizing a modified Blended Latent Diffusion (Avrahami et al., 2023), assuming a pre-existing layered structure as input, harnessing the power of machine learning to synthesize cohesive image outputs from these pre-arranged layers.

One significant improvement provided by our method, unlike Collage Diffusion, is the ability to have completely independent layers. Additionally, we employ a different approach, focusing on user-oriented and real-time editing experiences.

Therefore, our method is a training and fine-tuning-free approach that utilizes masks and text as input, offering a broad range of edits, including most semantic and stylistic editing tasks, and some structural edits, such as changing the size of objects.

As mentioned in section 1, we use an LDM-based variant of image generative models. Similar to (Lugmayr et al., 2022; Avrahami et al., 2023), we make intermediate adjustments to the latent space using the pre-trained LDM. Therefore, Layered Diffusion Brushes does not require any additional training and only modifies the latents during the reverse diffusion process.

1 presents the algorithm for generating intermediate latents. This part simply caches two latents, $Z_{r}$ and $Z_{t}$. Here, $Z_{r}$ serves as the initial latent for editing. In this formulation, $r=N-n$, where $N$ is the total number of steps to create the initial image, and $n$ is the total number of steps for editing. $Z_{L1}$ represents the latent used for blending the edited latent in layer 1.

We first initialize the sample $Z_{0}=\epsilon_{0}$ and noise level $\sigma_{0}$ ($i=0$) if the image is generated using the DM. If a real image is used, we obtain the initial noise latent for the image using the Direct Inversion method presented in (Ju et al., 2023). Based on our testing, this method is significantly faster in terms of speed and comparable in terms of performance to other methods in the literature, including null text inversion (Mokady et al., 2023) and Negative-Prompt Inversion (Miyake et al., 2023) methods.

Algorithm 2 contains the main backbone of the system, which handles both editing and layering functionalities.

To initiate an edit, the algorithm begins by generating a new noise pattern $Z^{\prime}_{0}=\epsilon^{\prime}k$ sampled from $\mathcal{N}(0,S^{\prime 2}{noise}I)$. Here, we generate a new noise pattern from a different seed (set per-layer). Then, $Z^{\prime}0$ is scaled to match the variance of $Z{r}$ obtained from the previous step. This ensures that the additive noise is not too different from the latent for editing, which would result in visual artifacts. The new noise is added to the original latent, controlled by the mask and $\alpha_{k}$. The effect of using different values for $n$ and $\alpha$ will be discussed in detail in 3.

During the edit loop, at step $t$, in each layer, the latent of the new seed $S^{\prime}$ and the previous layer’s latent are merged using the mask, thus confining the influence of the new latent to the masked region. Subsequently, the new latent region is progressively denoised and integrated into the existing image contents from steps $t$ through $N$. To keep the UI simple and reduce the number of user-controlled parameters, we fix the value of $t$ in our implementation. Preliminary tests suggested that the value of $t=N-2$ works well for many applications.

Finally, there is separate logic that tracks all the layers and their corresponding parameters, which is triggered whenever a new layer is created or if the user switches between the layers. Consequently, all the layers are created and stacked one by one, and the appropriate $Z_{L_{k}}$ is provided to the Algorithm 2 to perform the edit.

When creating a new layer, the user specifies a mask $m$. For each layer and the corresponding mask $m_{k}$, the user specifies an integer $n_{k}$ (per 2), which is the total number of steps that the noise is introduced to the latent, and a tunable parameter $\alpha_{k}$ to control the additive noise strength.

We show the overview of the core algorithm of the proposed method in 2 and also present the updated pseudocode for the denoising process:

Our goal with Layered Diffusion Brushes was to design a practical tool for artists and graphic designers to use. In particular, we aimed to design a set of controls that would be intuitive and flexible, without overwhelming users with excessive choices. Figure 3 provides an overview of the user interface. As illustrated in the figure, the UI comprises the following primary sections (each section highlighted with the corresponding number on the image):

1. (1)

   Model Section

   • •

     This section in the UI enables users to load various model combinations, including pre-trained models, schedulers, and LoRA (Hu et al., 2021).

2. (2)

   Generation section

   • •

     This section allows users to either generate a new image using a seed and prompt combination, or upload a real image that will be inverted.

3. (3)

   Image Canvas

   • •

     This canvas serves as the workspace where users interact with and make edits to images.

4. (4)

   Editing Section

   • •

     This section empowers users to create and access different layers and adjust various controls to make desired edits.

Two types of interactions are provided for performing edits, as shown in Figure 4:

Box Mode: Users can click or drag on the image to move a resizable square mask around the image. This option enables users to quickly and interactively explore how various parts of the image will change in response to a given set of editing settings (prompt and strength), simply by moving the cursor.

Custom Mask Mode: Users can draw the mask in any shape they like using a resizable paintbrush, clicking and dragging to add to the mask. Users can navigate between new generation samples by scrolling the mouse up or down while hovering over the image, allowing them to rapidly explore variations on their edit.

We envision that users could use the box option first to choose a good location to place their edit and refine the prompt/strength settings, followed by the custom mask to refine the shape and appearance of the edit.

Proper utilization of the layering capability enables the tool to achieve more complex effects: users can stack, hide and unhide layers and adjust the strength of each layer. Furthermore, each layer can be edited independently without substantially affecting the appearance of edits in higher or lower layers.

Here are the main hyperparameters used in the Layered Diffusion Brushes editing method:

• •

  Number of regeneration steps ($n$): An integer value that specifies the number of steps that the Layered Diffusion Brushes will run to make the edit

• •

  Brush Strength ($\alpha$): A float number that corresponds to the $\alpha$ value in equation (1) which controls how strong the noise should be applied at the targeted region. Initial $\alpha^{*}$, specified by the user has a value between 0 and 100, which will be scaled later on in the algorithm using the following equation:

  (1)

  $$\alpha=\frac{\sqrt{\left|\frac{\alpha}{100}\cdot\left(\sigma-2\cdot\frac{\text{Cov}(Z_{n_{k}},x^{\prime}_{0})}{\text{Var}(Z_{n_{k}})}\right)\right|}}{\sqrt{\frac{\sum_{i=1}^{m}\sum_{j=1}^{n}[m_{ij}\neq 0]}{W}}}$$

  where $x^{\prime}_{0}$ and $Z_{n_{k}}$ are the new noise latent and latent for regeneration respectively (as notes on …), $\sigma$ is the acceptable range for the variance of the $Z_{n_{k}}$(we used $\sigma$=0.25), $m$ is the corresponding mask, and $W$ is the width of the $Z_{n_{k}}$ (W=512)

• •

  Seed Number ($s^{\prime}$): An integer number that will be used for generating the Gaussian noise pattern in the specified region. As with normal image generation, the UI provides buttons to randomize the seed or reuse the previous seed. Moving the box around (in box mode) or using the scroll wheel (in custom mask mode) will adjust the seed automatically.

• •

  Brush Size $r$: An integer value that dictates the radius of the box when utilized in box mode, or the size of the brush in custom mask mode. The formulation of $\alpha$ in Equation 1 aims to mitigate the impact of varying brush sizes on the quality of edits by considering the number masked pixels and adjusting accordingly.

The number of edit steps ($n$) and the brush hardness ($\alpha$) parameters jointly control the magnitude of the intermediate additive noise and ultimately the amount of applied change to the latent image. Higher values for the brush hardness will result in a bigger change. Figure 5 (top row) shows the effect of a wide range of different hardness values while keeping other parameters the same. As shown, if the mask strength is too high, the LDM cannot fully denoise, and the additive noise causes artifacts. Conversely, if the $\alpha$ value is too small, Layered Diffusion Brushes will not be able to make sufficient adjustments. We use a generalized version of strength in order to make the parameter more intuitive, as discussed above.

Similarly, we tested the effect of using different $n$ values while keeping the rest of the parameters stationary. As demonstrated, if the editing is run for fewer steps (lower $n$) , the LDM cannot recover from it and will produce artifacts, while if $n$ has a higher value, the area inside the mask will be denoised too much and cannot properly blend well with the rest of the image.

Additionally, there is a correlation between the two parameters. When the noise is introduced in later stages of diffusion (higher value of $n$), the model has less time to recover from the additive noise. Therefore, it is recommended to adjust the $alpha$ value to a higher number. Conversely, if the noise is introduced in the early stages of diffusion, and if the value of $alpha$ is not large enough, the additive noise will be dissolved into the input image latent and several stages of diffusion will eliminate its effect. Therefore, if $n$ is small, the $alpha$ value should be larger to achieve optimal results.

We conducted a user study in order to evaluate the effectiveness of Layered Diffusion Brushes for providing targeted image fine-tuning, in comparison with InstructPix2Pix and the SD-Inpainting methods.

Figure 12 illustrates the outcomes of the post-study CSI survey. Overall, participants expressed positivity towards Layered Diffusion Brushes, indicating that it enhanced their enjoyment, exploration, expressiveness, and immersion, while also deeming the results worth their effort.