PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering

Image composition entails seamlessly incorporating the given object into the specific visual context without altering the object’s appearance while ensuring natural transitions. Earlier studies employ personalized concept learning (Gal et al., 2022, 2023; Kawar et al., 2023; Kumari et al., 2023; Ruiz et al., 2023), yet they often rely on costly instance-based optimization and encounter limitations in generating concepts with specified backgrounds. Although these challenges are effectively addressed by utilizing diffusion models to explicitly incorporate additional guiding images (Song et al., 2022; Yang et al., 2023), retraining these pre-trained models on customized datasets risks compromising their rich prior knowledge. Consequently, these methods exhibit limited compositional abilities beyond their training domain, and they still demand substantial computational resources.

A recent study (Lu et al., 2023) develops a training-free framework, TF-ICON, that leverages attention weights from several samplers to composite the self-attention map of the diffuser during the composition process. Despite achieving notable success in this task, it still faces substantial challenges in preserving the appearance of complex objects (Fig. 2 (left)) and synthesizing a natural coherence (Fig. 2 (right)). The primary issue resides in its composite self-attention maps: the incorporation of attention weights from different contexts introduces potential ambiguity. To be precise, in TF-ICON, each sampler’s weights are calculated within its specific global context (Vaswani et al., 2017), and the act of forcibly combining them leads to the synthesized confusion in coherent relations and loss of appearance information. This undoubtedly hampers its ability to accurately represent the intrinsic characteristics of the given object and establish a coherent relationship. These issues are further accentuated by its overemphasis on background generation (a tailored sampler). It is essential to underscore that the background inherently necessitates no alteration, given the user’s exclusive focus on foreground area generation. This superfluous emphasis not only introduces computational overhead but also leads to compromise on the foreground synthesis. Moreover, this approach introduces the unwanted artifacts in the transition area as shown in Fig. 4, further hindering the natural coherence establishment.

In summary, while the current training-free method has mitigated the need for costly optimization and retraining, it remains incapable of capturing the nuanced appearance of objects and forging dependable coherent relations. Urgent exploration of more effective steering mechanisms for training-free composition, without compromising efficiency, is imperative.

In this paper, we novelly formulate this task as a subject-guided local editing problem, focusing solely on foreground generation due to the unnecessary alteration of the background. To achieve this, as depicted in Fig. 3, we utilize the pre-trained Latent Diffusion Model (LDM) (Rombach et al., 2022) to edit local foreground areas evolvingly based on the given object and text. Then the resultant edited area is spatially combined with a certain noised background at each step to maintain the scene. To address the remaining issues, we propose a faster training-free method, dubbed PrimeComposer, a progressively combined diffusion model that integrates the user-provided object to the background through well-designed attention steering across different noise levels. This progressive steering is primarily facilitated by our Correlation Diffuser (CD). Specifically, in each step, we combine specific noisy-level versions of the provided object and background at the pixel level while simultaneously segmenting the synthesized subject from the previous step’s results. These components are then fed into the CD’s diffusion pipeline where the synthesized subject interacts with both the referenced object and background within self-attention layers. The interrelation information, encoded as prior weights, encapsulates rich mutual correlations and object appearance features. Consequently, we infuse them into LDM’s self-attention maps (yellow and orange regions in Fig. 3 (bottom right), respectively) to meticulously steer the preservation of object appearance and ensure harmonious coherence establishment. To fortify the steering impact, we further advance the classifier-free guidance (Ho and Salimans, 2022), elaborated in Sec. 4.5. Additionally, we introduce Region-constrained Cross-Attention (RCA), replacing cross-attention layers in LDM, to confine the impact of specific subject-related tokens to predefined regions in attention maps. This helps mitigate unwanted artifacts, thereby enhancing coherence in the transition area.

Note that CD is the only sampler for steering and all the infused attention weights are computed from the accordant context. However, intuitively infusing object appearance-related weights on all layers will result in the subject overfitting, as shown in Fig. 5, thereby leading to unexpected coherence problems: style inconsistency. Therefore, we propose to control appearance infusion only in the decoder part of the U-Net since the decoder has been proven to focus on learning the appearance and textures (Zhang et al., 2023b).

Our contributions can be summarized as follows:

(1) We formulate image composition as a subject-guided local editing problem and propose a faster training-free method to seamlessly integrate given objects into the specific visual scene across various domains. (2) We develop the CD to simultaneously alleviate the challenge of preserving complex objects’ appearance and synthesizing natural coherence by well-designed attention steering. (3) We introduce RCA to confine the impact of specific subject-related tokens in attention maps, thereby effectively addressing the unwanted artifacts. (4) Our method exhibits the fastest inference efficiency and extensive experiments demonstrate our superiority both qualitatively and quantitatively.

Image composition serves as a valuable tool for diverse downstream tasks, e.g., entertainment, and data augmentation (Dwibedi et al., 2017; Liu et al., 2021; Lu et al., 2023; Cai et al., 2024; Wang et al., 2024b). The practice for this task broadly falls into two categories: text-guided and image-guided composition.

Text-guided composition (Avrahami et al., 2023, 2022; Chefer et al., 2023; Feng et al., 2022; Liu et al., 2022) involves generating images based on a text prompt that specifies multiple objects. This approach allows for diverse appearances as long as the semantics align with the prompt. Despite its effectiveness, semantic errors may arise, especially with prompts involving multiple objects. These errors, including attribute leakage and missing objects, often necessitate extensive prompt engineering (Witteveen and Andrews, 2022).

Conversely, image-guided composition (Brown et al., 2022; Gafni and Wolf, 2020; Li et al., 2023; Song et al., 2022; Xue et al., 2022; Yang et al., 2023; Zhang et al., 2021; Lu et al., 2023) incorporates specific objects and scenarios from user-provided photos, potentially with the assistance of a text prompt. This approach presents greater challenges, particularly when dealing with images from different visual domains. Specifically, image-guided composition encompasses various sub-tasks (Niu et al., 2021), such as object placement (Azadi et al., 2020; Chen and Kae, 2019; Lin et al., 2018; Tripathi et al., 2019; Zhang et al., 2020b; Wang et al., 2023, 2024a), image blending (Wu et al., 2019; Zhang et al., 2020a), image harmonization (Cong et al., 2020; Cun and Pun, 2020; Jiang et al., 2021; Xue et al., 2022; Niu et al., 2024a, b), and shadow generation (Hong et al., 2022; Liu et al., 2020; Sheng et al., 2021; Zhang et al., 2019). These diverse tasks are typically tackled by distinct models and pipelines, showcasing the intricacy of image-guided composition. Recently, diffusion models have demonstrated impressive capabilities in image-guided composition by simultaneously tackling all these subtasks. While prior studies have explored personalized concept learning (Gal et al., 2022, 2023; Kawar et al., 2023; Kumari et al., 2023; Ruiz et al., 2023), they often rely on costly instance-based optimization and face limitations in generating concepts with specified backgrounds. To overcome these challenges, subsequent studies (Song et al., 2022; Yang et al., 2023) effectively incorporate additional guiding images into diffusion models through retraining pre-trained models on tailored datasets. However, this poses a risk of compromising their rich prior knowledge. Besides, these models exhibit limited compositional abilities beyond their training domain and demand substantial computational resources. A recent study (Lu et al., 2023) introduces a training-free method involving the gradual injection of composite self-attention maps through multiple samplers. Despite its remarkable success, it encounters challenges in preserving the appearance of complex objects and synthesizing natural coherence.

Diverging from the approaches mentioned earlier, we novelly formulate this task as an subject-guided local editing problem. Our progressively combined Diffusion, PrimeComposer, intricately depicts the object and achieves harmonious coherence through well-designed attention steering across a progression of noise levels.

Denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) are designed to reverse a parameterized Markovian image noising process. They start with isotropic Gaussian noise samples and gradually transform them into samples from a training distribution by iteratively removing noise. Given a data distribution $\textbf{{x}}_{0}\sim q(\textbf{{x}}_{0})$, the forward noising process produces a sequence of latent $\textbf{{x}}_{1},...,\textbf{{x}}_{T}$ by adding Gaussian noise with variance $\beta_{t}\in(0,1)$ at each time step t:

When $T$ is sufficiently large, the last latent $\textbf{{x}}_{T}$ approximates an isotropic Gaussian distribution.

An important property of the forward noising process is that any step $\textbf{{x}}_{t}$ can be directly sampled from $\textbf{{x}}_{0}$, without generating the intermediate steps:

where $\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$, $\alpha_{t}=1-\beta_{t}$, and $\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$.

To draw a new sample from the distribution $q(\textbf{{x}}_{0})$, the Markovian process is reversed. Starting from a Gaussian noise sample $\textbf{{x}}_{T}\sim\mathcal{N}(0,\mathbf{I})$ with $\alpha_{t}=1-\beta_{t}$, a reverse sequence is generated by sampling the posteriors $q(\textbf{{x}}_{t-1}|\textbf{{x}}_{t},\textbf{{x}}_{0})$.

However, $q(\textbf{{x}}_{t-1}|\textbf{{x}}_{t},\textbf{{x}}_{0})$ is unknown and depends on the unknown data distribution $q(\textbf{{x}}_{0})$. To approximate this function, a deep neural network $p_{\theta}$ is trained to predict the mean and covariance of $\textbf{{x}}_{t-1}$ given $\textbf{{x}}_{t}$ as input:

Rather than inferring $\mu(\textbf{{x}}_{t},t)$ directly, Ho et al. (Ho et al., 2020) propose to predict the noise $\epsilon_{\theta}(\textbf{{x}}_{t},t)$ that was added to $\textbf{{x}}_{0}$ to obtain $\textbf{{x}}_{t}$ according to Equation 2. Then, $\mu(\textbf{{x}}_{t},t)$ is derived using Bayes’ theorem:

For more detail see (Ho et al., 2020). In this work, we leverage the pre-trained text-to-image Latent Diffusion Model (LDM) (Rombach et al., 2022), a.k.a. Stable Diffusion, which applies the noising process in the latent space.

This section begins with an overview of our method, followed by an in-depth explanation of self-attention steering based on our Correlation Diffuser (CD). Subsequently, we will explore the details of our Region-constrained Cross-Attention (RCA). Finally, we will introduce our careful extension of classifier-free guidance (CFG) during inference.

In this paper, we formulate image composition as a subject-guided local image editing task and propose a faster training-free diffuser, PrimeComposer. Leveraging well-designed attention steering, primarily through the Correlation Diffuser, our method seamlessly integrates foreground objects into noisy backgrounds while maintaining scene consistency. The introduction of Region-constrained Cross-Attention further enhances coherence and addresses unwanted artifacts in prior methods. Our approach demonstrates the fastest inference efficiency and outperforms existing methods both qualitatively and quantitatively in extensive experiments.

This work was supported by National Natural Science Fund of China (62176064) and Shanghai Municipal Science and Technology Commission (22dz1204900).