FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models

The research on the controllable image generation task in the diffusion model can be broadly divided into three stages.

The first stage is mainly based on the iterative denoising process and achieves controllable generation effects by rationally using the input image to generate a deterministic guided generation paradigm [9, 36, 34]. This stage of the work controls the semantic similarity between the generated image and the input image by influencing the proportion of information mixing in the denoising network of U-Net. The second stage is based on the image generation strategy guided by the display classifier. Optimization is performed by adding gradient information from the classifier to the loss function. This idea first originated from classifier guidance[46] and further advanced in[11, 42]. Since then, numerous studies have broadened the scope of classifiers, extending the classification guidance of the diffusion model to encompass diverse modalities such as text, images, and other multi-modal data [3, 24, 52, 17, 32]. The third stage marks the era of large models based on implicit classifiers. To address the issue of declining diversity in classifier guidance, classifier-free guidance strategy [19] emerges later. This approach involves decomposing the gradient guidance from the explicit classifier into two components. One component is an unconditionally generated gradient prediction model, akin to the conventional DDPM [20]. The other is a gradient prediction model based on conditional generation, conceptualized as a U-Net network with an overlay of a cross-attention mechanism. The success of this approach has catalyzed the evolution of a variety of subsequent image editing technologies. These technologies utilize the diffusion model as a foundational framework and integrate the attention mechanism, resulting in notable progress in the application of the diffusion model across diverse fields. Notable projects in this domain include DALLE-2 [40], DreamFusion [38], Stable Diffusion Model (SDM) [41], and more.

Here, our primary focus lies on Grounded Generation and Layout-driven Generation within the context of the controllable generation problem. Representative works in this area include GLIGEN [29], ControlNet [58], and IP-Adapter [56]. We aim to investigate the generative capabilities of diffusion models in addressing semantic-level conditional guidance, particularly in scenarios with limited sample sizes. For example, appearance transfer task [47]. It needs to preserve the structure of the target image while applying the desired appearance attributes.

Explicit decomposition is aimed at breaking down the representation of data or a model into simpler, more independent components or factors. Specifically, this process involves splitting a high-dimensional representation space into multiple low-dimensional subspaces, each responsible for encoding a specific aspect or attribute of the data. Through explicit decomposition, neural networks can more easily understand various aspects of the data, such as geometry, color distribution, texture, etc., in the image [21].

Traditionally, filtering algorithms have been considered the explicit decomposition approach as they can break down input data into components at different frequencies or spatial scales. Examples of common filters include the Gaussian filter, Sobel filter [45], Adaptive filter [22], and Gabor filter [12, 43]. These filters can weigh data at different frequencies or scales to suppress or enhance specific features. Therefore, performing preprocessing operations may help neural networks better understand the structure and features of data in order to acquire more refined representations.

The forward diffusion defines a known Gaussian translation process. Then the image intermediate quantity $x_{t}$ at each moment of the forward process can establish a unique relationship with the input image $x_{0}$ as:

$$q(x_{t}|x_{0}):=N(x_{t};\sqrt{\bar{\alpha_{t}}}x_{0},(1-\bar{\alpha_{t}})I),\;\;\;x_{t}=\sqrt{\bar{\alpha_{t}}}x_{0}+(1-\bar{\alpha_{t}})\epsilon$$

(1)

The reverse denoising process of the diffusion model can be seen as a migration process to the target data distribution. During this process, the model will continuously try to reduce noise and be guided by the condition $c$ to restore the structure or characteristics of the specified data. Every migration process can be expressed as:

$$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}\left(x_{t},c,t\right)\right)+\sigma_{t}z$$

(2)

The focus of this process is the network’s prediction of noise distribution: $\epsilon_{\theta}\left(x_{t},c,t\right)$. This prediction is affected by the current moment $x_{t}$ and condition $c$, and condition $c$ comes from the external reference image $y$ mapping result. Therefore, if we perform filters $f_{\gamma}$ on either side of the input like making $c=\mathrm{Encoder}(y)$ become ${c{{}^{\prime}}=~{}\mathrm{Encoder}(f_{\gamma}(y))~{}}$, it will affect the prediction results of the noise distribution $\epsilon_{\theta}$, and even affect the migration direction of the generated distribution at that time node as illustrated in Figure 4. The impact caused by conditions $c$ or the input structure image $x_{0}$ would be reiterated at each sampling instance, and the minor changes introduced by filters would be involved in the entire generation process.

We observe that after applying filtering operations to a specific feature distribution in pixel space, the degree of expression for that distribution aligns with expectations. Moreover, the filtering operations targeting a specific feature distribution do not affect the expression of other features. This indicates that the influence of filters is independent in the feature space from the encoding of other representations. Therefore, prompts in pixel space offer a lightweight and convenient way to optimize the entanglement between feature representations in the diffusion model.

Based on the framework of combining ControlNet [58] and IP-Adapter [56], we examine the impact of filters at various stages of the sampling inference process and elucidate that the filter plays a guiding role in guiding the Gaussian distribution of the current data toward the target distribution during the migration and diffusion process.

We perform a detailed analysis of the sampling inference stage for the task of converting a bronze sketch to a photo, as shown in Figure 3. The sampled results vividly illustrate that detailed representations of arc patterns initially present in the early stages are weakened by the Gaussian filter, manifesting as point-like distributions. Besides, the Gaussian filter disrupts the continuous semantic expression of patterns. Concurrently, subsequent texture generation doesn’t emerge the negative impact of full-graph blurring, so it illustrates the property that the filtering operation is only effective for specific feature distributions.

The third row of the evolution sequence in Figure 3 illustrates the impact of applying the Gaussian filter only in the first 12 steps, where we observe the absence of redundant pattern features in the final generation result. A set of comparisons in lines 4 and 5 also showcase the effectiveness. In the first 6 steps, only ControlNet is utilized to regulate the structural layout. Introducing IP-adapter in the 7th step guides appearance features. The final results reveal that, even if the Gaussian filter does not initially suppress redundant features, it remains effective in later stages during the generation of detailed textures. The above findings highlight that the impact of filtering on the diffusion process is intuitive, controllable, and predictable.

We pay attention to the fact that in the sampling inference stage, the diffusion model has different dynamic evolution properties from the traditional generative model. In the basic theory of the diffusion model, the process of generating an image starts from a simple noise image [20], and through multiple iterations, the noise in the image is gradually removed until a final clear image is generated. From this perspective, the diffusion model generation process is continuous in time, and the image is generated through a gradually changing process. Therefore, we believe that the gradual evolution of the diffusion model makes it a dynamic generative model.

In the early days, the fast style transfer algorithm [15, 28, 16] is based on CNN, AE, and GAN [10, 61, 55], and their fitting effect on the target data distribution depends on the design of the training process. During the training process, the model attempts to capture the statistical characteristics of the entire data distribution and obtain the maximum likelihood representation of the data distribution. When training is completed, the generated samples exist statically in the latent space. Therefore, we consider this type of model to be the static generative model.

We noticed that prompts in pixel space may exhibit certain limitations in previous static generation models, but it can be well combined with dynamic generation models. As shown in Figure 4, we applied the same level of blur interference to the input data of various generative models. The results indicate that the impact of blur prompt on static generative models is relatively weak, while the outcomes of diffusion models show significant changes. We believe that for static generation models, using prompt on input data during the sampling inference stage will only have a slight impact on the mapping position of this input in the static distribution. And this position is close to the mapping result of the original image $x$. Therefore, prompts in pixel space will not significantly affect the generation results of static generation models. However, for diffusion models, as indicated by Equation 1, there is a predefined long-term dependency between the current time node image $x_{t}$ and the input image $x_{0}$, in the forward process. Therefore, the impact of using prompts on the input structure image $x_{0}$ will be executed again at each sampling, and the minor changes caused by prompts will participate in the entire generation process, thereby exerting a more significant influence on the generation results.

We leverage IP-Adapter[56] to obtain conditional encoding $C_{s}$, which controls rendering attributes in the generated image from the appearance reference $x_{app}$. Concurrently, we employ ControlNet[58] to obtain the conditional encoding $C_{c}$, responsible for regulating the geometric attributes from the structure reference image $x_{struct}$. These encodings, $C_{s}$ and $C_{c}$, represent features from their respective reference images:

	$\displaystyle C_{s}=\left\{feat_{\{m\}(x_{\{app\}})}\right\}_{\{m=1\}}^{\{M\}}$		(3)
	$\displaystyle C_{c}=\left\{feat_{\{n\}(x_{\{struct\}})}\right\}_{\{n=1\}}^{\{N\}}$

However, since $C_{s}$ and $C_{c}$ features are not fully disentangled, using them directly in latent space may cause structural conflicts. Drawing upon the successful performance of prompts demonstrated in the dynamic generation model above, we try to alleviate these conflicts by leveraging FilterPrompt, which adjusts feature frequencies or distributions. For instance, a color-removing prompt $F$ removes color features $feat$, modifying the encodings as follows:

	$\displaystyle C_{s}^{\prime}=\left\{feat_{\{m\}(F(x_{\{app\}}))}\right\}_{\{m=1\}}^{(M-k)}$		(4)
	$\displaystyle C_{c}^{\prime}=\left\{feat_{\{n\}(F(x_{(struct)}))}\right\}_{\{n=1\}}^{\{N-k\}}$

The pixel range affected by $F$ is intuitive and convenient, so we can estimate the impact of it on the migration of Gaussian distribution in Equation 2 and change the iterative process into:

$$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}\left(x_{t},c_{\{s\}}^{\prime},c_{\{c\}}^{\prime},t\right)\right)+\sigma_{t}z$$

(5)

In conclusion, we advocate for the adoption of our innovative approach, FilterPrompt, which directly manipulates the frequency or distribution characteristics of specific image attributes, thereby influencing the subsequent expression levels of the representation.

Our framework are built based on combined filtering operations, ControlNet [58] and IP-Adapter [56], as shown in Figure 5. Subsequently, we map the low-level features in the appearance image to a global embedding $C_{s}$ and concatenate it with the default text prompt embedding $C_{t}$ of SDM. This process can be described as $X_{t}=C_{t}\bigoplus C_{s}$. These two parts in hidden state $x_{t}$ are processed separately at each denoising generation. A portion of $x_{t}$ is delegated to ControlNet, which projects the latent distribution into a fused distribution controlled by high-level features $C_{c}$. The global embedding of another part in $x_{t}$ utilizes the IP-Adapter for decoding, unfolding, and guiding the generation of low-level features. We use $x_{t-1}$ to represent the hidden state predicted at the next moment in Equation 2. The intermediate hidden states obtained from both processes are weighted and summed according to Equation 6, achieving the effect of unifying representations related to Structure and Appearance into the latent space of SDM as shown in Equation 6.

$$X_{t-1}=\alpha\cdot ControlNet+\beta\cdot\lambda\cdot IP-Adapter$$

(6)

where $\alpha$, $\beta$ are weight control factors and $\lambda$ is scale control factor.

We define structure as the geometric features in the structure image and appearance as the rendering features from the reference image’s color and texture. We then assess baseline performance on these features before and after applying FilterPrompt (see Figure 6).

Firstly, we apply our approach on the ControlNet path for controlling structural details. The comparison of results among existing image preprocessing approaches and FilterPrompt indicates that our approach retains more details in the generated results and brings the colors closer to the target appearance. The combination of operations (including the ITV-R 601-2 luma transform method for decolorization, inversion, and contrast enhancement of the grayscale image) is succinctly referred to as $FilterPrompt_{struct}$. As illustrated in Figure 6-FilterPrompt(4), $FilterPrompt_{struct}$ best preserves structural information by enhancing brightness for clearer outlines, demonstrating FilterPrompt’s clarity and interpretability.

We then analyze the IP-adapter path, responsible for appearance, before and after applying FilterPrompt. Figure 6 shows that applying noise-processing filters to the appearance image affects the generated stroke details, aligning with the filter’s effects—e.g., Sharpen enhances fine strokes, while Gaussian blurs them. This verifies FilterPrompt’s capacity to control appearance.

Taking a step further, we explore the control effect of FilterPrompt on both paths in the baseline. In appearance task as shown in Figure 7), we aim is to transfer a bronze sketch to a photo with a specified appearance image without altering geometric features from the structure image. For structure control, $FilterPrompt_{struct}$ in the ControlNet path was used, but initial results showed redundant patterns. Applying a Gaussian filter in the IP-Adapter path suppressed high-frequency noise. When the Gaussian kernel increased to 4, redundant features diminished significantly.

Our quantitative analysis covers six specific appearance transfer: cat to cat, cat to dog, cat to wild, bird to bird, airplane to bird, and car to car. Following previous works [44, 30, 13, 1], we selected six metrics for our experiment. To evaluate the retention of geometric and semantic features from structure images in the generated images, we use three primary indicators: Structure Preservation (SP), Chamfer Distance (CD), and Fréchet Inception Distance (FID). Additionally, to assess the fidelity of low-level features between appearance images and generated images, we employ three specific metrics: Gray-Level Co-occurrence Matrix (GLCM), Peak Signal-to-Noise Ratio (PSNR), and Color Histogram Correlation (CHC).

• •

  Structure Preservation (SP): we utilize the marquee interaction mode of SAM [25] for selecting areas to obtain binary masks corresponding to structure images and their respective output images. Then, we compute their Intersection over Union (IoU) results as a measure of Structure Preservation.

• •

  Chamfer Distance (CD): we first extract the line drawings of the structure and generated images, and then filter out redundant details using the Canny operator. The high and low thresholds used by the Canny operator are set to 150 and 50, respectively. Finally, we calculate the chamfer distance between the line drawings as a measure of the gap between the sets of edge points in the two images. A smaller value indicates a higher degree of match between the shape or edge features in the structure and generated images.

• •

  Fréchet Inception Distance (FID): we calculate the FID score between the structure image and the generated image to quantify the extent to which the two images align in terms of their structural features.

• •

  Gray-Level Co-occurrence Matrix (GLCM): it is used to calculate the loss value of texture features between the appearance image and the generated image.

• •

  Peak Signal-to-Noise Ratio (PSNR): it is used to measure how well the generated image preserves the low-level features of appearance.

• •

  Color Histogram Correlation (CHC): it is used to calculate the color similarity between the generated image and the appearance image. Among them, we use mask to cover the background of the image.

Test Datasets: AFHQ [10], CUB-200-2011 [49], FGVC-Airaft [35], Stanford-Cars [26]. Among them, the three domain data of cat, dog, and wild are all from the AFHQ test set. We followed the setting of AFHQ’s test set, with 500 images for each category, and randomly selected 500 images from three other datasets as appearance images. For every types of appearance transfer tasks there are 2000 pairs, Therefore, the data in Table 5 is based on the evaluation results of 12000 Structure-Appearance image pairs. We show examples of the comparison results in Figure 8.

The qualitative analysis experiments include a total of five domains (cat, dog, wild, bird, bronze). In addition to the datasets used in the quantitative analysis experiments as shown in Figure 9, additional data is: Bronze Dings [60]. In this task, the appearance reference and the structure image do not have semantic correspondence, and their relationships belong to different domains. So the focus in this task is to obtain the low-level texture features from the appearance image without semantic correspondence and then render it to the structure image.

We conducted a user study with 18 questions evaluating generated results in structure preservation, rendering feature transfer effectiveness, and overall quality. Each participant was paid 0.5$. To address the common issue of low-quality feedback caused by participants’ lack of understanding of task, we had eight facilitators provide detailed background information to participants on campus. Participants were then asked to select the most fitting options from anonymous choices based on their preferences. Finally, we received 215 valid survey submissions, with ours garnering a support rate of 51. 89% (cross-image 24. 00%, IP-Adapter 14. 98%, ControlNet + IP-Adapter 9. 13%).