Not All Noises Are Created Equally:
Diffusion Noise Selection and Optimization

Generative diffusion models, renowned for the impressive performance (Dhariwal and Nichol,, 2021), serve as the mainstream generative paradigm with wide applications in image generation (Nichol et al.,, 2021; Zhang et al.,, 2023; Saharia et al.,, 2022), image editing (Qi et al.,, 2023; Kawar et al.,, 2023), 3D generation (Gupta et al.,, 2023; Erkoç et al.,, 2023), and video generation (Ho et al., 2022a, ; Ho et al., 2022c, ). Diffusion-based Generative AI products attracted much attention and a large number users in recent years. Understanding and improving the capabilities of diffusion models has become an essentially important topic in machine learning.

A large body of works (Song et al.,, 2020; Fang et al.,, 2023; Podell et al.,, 2023; Sauer et al.,, 2023; Ho et al., 2022b, ; Lin et al.,, 2024) tried to enhance the generated results by working on model weight and architecture space. The importance of noise space is largely overlooked by previous studies, while it is known that diffusion models can generate diverse results, which, of course, contain good ones and bad ones. However, existing works suggest random noises are equal and fail to explore how the noises affect the quality of generated results.

In this work, we try to visit two fundamental issues on noise space of diffusion models. First, is it possible to select better noise according to some quantitative metric? Second, is it possible to optimize a given noise (rather than the model weights) to generate better results? Both answers are affirmative. Fortunately, we not only confirm the possibility but also propose practical algorithms.

Contributions. We summarize the three main contributions of this work as follows:

First, we are the first to hypothesize and empirically verify that not all noises are created equally. Specifically, random noises with high inversion stability usually lead to better generation than noises with lower inversion stability, where the inversion stability can be quantitatively given by the cosine similarity of the sampled noise $\epsilon$ and the inverse noise $\epsilon^{\prime}$. This quantitative metric naturally provides us with a novel noise selection method to select stable noises (e.g. the seed with the highest inversion stability among 100 noise seeds), which often correspond to better generated results. We present several qualitative results of noise selection in Figures 1 and 4.

Second, we further proposed a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. More specifically, we optimize an inversion-stability loss via gradient descent with respect to the sampled noise (rather than the convention model weight space). The proposed noise optimization method is the first one that works on noise space rather than model weight space to improve diffusion models. We present several qualitative results of noise optimization in Figures 1 and 6.

Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo. On one hand, the human preference winning rates of noise selection and noise optimization over the baseline can be up to $57\%$ and $72.5\%$, respectively, on DrawBench in terms of human preference. On the other hand, noise selection and noise optimization are also preferred by Human Preference Score (HPS) v2 (Wu et al., 2023b, ), the latest state-of-the-art human preference model trained on diverse high-quality human preference data, with winning rates up to $67\%$ and $88\%$, respectively. Human preference, regarded as the ground-truth ultimate evaluation metric for text-to-image generation, and objective evaluation metrics generally support our methods.

In this section, we formally introduce prerequisites and notations.

Notations. Suppose a diffusion model $\mathcal{M}$ can generate a clean sample $x_{0}$ based on some condition $c$, such as a text prompt, given a sampled random noise $\epsilon$.¹¹1For simplicity, we abuse the latent space and the original data space in the presence of latent diffusion. We denote the score neural network as $u_{\theta}(x_{t},t)$, the model weights as $\theta$, the noisy sample at the $t$-th step as $x_{t}$, and $T$ as the total number of denoising steps.

Diffusion Models. The diffusion models (Ho et al.,, 2020) typically denoise a Gaussian noise along a reverse diffusion path (steps: $T\rightarrow 1$) to generate an image step by step. The probability via $u_{\theta}$, denoted as $p_{\theta}$, represents the sampling probability given the previous step’s data. The starting point sampled from a Gaussian distribution, $p(x_{T})=\mathcal{N}(x_{T}|\mathbf{0},\mathbf{I})$. The probability of the whole chain, $p_{\theta}(x_{0:T})$, is shown as follows:

where $a_{t}=1-\beta_{t}$, $\overline{\alpha}_{t}=\prod^{s}_{t=1}\alpha_{s}$. The $\beta_{t}$ and $\sigma_{t}$ are the pre-defined parameters for scheduling the scales of adding noises. The $\epsilon_{t}$ is an additional sampling noise at the $t$-th step.

Noise Inversion. The noise inversion is to invert a clean data into a noise along a pre-defined diffusion path. We can write the DDIM inversion process (Hertz et al.,, 2022) as

where people approximate the denoising score prediction at $x_{t}$ with the inversion score prediction at $x_{t-1}$. Equation (1) can gradually transform a sampled noise $\epsilon$ into a generated sample $x_{0}$ along the denoising path, and Equation (2) can gradually transform a generated sample $x_{0}$ back to a noise $\epsilon^{\prime}$ along the noise inversion path. We note that the standard noising path which adds independent Gaussian noises is essentially different from the noise inversion path which adds the predicted noise of the score neural network $u_{\theta}$. While the generation denoising path and the noise inversion path are both guided by the score neural network $u_{\theta}$, the sampled noise $\epsilon$ and the inverse noise $\epsilon^{\prime}$ are close but not identical due to the cumulative numerical differences.

Fixed Points. We denote the denoising-inversion transformation, $\epsilon\to x_{0}\to\epsilon^{\prime}$, as the transformation function $\epsilon^{\prime}=F(\epsilon)$. If $\epsilon$ and $\epsilon^{\prime}$ are ideally identical, namely $\epsilon=F(\epsilon)$, we call $\epsilon$ a fixed point of this mapping function $F$. In this case, the inverse noise $\epsilon^{\prime}$ can perfectly recover the sample $x_{0}$ generated from $\epsilon$. This suggests that a state can remain fixed under some transformation. The fixed points have various great properties and many important applications in various fields, such as projective geometry (Coxeter,, 1998), Nash Equilibrium (Nash Jr,, 1950), and Phase Transition (Wilson,, 1971).

In this section, we first introduce the noise inversion stability hypothesis and show how it naturally leads to two novel noise-space algorithms, including noise selection and noise optimization.

Noise Inversion Stability. It is well known that fixed points are stable under the transformation and, thus, have great properties (Burton,, 2003; Connell,, 1959; Pata et al.,, 2019). May the fixed-point Gaussian noises under the denoising-inversion transformation $F$ also exhibit some advantages? As finding the fixed points of this complex dynamical system is intractable, unfortunately, we cannot empirically verify it. Instead, we can formulate Definition 1 to measure the stability of noise for the denoising-inversion transformation $F$.

We use cosine similarity to measure stability for simplicity, while it is also possible to use other similarity metrics. Our empirical analysis in Section 4 suggests that the simple cosine similarity metric works well.

Noise Selection. Inspired by the intriguing mathematical properties of fixed points, we hypothesize that the noise with higher inversion stability can lead to better generated results. If this hypothesis is reasonable, this naturally provides a novel and useful noise selection algorithm that selects the noise seed with the highest stability score from $K$ noise seeds (e.g. $K=100$ in this work). We present the pseudocode in Algorithm 1.

Noise Optimization. As we have an objective to increase the noise inversion stability, is it possible to actively optimize a given noise by maximizing the stability score? We further propose the noise optimization algorithm that directly performs Gradient Descent (GD) on the loss, $1-\cos(\epsilon,\epsilon^{\prime})$, with respect to $\epsilon$, where we keep the diffusion model weights and $\epsilon^{\prime}$ constant for each optimization step. We present the illustration of noise optimization in the right column of Figure 2. We present the pseudocode in Algorithm 2.

In this section, we conduct extensive experiments to demonstrate the effectiveness of our methods. We take text-to-image generation as our main setting.

In this section, we further discuss related works and the limitations of this work. While this work reports very interesting findings and proposes novel algorithms on noise space of diffusion models, it still has several limitations.

Related Work The noise inversion technique is mainly applied on image editing (Mokady et al.,, 2023; Meiri et al.,, 2023; Huberman-Spiegelglas et al.,, 2023) in very similar ways. They usually invert a clean image into a relatively noisy one via a few inverse steps and then denoise the inverse noisy images with another prompt to achieve instruction editing. Some works (Mao et al.,, 2023) in this line of research realized that editing noises can help editing generated results. Specifically, modifying a portion of the initial noise can affect the layout of the generated images. Other works (Liu et al.,, 2024; Shi et al.,, 2023) focused on dragging and dropping image content via interactive noise editing. However, the goal of previous studies is to control image layout under fine-grained control conditions, such as input layout or editing operations. In contrast, we focus on generally improving generated results of diffusion models by selecting or optimized a Gaussian noise according to the stability score.

Theoretical Understanding. With the inspirations from fixed points in dynamical systems, we still do not theoretically understand why not all noises are created for diffusion models. We formulated and empirically verified the hypothesis that random noises with higher inversion stability often lead to better generated results, it is still difficult to theoretically analyze how the performance of diffusion models mathematically depend on noise stability. We believe theoretically understanding noise selection and noise optimization will be a key step to further improve them.

Optimization Strategies. In this, we only applied simple gradient descent with multiple (e.g., $100$) steps to optimize the noise-space loss, but noise optimization seems like a difficult optimization task. In some cases, we observe that the loss does not converge smoothly. Due to computational costs and poor understanding towards noise-space loss landscape, we did not carefully fine-tune the hyperparameters or employ advanced optimizers, such as Adam (Kingma and Ba,, 2015) in this work. Thus, our current optimization strategy is far from releasing the power of noise-space algorithms. We think it will be very promising and important to better analyze and solve this emerging optimization task with advanced methods.

Computational Costs. Both noise selection and noise optimization require significantly more computational costs and time than the standard generation. For noise selection, we compute the inversion stability loss of 100 noise seeds and select the one with the highest stability score. Thus, we need to repeat the forward and inversion process for 100 times. For noise optimization, we perform gradient descent with 100 steps. Thus, we need to repeat the forward, inversion and gradient computing process each for 100 times. It may be difficult to specifically accelerate each step of noise selection, but it will be very likely to accelerate noise optimization with less GD steps in near future.

In this paper, we report an interesting noise inversion stability hypothesis and empirically observe that noises with higher inversion stability often lead to better generated results. This hypothesis motivates us to design two novel noise-space algorithms, noise selection and noise optimization, for diffusion models. To the best of our knowledge, we are the first to report that not all noises are created equally for diffusion models and the first to generally improve diffusion models without fine-tune the model parameters. Our extensive experiments demonstrate that the proposed methods can significantly improve multiple aspects of qualitative results and enhance human preference rates as well as objective evaluation scores. Moreover, the proposed methods can be generally applied to various diffusion models in a plug-and-play manner. While some limitations exist, our work has made the first solid step to explore this promising direction. We believe our work will motivate more studies on understanding and improving diffusion models from the perspective of noise space.