Gaussian Interpolation Flows

Generative modeling, which aims to learn the underlying data generating distribution from a finite sample, is a fundamental task in the field of machine learning and statistics (Salakhutdinov, 2015). Deep generative models (DGMs) find wide-ranging applications across diverse domains such as computer vision, natural language processing, drug discovery, and recommendation systems. The core objective of DGMs is to learn a nonlinear mapping, either deterministic or stochastic (with outsourcing randomness), which transforms latent samples drawn from a simple reference distribution into samples that closely resemble the target distribution.

Generative adversarial networks (GANs) have emerged as a prominent class of DGMs (Goodfellow et al., 2014; Arjovsky et al., 2017; Goodfellow et al., 2020). Through an adversarial training process, GANs learn to approximately generate samples from the data distribution. Variational auto-encoders (VAEs) are another category of DGMs (Kingma and Welling, 2014; Rezende et al., 2014; Kingma and Welling, 2019). In VAEs, the encoding and decoding procedures produce a compressed and structured latent representation, enabling efficient sampling and interpolation. Score-based diffusion models are a promising approach to deep generative modeling that has evolved rapidly since its emergence (Song and Ermon, 2019, 2020; Ho et al., 2020; Song et al., 2021b, a). The basis of score-based diffusion models lies in the notion of the score function, which characterizes the gradient of the log-density function of a given distribution.

In addition, normalizing flows have gained attention as another powerful class of DGMs (Tabak and Vanden-Eijnden, 2010; Tabak and Turner, 2013; Kobyzev et al., 2020; Papamakarios et al., 2021). In normalizing flows, an invertible mapping is learned to transform a simple source distribution into a more complex target distribution by a composition of a series of parameterized, invertible and differentiable intermediate transformations. This framework allows for efficient sampling and training by maximum likelihood estimation (Dinh et al., 2014; Rezende and Mohamed, 2015). Continuous normalizing flows (CNFs) pursue this idea further by performing the transformation over continuous time, enabling fine-grained modeling of dynamic systems from the source distribution to the target distribution. The essence of CNFs lies in defining ordinary differential equations (ODEs) that govern the evolution of CNFs in terms of continuous trajectories. Inspired by the Gaussian denoising approach, which learns a target distribution by denoising its Gaussian smoothed counterpart, many authors have considered simulation-free estimation methods that have shown great potential in large-scale applications (Song et al., 2021a; Liu et al., 2023; Albergo and Vanden-Eijnden, 2023; Lipman et al., 2023; Neklyudov et al., 2023; Tong et al., 2023; Chen and Lipman, 2023; Albergo et al., 2023b; Shaul et al., 2023; Pooladian et al., 2023). However, despite the empirical success of simulation-free CNFs based on Gaussian denoising, rigorous theoretical analysis of these CNFs have received limited attention thus far.

In this work, we explore an ODE flow-based approach for generative modeling, which we refer to as Gaussian Interpolation Flows (GIFs). This method is derived from the Gaussian stochastic interpolation detailed in Section 3. GIFs represent a straightforward extension of the stochastic interpolation method (Albergo and Vanden-Eijnden, 2023; Liu et al., 2023; Lipman et al., 2023). They can be considered a class of CNFs and encompass various ODE flows as special cases. According to the classical Cauchy-Lipschitz theorem, also known as the Picard-Lindelöf theorem (Hartman, 2002b, Theorem 1.1), a unique solution to the initial value problem for an ODE flow exists if the velocity field is continuous in the time variable and uniformly Lipschitz continuous in the space variable. In the case of GIFs, the velocity field depends on the score function of the push-forward measure. Therefore, it remains to be shown that this velocity field satisfies the regularity conditions stipulated by the Cauchy-Lipschitz theorem. These regularity conditions are commonly assumed in the literature when analyzing the convergence properties of CNFs or general neural ODEs (Chen et al., 2018; Biloš et al., 2021; Marion et al., 2023; Marion, 2023; Marzouk et al., 2023). However, there is a theoretical gap in understanding how to translate these regularity conditions on velocity fields into conditions on target distributions.

The main focus of this work is to study and establish the theoretical properties of Gaussian interpolation flow and its corresponding flow map. We show that the regularity conditions of the Cauchy-Lipschitz theorem are satisfied for several rich classes of probability distributions using variance inequalities. Based on the obtained regularity results, we further expose the well-posedness of GIFs, the Lipschitz continuity of flow mappings, and applications to generative modeling. The well-posedness results are crucial for studying the approximation and convergence properties of GIFs learned with the flow or score matching method. When applied to generative modeling, our results further elucidate the auto-encoding and cycle consistency properties exhibited by GIFs.

In this section, we include several preliminary setups to show notations, basic assumptions, and several useful variance inequalities.

The Borel $\sigma$-algebra of ${\mathbb{R}}^{d}$ is denoted by $\mathcal{B}({\mathbb{R}}^{d})$. The space of probability measures defined on $({\mathbb{R}}^{d},\mathcal{B}({\mathbb{R}}^{d}))$ is denoted as $\mathcal{P}({\mathbb{R}}^{d})$. For any ${\mathbb{R}}^{d}$-valued random variable $\mathsf{X}$, we use $\mathbb{E}[\mathsf{X}]$ and $\mathrm{Cov}(\mathsf{X})$ to denote its expectation and covariance matrix, respectively. We use $\mu*\nu$ to denote the convolution for any two probability measures $\mu$ and $\nu$, and we use $\overset{d}{=}$ to indicate two random variables have the same probability distribution. For a random variable $\mathsf{X}$, let $\mathrm{Law}(\mathsf{X})$ denote its probability distribution. Let $g:{\mathbb{R}}^{k}\to{\mathbb{R}}^{d}$ be a measurable mapping and $\mu$ be a probability measure on ${\mathbb{R}}^{k}$. The push-forward measure $f_{\#}\mu$ of a measurable set $A$ is defined as $f_{\#}\mu:=\mu(f^{-1}(A))$. Let $N(m,\Sigma)$ denote a $d$-dimensional Gaussian random variable with mean vector $m\in{\mathbb{R}}^{d}$ and covariance matrix $\Sigma\in{\mathbb{R}}^{d\times d}$. For simplicity, let $\gamma_{d,\sigma^{2}}:=N(0,\sigma^{2}{\mathbf{I}}_{d})$, and let $\varphi_{m,\sigma^{2}}(x)$ denote the probability density function of $N(m,\sigma^{2}{\mathbf{I}}_{d})$ with respect to the Lebesgue measure. If $m=0,\sigma=1$, we abbreviate these as $\gamma_{d}$ and $\varphi(x)$. Let $L^{p}({\mathbb{R}}^{d};{\mathbb{R}}^{\ell},\mu)$ denote the $L^{p}$ space with the $L^{p}$ norm for $p\in[1,\infty]$ w.r.t. a measure $\mu$. To simplify the notation, we write $L^{p}({\mathbb{R}}^{d},\mu)$ if $\ell=1$, $L^{p}({\mathbb{R}}^{d};{\mathbb{R}}^{\ell})$ if the Lebesgue measure is used, and $L^{p}({\mathbb{R}}^{d})$ if both hold.

Simulation-free CNFs represent a potent class of generative models based on ODE flows. Albergo and Vanden-Eijnden (2023) and Albergo et al. (2023b) introduce an innovative CNF that is constructed using stochastic interpolation techniques, such as Gaussian denoising. They conduct a thorough investigation of this flow, particularly examining its applications and effectiveness in generative modeling.

We study the ODE flow and its associated flow map as defined by the Gaussian denoising process. This process has been explored from various perspectives, including diffusion models and stochastic interpolants. Building upon the work of Albergo and Vanden-Eijnden (2023) and Albergo et al. (2023b), we expand the stochastic interpolant framework by relaxing certain conditions on the functions $a_{t}$ and $b_{t}$, offering a more comprehensive perspective on the Gaussian denoising process.

In our generalization, we introduce an adaptive starting point to the stochastic interpolation framework, which allows for greater flexibility in the modeling process. By examining this modified framework, we aim to demonstrate that the Gaussian denoising principle is effectively implemented within the context of stochastic interpolation.

A natural generalization of the vector interpolant (3.1) is to construct a set interpolant between two convex sets through Minkowski sum, which is common in convex geometry. A set interpolant stimulates the construction of a measure interpolant between a structured source measure and a target measure.

As noted, we can construct a measure interpolation using a Gaussian convolution path. The measure interpolation is particularly relevant to Gaussian denoising and Gaussian channels in information theory as elucidated in Remark 16. Because of this connection with Gaussian denoising, we call the measure interpolation a Gaussian stochastic interpolation. The Gaussian stochastic interpolation can be understood as a collection of linear combinations of a standard Gaussian random variable and the target random variable. The coefficients of the linear combinations vary with time $t\in[0,1]$ as shown in Definition 6. Later in this section, we will show this Gaussian stochastic interpolation can be transformed into a deterministic ODE flow.

Gaussian stochastic interpolation has been investigated from several perspectives in the literature. The rectified flow has been proposed in Liu et al. (2023), and its theoretical connection with optimal transport has been investigated in Liu (2022). The formulation of the rectified flow is to learn the ODE flow defined by stochastic interpolation with linear time coefficients. In Section 2.3 of Liu et al. (2023), there is a nonlinear extension of the rectified flow in which the linear coefficients are replaced by general nonlinear coefficients. Albergo et al. (2023b) extends the stochastic interpolant framework proposed in (Albergo and Vanden-Eijnden, 2023) by considering a linear combination among three random variables. In Section 3 of Albergo et al. (2023b), the original stochastic interpolant framework is recovered as a one-sided interpolant between the Gaussian distribution and the target distribution. Moreover, Lipman et al. (2023) propose a flow matching method which directly learns a Gaussian conditional probability path with a neural ODE. In Section 4.1 of (Lipman et al., 2023), the velocity fields of the variance exploding and variance preserving probability flows are shown as special instances of the flow matching framework. We summarize these formulations as Gaussian stochastic interpolation by slightly extending the original stochastic interpolant framework.

Motivated by the time-varying properties of the Gaussian stochastic interpolation, we derive that its marginal flow satisfies the continuity equation. This result characterizes the dynamics of the marginal density flow of the Gaussian stochastic interpolation.

We are now ready to define Gaussian interpolation flows by representing the continuity equation (3.4) with Lagrangian coordinates (Ambrosio and Crippa, 2014). A basic observation is that GIFs share the same marginal density flow with Gaussian stochastic interpolations. The continuity equation (3.4) plays a central role in the derandomization procedure from Gaussian stochastic interpolations to GIFs. We additionally illustrate GIFs using a two-dimensional example as in Figure 2.

We have explicated the idea of Gaussian denoising with the procedure of Gaussian stochastic interpolation or a Gaussian channel with increasing SNR w.r.t. time. By interpreting the process as an ODE flow, we derive the framework of Gaussian interpolation flows. First and foremost, an intuition is that the regularizing effect of Gaussian denoising would ensure the Lipschitz smoothness of the velocity field. Since the standard Gaussian distribution is both $1$-semi-log-concave and $1$-semi-log-convex, its convolution with a target distribution will maintain its high regularity as long as the target distribution satisfies the regularity conditions. We rigorously justify this intuition by establishing spatial Lipschitz estimates for the velocity field. These estimates are established based on the upper bounds and lower bounds regarding the Jacobian matrix of the velocity field $v(t,x)$ according to the Cauchy-Lipschitz theorem, which are given in Proposition 20 below. To deal with the Jacobian matrix $\nabla_{x}v(t,x)$, we introduce a covariance expression of it and present the associated upper bounds and lower bounds.

The velocity field $v(t,x)$ is decomposed into a linear term and a nonlinear term, the score function $s(t,x)$. To analyze the Jacobian $\nabla_{x}v(t,x)$, we only need to focus on $\nabla_{x}s(t,x)$, that is, $\nabla^{2}_{x}\log p_{t}(x)$. To ease the notation, we would henceforth use $\mathsf{Y}$ for $\mathsf{X}_{1}$. Correspondingly, we replace $p_{1}(x)$ with $p_{1}(y)$ for the density function of $\mathsf{Y}$.

According to Bayes’ theorem, the marginal density $p_{t}$ of $\mathsf{X}_{t}$ satisfies

$$p_{t}(x)=\int p(t,x|y)p_{1}(y)\mathrm{d}y$$

where $\mathsf{Y}\sim p_{1}(y)$ and $p(t,x|y)=\varphi_{b_{t}y,a_{t}^{2}}(x)$ is a conditional distribution induced by the Gaussian noise. Due to the factorization $p_{t}(x)p(y|t,x)=p(t,x|y)p_{1}(y)$, the score function $s(t,x)$ and its derivative $\nabla_{x}s(t,x)$ have the following expressions

$\displaystyle s(t,x)=-\nabla_{x}\log p(y|t,x)-\tfrac{x-b_{t}y}{a_{t}^{2}},\quad\nabla_{x}s(t,x)=-\nabla^{2}_{x}\log p(y|t,x)-\tfrac{1}{a_{t}^{2}}{\mathbf{I}}_{d}.$

Thanks to the expressions above, a covariance matrix expression of $\nabla_{x}s(t,x)$ is endowed by the exponential family property of $p(y|t,x)$.

The representation (4.3) can be used to upper bound and lower bound $\nabla_{x}v(t,x)$. This technique has been widely used to deduce the regularity of the score function concerning the space variable (Mikulincer and Shenfeld, 2021, 2023; Chen et al., 2023b; Lee et al., 2023; Chen et al., 2023a). The covariance matrix expression (4.2) of the score function has a close connection with the Hatsell-Nolte identity in information theory (Hatsell and Nolte, 1971; Palomar and Verdú, 2005; Wu and Verdú, 2011; Cai and Wu, 2014; Wibisono et al., 2017; Wibisono and Jog, 2018a, b; Dytso et al., 2023a, b).

Employing the covariance expression in Lemma 18, we establish several bounds on $\nabla_{x}v(t,x)$ in the following proposition.

Comparing part (a) with part (d) in Proposition 20, we can see that the bounds in (a) are consistent with those in (d) in the sense that (a) is a limiting case of part (d) as $\sigma\to 0$. The lower bound in part (a) blows up at time $t=1$ owing to $a_{1}=0,$ while in part (d) it behaves well since the lower bound in part (d) coincides with a lower bound indicated by the $\tfrac{1}{\sigma^{2}}$-semi-log-convex property. It reveals that the regularity of the velocity field $v(t,x)$ with respect to the space variable $x$ improves when the target random variable is bounded and is subject to Gaussian perturbation.

The lower bound in part (b) and the upper bound in part (c) are tight in the sense that both of them are attainable for a Gaussian target distribution, that is,

$$\nabla_{x}v(t,x)=\frac{\beta a_{t}\dot{a}_{t}+b_{t}\dot{b}_{t}}{\beta a_{t}^{2}+b_{t}^{2}}{\mathbf{I}}_{d}\quad\text{if $\nu=\gamma_{d,1/\beta}$.}$$

The upper and lower bounds in Proposition 20-(a) and (e) become vacuous as they both blow up at time $t=1$. The intuition behind is that the Jacobian matrix of the velocity field can be both lower and upper bounded at time $t=1$ only if the score function of the target measure is Lipschitz continuous in the space variable $x$. Under an additional Lipschitz score assumption (equivalently, $\beta$-semi-log-convex and $\kappa$-semi-log-concave for some $\beta=-\kappa\geq 0$), the upper and lower bounds in part (a) and part (e) can be strengthened at time $t=1$ based on the lower bound in (b) and the upper bound in part (c).

According to Proposition 20-(a) and (c), there are two upper bounds available that shall be compared with each other. One is the $D^{2}$-based bound in part (a), and the other is the $\kappa$-based bound in part (c). According to the proof of Proposition 20 given in the Appendix, these two upper bounds are equal if and only if the corresponding upper bounds on $\mathrm{Cov}(\mathsf{Y}|\mathsf{X}_{t}=x)$ are equal, that is,

$$D^{2}=\left(\kappa+\frac{b_{t}^{2}}{a_{t}^{2}}\right)^{-1}.$$

(4.4)

Then the critical case is $\kappa D^{2}=1$ since simplifying Eq. (4.4) reveals that

$$D^{-2}-\kappa=\frac{b_{t}^{2}}{a_{t}^{2}}.$$

(4.5)

We note that $b_{t}^{2}/a_{t}^{2}$, ranging over $(0,\infty)$, is monotonically increasing w.r.t. $t\in(0,1)$. Suppose that $\kappa D^{2}>1$. Then (4.5) has no root over $t\in(0,1),$ which implies that the $\kappa$-based bound is tighter over $[0,1)$, i.e.,

$$D^{2}>\left(\kappa+\frac{b_{t}^{2}}{a_{t}^{2}}\right)^{-1},\quad\forall t\in[0,1).$$

Otherwise, suppose that $\kappa D^{2}<1$. Then (4.5) has a root $t_{1}\in(0,1),$ which implies that the $D^{2}$-based bound is tighter over $[0,t_{1})$, i.e.,

$\displaystyle D^{2}<\left(\kappa+\frac{b_{t}^{2}}{a_{t}^{2}}\right)^{-1},\quad\forall t\in[0,t_{1}),$

and that the $\kappa$-based bound is tighter over $[t_{1},1)$, i.e.,

$\displaystyle D^{2}\geq\left(\kappa+\frac{b_{t}^{2}}{a_{t}^{2}}\right)^{-1},\quad\forall t\in[t_{1},1).$

Next, we present several upper bounds on the maximum eigenvalue of the Jacobian matrix of the velocity field $\lambda_{\max}(\nabla_{x}v(t,x))$ and its exponential estimates for studying the Lipschitz regularity of the flow maps as noted in Lemma 29.