Gaussian Information Bottleneck and the Non-Perturbative Renormalization Group

A broad goal of this paper is to explore structural similarities between IB and RG. The semigroup structure present in Wilsonian RG is crucial to its explanation of scaling phenomena, so its presence in GIB is a promising sign. The traditional picture is this: consider the RG transformations $\mathcal{R}_{b_{1}}$ and $\mathcal{R}_{b_{2}}$ which rescale length by factors $b_{1}$ and $b_{2}$, respectively. Then a fundamental property of $\mathcal{R}$ is that $\mathcal{R}_{b_{1}}\mathcal{R}_{b_{2}}=\mathcal{R}_{b_{1}b_{2}}$. This structure imposes a strong constraint on the behavior of the flow near a fixed point. If $\sigma$ represents an eigenvector of the Jacobian matrix at the fixed point, then its associated eigenvalue $\lambda_{\sigma}$ will scale as $b^{y_{\sigma}}$[32]. In short, the semigroup typically allows one to define the critical exponent $y_{\sigma}$.

The operator $\circ$ we introduced does not immediately lend itself to this sort of analysis. However, we can introduce a function $b(\beta)$ which satisfies $b(\beta_{2}\circ\beta_{1})$ $=$ $b(\beta_{2})b(\beta_{1})$. By inspection, this function is given by:

This quantity is interesting because it is analogous to the length-rescaling factor found in typical Wilsonian or Kadanoff RG schemes, yet in IB there is no need for a notion of space, and hence rescaling length generally means nothing. Compare this with, for example, a momentum-shell decimation scheme. One identifies the rescaling factor by comparing the new and old UV cutoffs, and so it acquires the meaning of a length-rescaling factor. Here, $b$ is determined entirely by the Lagrange multiplier $\beta$ and the structure of GIB, both of which are defined without deference to an a priori existing notion of spatial extent. As discussed in the introduction, connecting IB to RG is attractive, in part, precisely because IB is an information-theoretic framework and does not rely on physical interpretations. Hence this rescaling factor $b$ should be considered an information-theoretic quantity in the same way as $\beta$.

Can $b(\beta)$ as defined above be used in the same way as the rescaling factor $b$ is used in RG? First, limits of the IB problem involving extremal values of $\beta$ should match intuition about $b$ in an RG context. Indeed, for $\beta\to\infty$, the zero-coarsening limit, $b(\beta)\to 1$. Next, by the data processing inequality, at $\beta=1$ the optimal IB solution is degenerate with complete coarsening, i.e. $\tilde{X}$ becomes independent of $X$. Correspondingly, the limit $\beta\to 1$ gives $b(\beta)\to\infty$. Next, let us recall the scope of the GIB framework. GIB makes statements only about completely Gaussian statistics, so no anomalous scaling will appear, and thus a discussion of critical exponents is hard to motivate. Second, GIB is defined for finite-dimensional $X$ and $Y$, so we cannot simply connect it to, say, momentum-shell Wilsonian RG, which only makes statements about infinite systems. Finally, and related to the last point, we have not identified yet what the analogous “model space” is in the context of IB, or how an optimal GIB map could represent an RG transformation in that space. This will be the subject of the next section, where we show that the non-deterministic nature of IB coarsening aligns exactly with existing soft-cutoff RG methods.

Whether or not this analysis helps to formally connect IB and RG, it is interesting to ask whether other IB problems exhibit semigroup structure. One could imagine, for example, that a series of high-$\beta$ compression steps (low-compression limit) might be easier than one large compression step. If this is the case, IB problems with semigroup structure may benefit from an iterative chaining scheme similar to the one we present here. One possible application of this structure is the construction of feed-forward neural networks with IB objectives. If the IB problem in question has semigroup structure, then the task of training the entire network can be reduced to training the layers one-by-one on smaller (higher-compression) IB problems. This has benefits in biological systems, such as biochemical and neural networks, where processing is often hierarchical, likely as a result of underlying evolutionary and developmental constraints. Biological systems are also shaped by their output behavior, which sets a natural relevance variable in the arc from sensation to action.

The renormalization group is not a single coherent framework, but rather a collection of theories, computational tools, and loosely-defined motifs. As such, it is probably not possible to succinctly define RG on the whole. A common theme, at least, is that RG techniques describe how the effective model of a given system changes as degrees of freedom are added or removed. The modern view of RG theory, which is largely due to Wilson [33, 34, 35, 36, 37] and Kadanoff [38], concerns itself with the removal of degrees of freedom through a process known as decimation, in which a thermodynamic quantity (typically the partition function) is re-written by performing a configurational sum or integral over a subset of the original modes. Here, even before discussing rescaling and renormalization, we must make procedural choices. To begin, one must specify the subset of degrees of freedom which are to be coarsened off. In theories where modes are labelled by wavenumber or momentum, one typically establishes a cutoff and decimates all modes with momentum above it. As a result, those modes are completely removed from the system description, and their statistics are incorporated into the couplings which parameterize the new effective theory. Another consideration is the practicality of carrying out such a procedure. If the model in consideration can be expanded in a perturbation series about a Gaussian model, and if the non-Gaussian operators are irrelevant or marginal under the flow, then this analysis is amenable to perturbative RG. However, this is often not the case, for example in systems far from their critical dimension, or in non-equilibrium phase transitions, where there may not even be critical dimensions [39, 40].

In non-perturbative RG (NPRG) approaches, the need for a perturbative treatment is removed by working from a formally exact flow equation at the outset. The first such treatment was put forth in 1973 by Wegner and Houghton, who used Wilson’s idea of an infinitesimal momentum-shell integration to derive an exact flow equation for the full coarse-grained Hamiltonian [41]. Because this equation describes the evolution of the Hamiltonian for every field configuration, this and other NPRG flow equations are called integro-differential equations, and the NPRG is sometimes referred to as the functional renormalization group (FRG). Later, Wilson and Kogut [35], as well as Polchinski [42], proposed new NPRG flow equations in which the cutoff was not described explicitly through a literal demarcation between included and excluded modes, but instead through non-deterministic coarsening, so that the effective Hamiltonian satisfies a functional generalization of a diffusion equation ³³3This interpretation is explicitly presented in the Wilson-Kogut paper. Given that their approach is formally equivalent with the one put forth by Polchinski, the interpretation should apply to that framework as well.. These approaches were introduced, at least in part, as a response to difficulties ⁴⁴4Under the sharp cutoff construction, some issues include the generation of non-local position-space interactions [35], unphysical nonanalyticities in correlation functions, and the need to evaluate ambiguous expressions such as $\delta(x)f(\Theta(x))$ where the function $f$ is not known a priori [49]. that arise from the sharp cutoff in the Wegner-Houghton construction. Correspondingly, the Wilson-Polchinski FRG approach can be thought to give a soft cutoff, where modes can be “partially coarsened”.

The most common NPRG approach in use today was first described in 1993 by C. Wetterich [45]. Like the Wilson-Polchinski NPRG, the Wetterich approach uses a soft cutoff, but the objects computed by this framework are fundamentally different. Instead of computing the effective Hamiltonian of modes which are below the cutoff, the Wetterich framework computes the effective free energy of modes above the cutoff. For this reason, we say that the Wilson-Polchinski framework is UV-regulated and Wetterich is IR-regulated. Yet, despite this difference in perspective, the Wetterich formalism still describes the flow effective models make from their microscopic to macroscopic pictures. In this section, we will explore how the soft-cutoff construction is related to a notion of non-deterministic coarsening, and in turn, the information bottleneck framework. An in-depth discussion of the philosophy and implementation of NPRG techniques would be distracting, so we instead refer the reader to a number of good references on the topic [46, 47, 48, 49].

So far we have not explained how one actually imposes a soft-cutoff scheme. We begin by examining the Wetterich setup, in which one writes the effective (Helmholtz) free energy at cutoff $k$:

The bare action, given by $S$, is the microscopic theory which is known a priori. The source $J$ allows us to take (functional) derivatives of this object to obtain cumulants (connected Green’s functions). The remaining term $\Delta S_{k}[\chi]$ is known as the deformation, and it is this term which enforces the cutoff. It is written as a bilinear in $\chi$:

For compactness, we will often resort to a condensed notation and express integrals instead as contraction over suppressed continuous indices. For example, the deformation may be re-written:

The kernel (matrix) $R$ is known as the regulator, and it controls the “shape” of the cutoff. Almost always, it is chosen to be diagonal in Fourier basis so that the cutoff $k$ has the interpretation of a wavenumber or momentum. The resulting Fourier-transformed regulator $R_{k}(q)$ has some freedom in its definition, but it must satisfy the following properties [30]:

1. 1.

   $\lim_{q^{2}/k^{2}\to 0}R_{k}(q)>0$

2. 2.

   $\lim_{k^{2}/q^{2}\to 0}R_{k}(q)=0$

3. 3.

   $R_{k}(q)\to\infty\quad\forall q\quad\text{ as }\quad k\to\infty$

In principle, given the coarsening parameters $A_{k}$ and $\Delta_{k}$ for all $k$, the exact flow equation for $\Gamma^{\text{av}}_{k}$ is determined. Wetterich gives explicit choices for these parameters, while Wilson and Polchinski independently give their own (though in slightly different fashion). The term $C_{k}$ is a normalizing constant which is essentially unimportant to the remainder of our discussion.

Now we connect the IR and UV approaches to show that they are complementary, and in some sense, equivalent. In particular, suppose we know $P_{k}[\tilde{\chi}|\chi]$ for all $k$. Then, from this single object, one can construct both the IR-regulated and UV-regulated flows. This should make intuitive sense; the IR-regulated part tracks the thermodynamics of the already-integrated modes, while the UV-regulated part tracks the model of the unintegrated modes. This can all be seen clearly by writing out the full sourced partition function $Z[J]$ and invoking the normalization of the coarsening map.

In the final expression, the normalizing constant $C_{k}$ has been dropped. Readers familiar with the Polchinski formulation will immediately recognize $W_{k}[\tilde{J}[\tilde{\chi}]]$ as the effective interaction potential. However, the argument to this potential is shifted by the source $J$, which therefore enters nonlinearly, unlike in Polchinski’s approach. This difference is due to the fact that we define a flow for each initial source configuration, instead of adding a linear source term to the vacuum flow.

As promised, the existence of a family of distributions $P_{k}[\tilde{\chi}|\chi]$ with a known parameterization $(A_{k},\Delta_{k})$ allows us to define an IR regulator scheme, and therefore compute the NPRG flow both above and below the cutoff. The deformation term $\Delta S_{k}$ ultimately came from the $\chi^{2}$ term present in the coarsening map, which could be interpreted as a free energy. We also identify immediately that the IR regulator $R_{k}$ corresponding to a given choice of coarsening map is given by $A_{k}^{\dagger}\Delta_{k}^{-1}A_{k}$.

We will next use this viewpoint to introduce information bottleneck into the discussion. In particular, we will associate the coarsening map $P_{k}[\tilde{\chi}|\chi]$ with the IB coarsening map $P_{\beta}(\tilde{x}|x)$ and examine some consequences. This discussion comes with some restrictions. Firstly, one should note that all soft-cutoff NPRG frameworks, regardless of the structure of the microscopic action, assume a Gaussian coarsening map. With a non-Gaussian $P_{k}[\tilde{\chi}|\chi]$, the flow may still be defined, but it will not, in general, satisfy any known exact flow equations. This is easiest to see in the IR Wetterich formalism, since a non-Gaussian $P_{k}$ would yield a $\Delta S_{k}[\chi]$ which is no longer bilinear in $\chi$, and hence one could not write the flow equation in terms of the exact effective propagator, as it usually is. Indeed, the more general $\Delta S_{k}[\chi]$ could have terms at arbitrarily high order in $\chi$, and thus require arbitrarily high-order derivatives of $\Gamma_{k}$ in the flow equation. So, while it is not impossible to seriously consider non-Gaussian $P_{k}[\tilde{\chi}|\chi]$, it is certainly inadvisable without good reason.

With this in mind, we must also note that IB has an exact solution involving Gaussian $P_{\beta}(\tilde{x}|x)$, but only when the variables $X$ and $Y$ are jointly Gaussian. By analogy, this restricts us to discussing theories where the bare action $S[\chi]$, or perhaps more accurately, the bare Hamiltonian $\mathcal{H}[\chi]$ contains only linear and bilinear terms in $\chi$. While everything presented above holds for general $S$, everything that follows will be totally Gaussian so that IB optimality can be exactly satisfied. Finally, note that IB may not be well-defined for infinite-dimensional random variables such as fields, so our scope is further limited to finite-dimensional multivariate Gaussian distributions of classical variables.

In the last section, we briefly introduced soft-cutoff NPRG approaches and argued that both UV- and IR-regulated flows can be defined given a family of Gaussian coarsening maps $P_{k}[\tilde{\chi}|\chi]$. Broadly, we aim to show in this paper that IB and RG can be connected by identifying this map with the IB-optimal coarsening map $P_{\beta}(\tilde{x}|x)$. By this we do not mean to say that the family of maps produced by IB are the “correct” starting point for NPRG. Instead, we simply note that IB-optimality is a constraint one could impose on the coarse-graining scheme. Assuming we do so, what characteristics does the IB-RG scheme carry? Using the exact solution to GIB and Eq. (8), we identify the regulator, or soft-cutoff scheme, required by IB optimality for some known initial statistics $P(x,y)$

Here the $\beta_{i}$ are critical bottleneck values, indexed by the components of the so-called “natural” basis, which is found by diagonalizing the canonical correlation matrix $\Sigma_{X}^{-1}\Sigma_{X|Y}$ as discussed in section II. The critical bottleneck values $\beta_{i}$ are given by $(1-\lambda_{i})^{-1}$. If $V$ is the matrix of right eigenvectors of this matrix, then $s_{i}$ is given by $[V^{T}\Sigma_{X}V]_{ii}$. Notice also that this regulator is diagonal in natural basis. $\Theta$ denotes the Heaviside step function.

In the typical context, $R$ is diagonalized by a Fourier transform, and thus it represents a cutoff in wavevector or momentum. Here this notion is generalized, and instead of identifying a cutoff wavenumber $k$, we should consider the cutoff to be of information-theoretic origin, and fundamentally defined by $\beta$. Consequently, the degree to which the mode labelled by $i$ is coarsened should be found by comparing its corresponding critical value $\beta_{i}$ to the cutoff $\beta$. As such, we can essentially make the replacements $k^{2}\to\beta$ and $q^{2}\to\beta_{i}$, with the caveat that $\beta$ and $\beta_{i}$ should approach unity as $k^{2}$ and $q^{2}$ go to zero.

In Figure 1, we plot $R^{(\text{IB})}$ obtained from the first toy model presented in section IV.2 and compare it against the well-known Litim regulator [30], denoted $R^{(\text{L})}$ and given in Eq. (11). Ignoring for now the particulars of the model, we point out that the IB and Litim regulators appear qualitatively similar, and for fixed parameters $t$ and $\eta$, all limits involving $q$ and $k$ satisfy the regulator scheme requirements. Moreover, we see that the NPRG and IB notions of mode relevance are in agreement. Smaller canonical correlation eigenvalues $\lambda$ (top plot) correspond to collective modes which get integrated out later in the flow. This is reflected in the structure of the soft cutoff, which increasingly suppresses fluctuations as $q\to 0$.

Is it okay to take (9) seriously as an IR regulator scheme? Let us attempt to compare with the conditions outlined in the last section. The typical interpretation of the first requirement on $R$ is that the lowest energy modes should be given extra mass by the regulator so that they are “frozen out” of the configurational integral. In fewer words, there should not be soft modes in intermediate stages of the flow. By analogy, it must be true that $(R_{\beta})_{11}>0$, where we take $\beta_{1}=\min_{i}\beta_{i}$ to represent the most “relevant” mode (in the IB sense). Indeed, for all $\beta$, this is satisfied by (9). Next, $R$ must vanish for the $i^{\text{th}}$ mode when the cutoff $\beta$ is taken sufficiently far below $\beta_{i}$. Because of the step function, this is satisfied. Finally, each diagonal component $(R_{\beta})_{ii}$ should diverge as $\beta\to\infty$ so that at zero compression, only the saddle point configuration of the microscopic theory contributes to the generating function, or whichever thermodynamic potential we are interested in. If $\beta_{i}$ are all finite, then this limit holds as well ⁵⁵5Two edge cases may appear important; $\beta_{i}\to 1$ and $\beta_{i}\to\infty$. Because $\beta_{i}=(1-\lambda_{i})^{-1}$, these correspond to limits where a mode $[V^{T}X]_{i}$ is deterministically related to $Y$ or completely independent of $Y$, respectively. Complete deterministic dependence between $X$ and $Y$ should not be considered without modification, and in the case of independence, those modes may be removed as a formality..

Because it satisfies all of the properties required of a typical regulator in a soft-cutoff scheme, we call (9) the “IB regulator” and denote it $R^{(\text{IB})}$. This identification has some interesting consequences, which will be explored in the coming section. One particularly striking feature is that the cutoff scheme is now parameterized by the family of distributions $P(x,y)$. In IB theory, these distributions formalize the notion of “important features” of $X$ implicitly through its correlations with $Y$. This means that the RG scheme selected by a given set of IB solutions will not favor, for instance, “long distance modes” unless $P(x,y)$ is chosen to enforce that. Instead, the analogue of long distance modes are those modes which have the most information about $Y$. In section IV.2 we will attempt to clarify this by calculating the IB regulator explicitly in a simple, familiar context.

The apparent goal of Information Bottleneck is to identify the coarsening map $P_{\beta}(\tilde{x}|x)$ for some set of $\beta$ values. This seems to align poorly with the problem statement and goals of NPRG, in which the coarsening map $P_{k}[\tilde{\chi}|\chi]$ is taken as the starting point and used to derive the flow equations. Is it really true that solving IB only gets us to the starting point of an RG scheme, after which we still need to “do the RG part?” In this section, we investigate one way to resolve this dissonance by noting that the quantities one would usually consider to be the results of the NPRG flow can be used to parameterize $P_{\beta}(\tilde{x}|x)$ itself. From this viewpoint, one may organize the computation around a set of self-consistent update equations instead of a set of flow equations.

The function $Z_{\beta}(x)$ normalizes $P_{\beta}(\tilde{x}|x)$ and therefore also depends on $P_{\beta}(\tilde{x}|x)$ through the above equations.

We reiterate that this self-consistent updating scheme comes from IB optimality, written in terms of objects we would usually calculate in NPRG. The idea of a self-consistent updating scheme which determines the IR-regulated statistics and UV-regulated dynamics simultaneously is interesting. In addition to essentially replacing the flow-equation description, it is very non-perturbative in nature. However, it seems wrong that imposing a constraint on $P(\tilde{x}|x)$ should make anything easier, especially given the fact that IB enforces a goal which is only sometimes aligned with the typical goals of RG analysis. A natural question, then, is whether IB has actually provided any new leverage. More precisely, if we really have given up the flow equation in favor of a self-consistency scheme, does this new scheme actually help to calculate the objects of interest as the flow equation usually would? If so, why would IB optimality be necessary?

In principle, the self-consistent structure imposed by IB-optimality obviates the need for a traditional cutoff/flow equation description. However, the opposite is also true: if the cutoff scheme and flow equation are known, then the self-consistency conditions are displaced. Because GIB is exactly solvable, we are able to examine both approaches here. In Eq. (9), we present a soft cutoff scheme which arises from the constraint of GIB-optimality, but it is given in terms of quantities which have no physical context, and so it is hard to say a priori how it relates to existing cutoff schemes structurally. In the next section, we consider a toy model which provides this physical context and therefore affords us a glimpse into how IB-optimal NPRG schemes differ structurally from those already employed.

In the Wetterich NPRG, the cutoff is enforced through a deformation $\Delta S_{k}[\chi]=\frac{1}{2}\chi^{\dagger}R_{k}\chi$ added to the bare action or Hamiltonian. In section III.2, we identified this structure as the free energy of a Gaussian coarsening map from the bare degrees of freedom $\chi$ to some compressed representation $\tilde{\chi}$. We then defined the IB regulator through the deformation produced by the map solves the Gaussian Information Bottleneck problem, and showed that it satisfies the various “design” constraints traditionally placed upon it. An immediate consequence of this construction is that the regulator design space is now parameterized by the joint distributions $P(x,y)$ which define the starting point of IB, and for many such distributions, the preferred basis selected by IB will look nothing like Fourier modes. Of course, for finite systems not organized in a lattice, this is unsurprising; the Fourier basis will not exist in any familiar sense. However, for practitioners of NPRG, it may cause discomfort to consider a regulator $R_{v(\beta)}(u)$ in which the numbers $v$ and $u$ do not represent radii in momentum space. In contrast, for the majority of applications, the standard cutoff scheme is provided by the Litim regulator

Which should be interpreted as a soft momentum-space cutoff. The Litim regulator sees widespread use both because it is optimized to give good convergence properties in certain contexts ⁶⁶6The Litim regulator was introduced in the context of NPRG analysis of the $O(N)$ model. Its favorable or “optimal” characteristics are manifested through improved convergence properties of so-called “threshold functions,” which constitute a frequently encountered class of momentum integrals involving the cutoff., and because its simple form often leads to analytically expressible flow equations (after appropriate truncation procedures) [53, 30].

Together, the Boltzmann weight $\mathcal{H}[\chi|h]$ and the distribution $P[h]$ describing the disorder statistics constitute a joint distribution $P[\chi,h]$ which is jointly Gaussian and thus—momentarily casting aside worries about the continuously infinite-dimensional random variables—a valid starting point for GIB. From the IB standpoint, the goal would usually be to construct a coarsened field $\tilde{\chi}(x)$ which discards some information about $\chi$ while encoding as much as possible about the statistics of $h$. However, the goal here is not to discuss $\tilde{\chi},$ but rather to better understand the NPRG cutoff scheme that IB imposes as a consequence of this starting point. Since we have assumed a canonical form for the bare Green’s function $G_{0}^{-1}$ and the source term is $h\cdot\chi$, the only remaining control over $P[\chi,h]$ is the two-point correlation of $h$:

To explore different forms of $R_{\beta}^{(\text{IB})},$ we therefore consider three different constructions of $H$. First, we choose $h$ to be totally uncorrelated at different points, with a constant variance at each point. Second, we choose $H$ diagonal in Fourier basis, but with some dispersion that adds position-space correlations. In both of these first examples, we will arrive at regulators with momentum-space cutoffs. It is the goal of the third case to present an $H$ which is not diagonal in momentum basis, thereby introducing a non-momentum cutoff structure.

Gaussian IB begins with a choice of joint distribution $P(x,y)$. As we have discussed, this distribution gives a constrained parameterization of a cutoff scheme which is analogous to the one employed in Wetterich NPRG. In the last section, we showed that not all choices $P(x,y)$ lead to collective modes $V^{T}X$ which have a canonical interpretation such as Fourier modes. That discussion was carried out under the assumption that the relevance variable $Y$ pertains to a source field with some disorder statistics. Generally speaking, this is only one way of constructing $Y$. Even within the constraint of $P(x,y)$ being jointly Gaussian, the physical interpretation of $x$ and $y$ can vary. Here we briefly discuss some of these alternative interpretations.

First, $Y$ may represent the environment of a set of variables $X$. This scenario is analogous to the one presented by Koch-Janusz et al. [12]. Consider a collection of spins on a lattice, and choose some enclosed region. Let $X$ be the state of the spins in that region and let $Y$ denote the state of those outside. In the case that these spins have Gaussian statistics, this is a valid starting point for GIB. With this setup, we expect that the most relevant collective modes would be relatively slowly varying in position. In fact, Gordon et al. recently formalized this idea for field theories not restricted to Gaussian statistics [13]. They consider a “buffer” zone between $X$ and $Y$ whose size is taken to infinity. In this limit, the first collective variables encoded by IB at strong compression (low $\beta$ in our notation) correspond to the operators with the smallest scaling dimensions, and hence the most relevant operators in the RG sense. Their approach is therefore promising for the analysis of systems with local interactions whose order parameter is not known a priori. More fundamentally, they have shown that $Y$ and $X$ can be chosen to enforce a traditional, “physical” definition of relevance.

Second, consider a stationary stochastic process with Gaussian statistics both in time and across variable index. We could choose $X$ to represent the current state of the system while $Y$ represents the future. Here, the most relevant modes are those projections of $X$ which vary the slowest. In fact, if we suppose that time has been properly discretized, this interpretation of the GIB problem is equivalent to a certain class of slow feature analysis problems [55].

Third, we can imagine another dynamical system in which variables $X$ which are driven by a stochastic signal $Y$ such that the joint distribution is Gaussian and stationary. Now, the features of $X$ which are most relevant are no longer simply the slowest-varying components. The cutoff scheme we find will depend on the statistics which generate $Y$, the manner in which $Y$ couples to $X$, the internal dynamics of $X$, and whether we take $Y$ to be in the past, future, or present.