Exact Phase Transitions in Deep Learning

Formal framework. Let $\ell(w,a)$ be a differentiable loss function that is dependent on the model parameter $w$ and a hyperparameters $a$. The loss function $\ell$ can be decomposed into a data-dependent feature learning term $\ell_{0}$ and a data-independent term $aR(w)$ that regularizes the model at strength $a$:

Learning amounts to finding the global minimizer of the loss:

Naively, one expects $L(a)$ to change smoothly as we change $a$. If $L$ changes drastically or even discontinuously when one perturb $a$, it becomes hard to tune $a$ to optimize the model performance. Thus, that $L(a)$ is well-behaved is equivalent to that $a$ is an easy-to-tune hyperparameter. We are thus interested in the case where the tuning of $a$ is difficult, which occurs when a phase transition comes into play.

It is standard to treat the first term in Eq. (1) as an energy. To formally identify the regularization term as an entropy, its coefficient must be proportional to the temperature:

where $\sigma^{2}$ controls the fluctuation of $w$ at zero temperature. We note that this identification is consistent with many previous works, where the term that encourages a lower model complexity is identified as an “entropy” Haussler et al., (1996); Vapnik, (2006); Benedek and Itai, (1991); Friston, (2009); Li and Sompolinsky, (2021). In this view, learning is a balancing process between the learning error and the model complexity. Intuitively, one expects phase transitions to happen when one term starts to dominate the other, just like thermodynamic phase transitions that take place when the entropy term starts to dominate the energy.

In this setting, the partition function is $Z(a)=\int dw\exp[-\ell(w,a)/T]$. We consider a special limit of the partition function, where both $T$ and $2\sigma^{2}$ are made to vanish with their ratio held fixed at $T/2\sigma^{2}=\gamma$. In this limit, one can find the free energy with the saddle point approximation, which is exact in the zero-temperature limit:

We thus treat $L$ as the free energy.

We formally define the order parameter and effective loss as follows.

In other words, an order parameter is a one-dimensional quantity whose minimization on $\bar{\ell}$ gives $L(a)$. The existence of an order parameter suggests that the original problem $\ell(w,a)$ can effectively be reduced to a low-dimensional problem that is much easier to understand. Physical examples are the average magnetization in the Ising model and the average density of molecules in a water-to-vapor phase transition. A dictionary of the corresponding concepts between physics and deep learning is given in Table 1.

Our theory deals with deep linear nets, the primary minimal model for deep learning. It is well-established that the landscape of a deep linear net can be used to understand that of nonlinear networks Kawaguchi, (2016); Hardt and Ma, (2016); Laurent and Brecht, (2018). The most general type of deep linear nets, with $L_{2}$ regularization and stochastic neurons, has the following loss:

where $x$ is the input data, $y$ the label, $U$ and $W^{(i)}$ the model parameters, $D$ the network depth, $\epsilon$ the noise in the hidden layer (e.g., dropout), $d_{0}$ the width of the model, and $\gamma$ the weight decay strength. We build on the recent results established in Ziyin et al., (2022). Let $b:=||U||/d_{0}$. Ref. Ziyin et al., (2022) shows that all the global minima of Eq. (5) must take the form $U=f(b)$ and $W_{i}=f_{i}(b)$, where $f$ and $f_{i}$ are explicit functions of the hyperparameters. Ref. Ziyin et al., (2022) further shows that there are two regimes of learning, where, for some range of $\gamma$, the global minimum is uniquely given by $b=0$, and for some other range of $\gamma$, some $b>0$ gives the global minimum. When $b=0$, the model outputs a constant $0$, and so this regime is called the “trivial regime,” and the regime where $b=0$ is not the global minimum is called the “feature learning regime.” In this work, we prove that the transition between these two regimes corresponds to a phase transition in the Ehrenfest sense (Definition 1), and therefore one can indeed refer to these two regimes as two different phases.

No-phase-transition theorems. The first result we prove is that there is no phase transition in any hyperparameter ($\gamma,\ E[xx^{T}],\ E[xy],\ E[y^{2}]$) for a simple linear regression problem. In our terminology, this corresponds to the case of $D=0$. The fact that there is no phase transition in any of these hyperparameters means that the model’s behavior is predictable as one tunes the hyperparameters. In the parlance of physics, a linear regressor operates within the linear-response regime.

Theorem 2 shows that a finite-depth net cannot have zeroth-order phase transitions. This theorem can be seen as a worst-case guarantee: the training loss needs to change continuously as one changes the hyperparameter. We also stress that this general theorem applies to standard nonlinear networks as well. Indeed, if we only consider the global minimum of the training loss, the training loss cannot jump. However, in practice, one can often observe jumps because the gradient-based algorithms can be trapped in local minima. The following theory offers a direct explanation for this phenomenon.

Phase Transitions in Deeper Networks. Theorem 4 shows that the quantity $b$ is an order parameter describing any phase transition induced by the weight decay parameter in Eq. (5). Let $b=||U||/d_{u}$, $A_{0}:=\mathbb{E}[xx^{T}]$, and $a_{i}$ be the $i$-th eigenvalue of $A_{0}$. The effective loss landscape is

where $x^{\prime}$ is a rotation of $x$. See Figure 1 for an illustration. The complicated landscape for $D>1$ implies that neural networks are susceptible to initialization schemes and entrapment in meta-stable states is common (see Supplementary Section A.1).

Theorem 5 shows that when $D=1$ in Eq. (5), there is a second-order phase transition precisely at

In machine learning language, $\gamma$ is the regularization strength and $||E[xy]||$ is the signal. The phase transition occurs precisely when the regularization dominates the signal. In physics, $\gamma$ and $||\mathbb{E}[xy]||$ are proportional to the temperature and energy, respectively. The phase transition occurs exactly when the entropy dominates the energy. Also, the phase transition for a depth-$1$ linear net is independent of the number of parameters of the model. For $D>1$, the size of the model does play a role in influencing the phase transition. However, $\gamma$ remains the dominant variable controlling this phase transition. This independence of the model size is an advantage of the proposed theory because our result becomes directly relevant for all model sizes, not just the infinitely large ones that the previous works often adopt.

For $D\geq 2$, we show that there is a first-order phase transition between the two phases at some $\gamma>0$. However, an analytical expression for the critical point is not known. In physics, first-order phase transitions are accompanied by latent heat. Our theory implies that this heat is equivalent to the amount of random noise we have to inject into the model parameters to escape from a local to the global minimum for a deep model. We illustrate the phase transitions studied in Figure 2. We also experimentally demonstrate that the same phase transitions take place in deep nonlinear networks with the corresponding depths (Supplementary Section A.3). While infinite-depth networks are not used in practice, they are important from a theoretical point of view Sonoda and Murata, (2019) because they can be used for understanding a (very) deep network that often appears in the deep learning practice. Our result shows that the limiting landscape has a zeroth-order phase transition at $\gamma=0$. In fact, zeroth-order phase transitions do not occur in physics, and it is a unique feature of deep learning.

Relevance of symmetry breaking. The phase transitions we studied also involve symmetry breaking. This can be seen directly from the effective landscape in Eq. (6). The loss is unaltered as one flip the sign of $b$, and therefore the loss is symmetric in $b$. Figure 3 illustrates the effect and importance of symmetry breaking on the gradient descent dynamics. Additionally, this observation may also provide an alternative venue for studying general symmetry-breaking dynamics because the computation with neural networks is both accurate and efficient.

Mean-Field Analysis. The crux of our theory can be understood by applying a simplified “mean-field” analysis of the loss function in Eq. (5). Let each weight matrix be approximated by a scalar $U=b_{D+1}$, $W_{i}=b_{i}$, ignore the stochasticity due to $\epsilon_{i}$, and let $x$ be one-dimensional. One then obtains a simplified mean-field loss:

where $c_{i}$’s are constants. The first term can be interpreted as a special type of $(D+1)$-body interaction. We now perform a second mean-field approximation, where all the $b_{i}$ take the same value $b$:

Here, $c_{0}^{\prime}$, $c_{1}^{\prime}$ and $c_{2}^{\prime}$ are structural constants, only depending on the model (depth, width, etc). The first and the third terms monotonically increase in $b$, encouraging a smaller $b$. The second term monotonically decreases in $b^{D+1}\mathbb{E}[xy]$, encouraging a positive correlation between $b$ and the feature $\mathbb{E}[xy]$. The leading and lowest-order terms regularize the model, while the intermediate term characterizes learning. For $D=0$, the loss is quadratic and has no transition. For $D=1$, the loss is identical to the Landau free energy, and a phase transition occurs when the second-order term flips sign: $c_{2}^{\prime}\gamma=c_{1}^{\prime}\mathbb{E}[xy]$. For $D>1$, the origin is always a local minimum, dominated by the quadratic term. This leads to a first-order phase transition. When $D\to\infty$, the leading terms become discontinuous in $b$, and one obtains a zeroth-order phase transition. This simple analysis highlights one important distinction between physics and machine learning: in physics, the most common type of interaction is a two-body interaction, whereas, for machine learning, the common interaction is many-body and tends to infinite-body as $D$ increases.

One implication is that $L_{2}$ regularization may be too strong for deep learning because it creates a trivial phase. Our result also suggests a way to avoid the trivial phase. Instead of regularizing by $\gamma||w||_{2}^{2}$, one might consider $\gamma||w||_{2}^{d+2}$, which is the lowest-order regularization that does not lead to a trivial phase. The effectiveness of this suggested method is confirmed in Supplementary Section A.2.

Posterior Collapse in Bayesian Deep Learning. Our results also identify an origin of the well-known problem posterior collapse problem in Bayesian deep learning. Posterior collapse refers to the learning failure where the learned posterior distribution coincides with the prior, and so no learning has happened even after training Dai and Wipf, (2019); Alemi et al., (2018); Lucas et al., (2019). Our results offer a direct explanation for this posterior collapse problem. In the Bayesian interpretation, the training loss in Eq. (5) is the exact negative log posterior, and the trivial phase exactly corresponds to the posterior collapse: the global minimum of the loss is identical to the global maximum of the prior term. Our results thus imply that (1) posterior collapse is a unique problem of deep learning because it does not occur in shallow models, and (2) posterior collapse happens as a direct consequence of the competition between the prior and the likelihood. This means that it is not a good idea to assume a Gaussian prior for the deep neural network models. The suggested fix also leads to a clean and Bayesian-principled solution to the posterior collapse problem by using a prior $\log p(w)\propto-||w||_{2}^{D+2}$.

The striking similarity between phase transitions in neural networks and statistical-physics phase transitions lends a great impetus to a more thorough investigation of deep learning through the lens of thermodynamics and statistical physics. We now outline a few major future steps:

1. 1.

   Instead of classification by analyticity, can we classify neural networks by symmetry and topological invariants?

2. 2.

   What are other possible phases for a nonlinear network? Does a new phase emerge?

3. 3.

   Can we find any analogy of other thermodynamic quantities such as volume and pressure? More broadly, can we establish thermodynamics for deep learning?

4. 4.

   Can we utilize the latent heat picture to devise better algorithms for escaping local minima in deep learning?

This work shows that the Ehrenfest classification of phase transitions aligns precisely with the number of layers in deep neural networks. We believe that the statistical-physics approach to deep learning will bring about fruitful developments in both fields of statistical physics and deep learning.