Symmetry Induces Structure and Constraint of Learning

Let us first define a general type of symmetry called mirror reflection symmetry.

Note that the vector $(I-2nn^{T})w$ is the reflection of $w$ with respect to the plane orthogonal to $n$. Also, the $L_{2}$ regularization term itself satisfies this symmetry for any $n$ because reflection is norm-preserving. An important quantity is the average of the two reflected solutions: $\bar{w}=(I-nn^{T})w$, where $\bar{w}$ is the fixed point of this transformation and is called a “symmetric solution.” This mirror symmetry can be generalized to the case where the loss function is invariant only when multiple mirror reflections are made.

By construction, $OO^{T}$ and $I-OO^{T}$ are projection matrices, and $I-2OO^{T}$ is an orthogonal matrix. There are a few equivalent ways to see this symmetry. First of all, it is equivalent to requiring the loss function to be invariant only after multiple simple mirror symmetry transformations. Let $m$ be a unit vector orthogonal to $n$. Reflections to both $m$ and $n$ give $(I-2mm^{T})(I-2nn^{T})=I-2(nn^{T}+mm^{T})$. The matrix $nn^{T}+mm^{T}$ is a projection matrix and, thus, an instantiation of $OO^{T}$. Secondly, because the composition of orthogonal unit vectors spans the space of projection matrices, $OO^{T}$ is nothing but a generic projection matrix $P$. Thus, this symmetry can be equivalently defined with respect to $P$ such that $\ell_{0}(w)=\ell_{0}((I-2P)w)$. If we let $O$ or $P$ be rank-$1$, the symmetry reduces to the simple mirror symmetry in Definition 1. We will see in Section 3 that many common types of symmetries in deep learning imply mirror symmetries.

We also make a reasonable smoothness assumption, which is only needed for part 4 of the theorem. This assumption is benign because any $C^{2}$ function satisfies this assumption in a bounded space.

With these definitions, we are ready to prove the following theorem.

Parts 1 and 2 are statements regarding the local gradient geometry, regardless of the weight decay. Parts 3 and 4 are local and global statements regarding the role of weight decay, which points to a novel mechanism through which weight decay regularizes a neural network. The fact that the constrained solutions become local minima even at a finite weight decay means that weight decay favors a sparse solution (sparse in the subspace of symmetry breaking). This is different from the behavior of weight for linear regression, where the model only becomes sparse when the weight decay is infinite. Therefore, textbooks often say that $L_{2}$ regularization leads to a dense solution (Bishop & Nasrabadi, 2006; Hastie et al., 2009). The fact that sparse solutions are favored under weight decay is thus a unique feature of deep learning. See Figure 1 for an illustration of mirror symmetries and the intuition behind the proof. We will explain these parts of the theorem in depth below. Lastly, note that this theorem applies to arbitrary function $\ell_{0}$ that has the mirror symmetry, which does not need to be a loss function.

To discuss the implication of symmetries, we introduce the concept of a “stationary condition.”

A stationary condition can be seen as a special case of an absorbing state, which is a major theme in the study of Markov processes and is associated with complex phase-transition-like behaviors (Norris, 1998; Dickman & Vidigal, 2002; Hinrichsen, 2000). Part 1 of Theorem 1 implies the following.

Alternatively, a stationary condition can be seen as a generalization of a stationary point because every stationary point in the landscape implies the existence of a stationary condition – but not vice versa. For example, some functions of the parameters might reach stationarity before the whole model reaches stationarity. The existence of such conditions implies that there are special subspaces in the landscape such that the dynamics of (S)GD within these subspaces will not leave it. See Appendix Figure 5 for an illustration of the stationary conditions.

Part 2 of Theorem 1 has important implications for the local geometry of the loss and the dynamics of SGD. Let $H$ denote the Hessian of the loss $L$ or that of the per-sample loss $\ell$. Part 2 states that $H$ close to symmetry solutions are partitioned by the symmetry condition $I-2P$ to two subspaces: one part aligns with the images of $P$, and the other part must be orthogonal to it. Namely, one can transform the Hessian into a two-block form, $H_{\perp}$ and $H_{\parallel}$, with $O$.¹¹1Let $\tilde{O}$ be any orthogonal matrix whose basis includes all the eigenvectors of $O$. Then, $O^{T}HO$ will be a two-block matrix. Note that the parameters might also contain other symmetries, so $H_{\parallel}$ and $H_{\perp}$ may also consist of multiple sub-blocks. This implies that close to the symmetric solutions, the Hessian of the loss will take a highly structured form simultaneously for all data points or batches. See Figure 2.

The fact that the Hessian of neural networks takes a similar structure after training is supported by empirical works. For example, the illustrative Hessian in Figure 2 is similar to that computed in Sagun et al. (2016). That the actual Hessians after training are well approximated by smaller blocks is supported by Wu et al. (2020). Blockwise Hessian matrices can also be related to the existence of gaps in the Hessian spectrum, which is widely observed (Sagun et al., 2017; Ghorbani et al., 2019; Wu et al., 2020; Papyan, 2018).

It is instructive to consider the special case where $O=n^{T}$ is rank-$1$. Part 2 implies that $n$ must be an eigenvector of the Hessian whenever the model is at a symmetry solution. For example, we consider a two-layer tanh network with scalar input and outputs. The loss function can always be written as $\ell(w,u)=\frac{1}{2}\left(\sum_{i}^{d}u_{i}\tanh(w_{i}x)-y\right)^{2}$. For each index $i$, $u_{i}\tanh(w_{i}x)$ contains a symmetry with the identity mirror $I_{2}$. Therefore, the theory predicts that when $u\approx w\approx 0$, the Hessian consists of $d$ $2\times 2$ block matrices. If we also recognize that a tanh network approximates a linear network at the origin, we see that there are actually two mirrors: $(1,1)$ and $(1,-1)$, which are the eigenvectors of the Hessian according to the theory. This can be compared with a direct computation. When $w=u=0$, the nonvanishing terms of the Hessian are $\frac{\partial^{2}}{\partial{w_{i}}\partial{u_{i}}}\ell=-xy$:

This means the Hessian is indeed $d$-block and that the eigenvectors are $(1,1)$ and $(1,-1)$, agreeing with the theory. It is remarkable that we can identify all the eigenvectors of an arbitrarily wide nonlinear network by examining the symmetry in the model.

The loss symmetry has a major implication for the dynamics of SGD. Because the Hessian is block-wise with fixed blocks (with probability 1), the dynamics of SGD in the symmetry direction and the symmetry-breaking direction are essentially independent. Let $O$ denote the mirror and $P=OO^{T}$ the projection matrix. If $OO^{T}w=sn$ where $n$ is a unit vector, and $s$ is a small quantity, the model is perturbatively away from the symmetry solution. In this case, one can expand the loss function to leading orders in $s$:

where we have defined the sample Hessian restricted to the projected subspace: $H(x):=P\nabla_{w}^{2}\ell(x,w_{0})P$, which is a matrix of random variables. Note that all the odd-order terms in $s$ vanish due to the symmetry in flipping the sign of $s$. In fact, one can view the training loss $\ell_{\gamma}$ or $L_{\gamma}$ as a function of $s$, which we denote as $\tilde{L}(s)$, and this analysis implies that the loss landscape close to $s=0$ has a universal geometry. See Figure 2.

This allows us to characterize the dynamics of SGD in the symmetry directions:

where $\lambda$ is the learning rate. If one model $H$ as a random matrix, this dynamics reduces to the classical problem of random matrix product (Furstenberg & Kesten, 1960). In general, the symmetric solutions at $Pw=0$ are saddle points, while the attractivity of $Pw=0$ only depends on the sign of Lyapunov exponent of dynamics. This implies that these types of saddles points are often attractive.

To compare, we first consider GD. The largest negative eigenvalue of $\mathbb{E}_{x}[H]$, $\xi^{*}$, thus gives the speed at which SGD escapes the stationary condition: $\|Pw_{t}\|\propto\exp(-\xi^{*}t)$. When weight decay is present, all the eigenvalues of $H$ will be positively shifted by $\gamma$, and, therefore, if and only if $\xi^{*}+\gamma>0$, GD will be attracted to these symmetric solutions. In this sense, $\xi^{*}$ gives a critical weight decay value at which a symmetry-induced constraint is favored.

For SGD, the dynamics is qualitatively different. The naive expectation is that, when using SGD, the model will escape the stationary condition faster due to the noise. However, this is the opposite of the truth. The existence of the SGD noise due to minibatch sampling makes these stationary conditions more attractive. The stability of the type of dynamics in Eq. (4) can be analyzed by studying the condition for convergence in probability of the solution $Pw=0$ (Ziyin et al., 2023a). One can show that $Pw$ converges to $0$ in probability if and only if the Lyapunov exponent of the process $\Lambda$ is negative, which is possible even if this critical point is a strict saddle. When does a subspace of $Pw$ converge (or collapse) to zero? A qualitatively correct critical learning rate can be derived using a diagonal approximation of the Hessian. In this case, each subspace of $H(x)$ has its own Lyapunov exponent and can be analytically computed. Let $\xi(x)$ denote the eigenvalue of $H(x)$ in this subspace. Then, this subspace collapses when $\Lambda=\mathbb{E}_{x}[\log|1-\lambda(\xi(x)+\gamma)|]<0$, which is negative for a large learning rate (see Appendix C for a formal treatment). The meaning of this condition becomes clear by expanding to the second order in $\lambda$ to obtain:

The numerator is the eigenvalue of the empirical loss, and the denominator can be identified as the minibatch noise effect (Wu et al., 2018), which becomes larger if the batch size is small or if the dataset is noisy. Therefore, this phenomenon happens due to the competition between the signal and noise in the gradient. This example shows that at a large learning rate, the stationary conditions are favored solutions of SGD, even if they are not favored by GD. Also, convergence to these symmetry-induced saddles is not a unique feature of SGD but happens for Adam-type dynamics as well (Ziyin et al., 2021, 2023a).

Two novel applications of this analysis are to learning a sparse model and a low-rank model. See Figure 3. We first apply it to a linear regression with rescaling symmetry. It is known that when both weight decay and rescaling symmetries are present, the solutions are sparse and identical to lasso (Ziyin & Wang, 2023). Our result shows that even without weight decay, the solutions are sparse at a large learning rate. Then, we consider a matrix factorization problem. Classical results show that the solutions are low-rank when weight decay is present (Srebro et al., 2004). Our result shows that even if there is no weight decay, SGD at a large learning rate or gradient noise converges to these low-rank saddles. The fact that these constrained structures disappear completely when the symmetry is removed supports our claim that symmetry is the cause of them.

Another strong piece of evidence for the relevance of the theory to real neural networks is that after training, the Hessian of the loss function is observed to contain many small negative eigenvalues, which hints at the convergence to saddle points (Sagun et al., 2016, 2017; Ghorbani et al., 2019; Alain et al., 2019). A related phenomenon is that of pathological Fisher information. Our result implies that the Fisher information is singular close to any symmetry solutions. Note that $O^{T}\nabla_{w}\ell(w,x)=0$ for a symmetry solution and any $x$. Therefore, the Fisher information has a zero eigenvalue along the directions orthogonal to any mirror symmetry, in agreement with previous findings (Wei et al., 2008; Cousseau et al., 2008; Fukumizu, 1996; Karakida et al., 2019a, b).

Parts 3 and 4 of Theorem 1 imply that constrained solutions are favored when weight decay is used. These results can be stated in an alternative way: that every mirror symmetry plus weight decay has an $L_{1}$ equivalent. To see this, let the loss function $L_{0}(w)$ be $O$-symmetric, and $P=OO^{T}$. Let $w$ be an arbitrary weight, which we decompose as $w=w^{\prime}+sPw/||Pw||$, where we define $s=||Pw||$. Let us define an equivalent loss function $\tilde{L}_{0}(w^{\prime},Pw/||Pw||,s^{2}):=L_{0}(w)$. By definition, we have successfully constructed the $L_{1}$ equivalent of the original loss.

where we introduced $|z|=s^{2}$. Therefore, along the symmetry-breaking direction, the loss function has an equivalent $L_{1}$ form. One can also show that $\tilde{L}_{0}$ is well defined as an $L_{1}$-constrained loss function. If $L_{0}$ is differentiable, $\tilde{L}_{0}$ is differentiable except at $s=0$. Thus, it suffices to show that the right derivative of $\tilde{L}_{0}$ with respect to $z$ exists at $z=0_{+}$. As we have discussed, at $z=0$, the expansion of $L_{0}$ is second order in $s$. This means that the leading order term of $\tilde{L}_{0}$ is first order in $z$, and so the $L_{1}$ penalty is well-defined for this loss function. Meanwhile, if there is no symmetry, this reparametrization will not work because $s=0$ will have a divergent derivative.

Sparsity and low-rankness are typical structured constraints that practitioners often want to incorporate into their models (Tibshirani, 1996; Meier et al., 2008; Jaderberg et al., 2014). However, the known methods of achieving these structured constraints tend to be tailored for specific problems and based on nondifferentiable operations. Our theory shows that incorporating symmetries is a general and scalable way to introduce such constraints into deep learning. Consider solving the following constrained problem: $\min_{\theta}L(\theta)$ s.t. as many elements of $P\theta$ are zero as possible. Here, $P=OO^{T}$ is a projection matrix. Our theory implies an algorithm for enforcing such constraints in a differentiable way: introducing an artificial $O$-symmetry to the loss function encourages the constraint $O^{T}\theta=0$, which can be achieved by running GD on the following loss function:

where $w,\ u,\ v$ have the same dimension as $\theta$ and $T(w,u,v)=(I-P)v+(Pw)\odot(Pu)$, where $\odot$ denotes the Hadamard product. We call the algorithm DCS, standing for differentiable constraint by symmetry. This parameterization introduces the mirror symmetry to which $O^{T}T(w,u,v)=0$ is a stationary condition. By Theorem 1, a sufficiently large $\alpha$ ensures that $O^{T}T(w,u,v)=0$ is an energetically favored solution. Also, note that this parametrization is a “faithful” parametrization in the sense that it is always true that $\min_{w,u,v}L(T(w,u,v))=\min_{\theta}L(\theta)$. A special case of this algorithm is the spred algorithm (Ziyin & Wang, 2023), which focuses on the rescaling symmetry and has been found to be efficient in model compression problems. See Section B for an application of the algorithm to ResNet18.

The simplest type of symmetry in deep learning is the rescaling symmetry. Consider a loss function $\ell_{0}$ for which the following equality holds for any $x$, arbitrary vectors $u,\ w$ and $\rho\in\mathbb{R}_{/\{0\}}$:

For the rescaling symmetry and for all the problems we discuss below, it is also possible for $\ell_{0}$ to contain other parameters $v$ that are irrelevant to the symmetry: $\ell_{0}=\ell_{0}(u,w,v)$. Since having such $v$ or not does not change our result, we omit writting $v$.

The following theorem states that this symmetry leads to sparsity in the parameters.

To prove this using the main theorem is simple. When the rescaling symmetry exists between two scalars $u$ and $w$, there are two planes of mirror symmetry: $n_{1}=(1,1)$ and $n_{2}=(1,-1)$. Here, $n_{1}$ symmetry implies that $u=-w$ is a symmetry solution, and $n_{2}$ symmetry implies that $u=w$ is a symmetry solution. Applying Theorem 1 to these two mirrors implies that $u=0$ and $w=0$ is a symmetry solution and obeys Theorem 2. When $u\in\mathbb{R}^{d_{1}}$ and $w\in\mathbb{R}^{d_{2}}$ are vectors of arbitrary dimensions and have the rescaling symmetry, one can identity the implied mirror symmetry as $O=I$, and so $I-2P=-I$: the loss function is symmetric to a simultaneous flip of all the signs of $u$ and $w$. Applying Theorem 1 to this mirror again finishes the proof.

This symmetry usually manifests itself when part of the parameters is linearly connected. Previous works have used this property to either understand the inductive bias of neural networks or design efficient training algorithms. When the model is a fully connected ReLU network, Neyshabur et al. (2014) showed that having $L_{2}$ is equivalent to $L_{1}$ constraints of weights. Ziyin & Wang (2023) designed an algorithm to compress neural networks by transforming a parameter vector $v$ to $u\odot w$, where $\odot$ is the Hadamard product.

For numerical evidence, see Figure 3-left and 4. Here, we consider a linear regression task with noisy Gaussian data, where the loss function is $\ell=(v^{T}x-y)$, where $v$ is either directly trained or parametrized as the Hadamard product of two-parameter vectors to artificially introduce rescaling symmetry: $v=u\odot w$. We see that without such symmetry, the model never converges to a sparse solution, whereas the symmetrized parametrization converges to symmetry solutions, in agreement with the theory.

A more involved but common type of symmetry is the rotation symmetry, which also appears in a few slightly different forms in deep learning. This type of symmetry appears in matrix factorization problems, where it is a main cause of the emergence of saddle points (Li et al., 2019). It also appears in Bayesian deep learning (Tipping & Bishop, 1999; Kingma & Welling, 2013; Lucas et al., 2019; Wang & Ziyin, 2022), self-supervised learning (Chen et al., 2020; Ziyin et al., 2023b), and transformers in the form of key-query matrices (Vaswani et al., 2017; Dong et al., 2021).

Now, we show that rotation symmetry in the landscape leads to low rankness. We use the word “rotation” in a broad sense, including all orthogonal transformations. There are two types of rotation symmetry common in deep learning. In the first kind, we have for any $W$,

for any orthogonal matrix $\Omega$ such that $\Omega\Omega^{T}=I$ and $W$ is a set of weights viewed as a matrix or vector whose left dimension matches the right dimension of $\Omega$.

Part 1 of the statement deserves a closer look. $n^{T}W=0$ implies that $W$ is low-rank and $n$ is a left eigenvector of $W$. That the gradient vanishes in this direction means that once the weight matrix becomes low-rank, it will always be low-rank for the rest of the training. To prove it using Theorem 1, we note that for any projection matrix $\Pi$, the matrix $I-2\Pi$ is an orthogonal matrix because $(I-2\Pi)(I-2\Pi)^{T}=(I-2\Pi)^{2}=I$. Therefore, the rotation symmetry already implies that for any $\Pi$ and $W$, $\ell_{0}((I-2\Pi)W)=\ell_{0}(W)$. To apply the theorem, we need to view $W$ as a vector, and the corresponding reflection matrix is ${\rm diag}(I-2\Pi,...,I-2\Pi)$, a block-wise repetition of the matrix $I-2\Pi$, where each block corresponds to a column of $W$. By construction, $P$ is also a projection matrix. Since this holds for an arbitrary $\Pi$, one can choose $\Pi$ to match the desired plane in Theorem 3, which can be then proved by invoking Theorem 1.

A more common symmetry is a “double” rotation symmetry, where $\ell_{0}$ depends on two matrices $U$ and $W$ and satisfies $\ell_{0}(U,W)=\ell_{0}(UR,R^{T}W)$, for any orthogonal matrix $R$ and any $U$ and $W$. Namely, the loss function is invariant if we simultaneously rotate two different matrices with the same rotation. In this case, one can show something similar: $n^{T}W=0$ and $Un=0$ for some fixed direction $n$ is the favored solution.

See Figure 3-mid, which shows that low-rank solutions are preferred in matrix factorization when the gradient noise is large (namely, when the learning rate is large or the label noise is strong), whereas such a tendency disappears when one removes the rotation symmetry by introducing a residual connection. An additional experiment with weight decay is presented in the Appendix.³³3It is now worthwhile to clarify the difference between continuous and discrete symmetries because rescaling and rotation symmetries are continuous transformations, and it seems like continuous symmetries can also cause these constraints. This is not true – the constraints are only consequences of the existence of reflection surfaces. Having continuous symmetries is one convenient way to induce certain types of mirror symmetry, but not the only way. Because transformers have the double rotation symmetry in the self attention, we also perform an experiment with transformers in an in-context learning task in Section B.6.

The most common type of symmetry in deep learning is permutation symmetry. It shows up in virtually all architectures in deep learning. A primary and well-studied example is that in a fully connected network, the training objective is invariant to any pairwise exchange of two neurons in the same hidden layer. We refer to this case as the “special permutation symmetry” because it is a special case of the permutation symmetry we study below. Many recent works are devoted to understanding the special permutation symmetry (Simsek et al., 2021; Entezari et al., 2021; Hou et al., 2019).

Here, we study a more general and abstract type of permutation symmetry. The loss function has a permutation symmetry between parameter subsets $\theta_{a}$ and $\theta_{a}$ if, for any $\theta_{a}$ and $\theta_{b}$,⁴⁴4A special case is a hidden layer of a network; let $w_{a}$ and $u_{a}$ be the input and output weights of neuron $a$, and $w_{b}$, $u_{b}$ be the input and output weights of neuron $b$. We can thus let $\theta_{a}:=(w_{a},u_{a})$ and $\theta_{b}:=(w_{b},u_{b})$.

When there are multiple pairs that satisfy this symmetry, one can combine this pairwise symmetry to generate arbitrary permutations. From this perspective, permutation symmetries appear far more common than they are recognized. Another example is that a convolutional neural network is invariant to a pairwise exchange of two filters, which is rarely studied. A scalar rescaling symmetry can also be regarded as a special case of permutation symmetry.

Here, we show that the permutation symmetry tends to make the neurons become identical copies of each other (namely, encouraging $\theta_{a}$ to be as close to $\theta_{b}$ as possible).

To prove it, one can identify the projection as

Let $\theta=(\theta_{1},\theta_{2})$ denote a vector combination of both sets of the parameters. The permutation symmetry thus implies the mirror symmetry: $\ell_{0}(\theta)=\ell_{0}((I-2P)\theta)$. The symmetry solution is $\theta_{1}=\theta_{2}$, and applying the master theorem to this mirror allows us to obtain Theorem 4.

This theorem implies that a permutation symmetry can be seen as a generalized form of ensembling smaller submodels.⁵⁵5It is not true that the origin is always favored when a mirror symmetry exists. Consider this loss: $L_{\gamma}(w_{1},w_{2})=[(w_{1}+w_{2})-1]^{2}+\gamma(w_{1}^{2}+w_{2}^{2})$. A permutation symmetry exists between $w_{1}$ and $w_{2}$. The condition $\theta_{a}-\theta_{b}=0$ is satisfied for all solutions of the loss whenever $\gamma>0$. Meanwhile, no solution satisfies $\theta_{a}=\theta_{b}=0$. Restricted to fully connected networks, this type of homogeneous ensembling can be called a “neuron collapse.” This identification of the stationary subspace agrees with the result in Simsek et al. (2021). Special cases of this result have been proved previously. For a fully connected network, Fukumizu & Amari (2000) showed that the solutions of subnetworks are also solutions of the larger network, and Chen et al. (2023) demonstrated that these subnetwork solutions of fully connected networks can be attractive when the learning rate is large. Our result is more general because it does not restrict to the special permutation symmetry induced by fully connected networks. A novel application is that the networks have block-wise neurons and activation patterns whenever weight decay is present.

We train a Resnet18 on the CIFAR-10 dataset, following the standard training procedures. We compute the correlation matrix of neuron firing of the penultimate layer of the model, which follows a fully connected layer. We compare the matrix for both training with and without weight decay and for both pre- and post-activations (see Appendix B). See Figure 3-right, which shows that homogeneous solutions are preferred when weight decay is used, in agreement with the prediction of Theorem 1.

Also, our theory implies that the commonly observed loss of plasticity problem in continual and reinforcement learning (Lyle et al., 2023; Abbas et al., 2023; Dohare et al., 2023) is attributable to symmetries in the model. For a given task, weight decay or a finite learning rate makes the model converge to symmetry solutions, which tend to be low-capacity constrained solutions. If we train on an additional task, the capacity of the model can only decrease because the symmetry solutions are also stationary conditions, which SGD cannot escape. Fortunately, our theory suggests at least two ways to fix this problem: (1) use an alternative parameterization that explicitly removes the symmetry and/or (2) inject additive noise to the gradient to eliminate the stationary conditions. In fact, gradient noise injection is a known method to alleviate plasticity loss (Dohare et al., 2023). There are alternative ways to achieve (1). An easy way is to bias every (symmetry-relevant) parameter by a random bias: $w_{i}\to w_{i}+\beta_{i}$, where $\beta_{i}$ is a small fixed random variable.

A related phenomenon that symmetry can explain is the collapse of neural networks. The most common type of collapse is when the learned representation of a neural network spans a low-rank subspace of the entire available space, often leading to reduced expressive power. In Bayesian deep learning, a posterior collapse happens when the stochastic latent variables are low-rank (Lucas et al., 2019; Wang & Ziyin, 2022). This can be attributed to the double rotation symmetry of the encoder’s last layer weight and the decoder’s first layer weight. In self-supervised learning, a dimensional collapse happens when the representation of the last layer is low-rank (Tian, 2022), which has been found to be explained by the rotation symmetry of the last layer weight matrix. This also explains why many self-supervised learning methods focus on removing the symmetry (Bardes et al., 2021). The rank collapse that happens in self-attention may also be relevant (Dong et al., 2021). In supervised learning, the “neural collapse” happens when the learned representation of the penultimate learning becomes low-rank, which happens when weight decay is present (Papyan et al., 2020). Figure 3 shows that such a phenomenon can be attributed to the permutation symmetry in the fully connected layer. In summary, our result provides a unified perspective of the collapse phenomenon: collapses are caused by symmetries in the loss function. Our theory also suggests that these collapse phenomena have a natural interpretation as “phase transitions” in theoretical physics, where a collapse solution corresponds to a symmetric state with the “order parameter” being $O^{T}\theta$.⁶⁶6For example, see Ziyin et al. (2022) and Ziyin & Ueda (2022) for a study of these phase transitions in deep linear networks.