A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning

Given a group of transformations, the symmetry condition (1) imposes linear constraints on $F$ that can be enforced during the fitting process. However, there is one constraint per transformation, making direct enforcement impractical for continuous Lie groups such as rotations or translations. We observe that for smoothly-parametrized, connected groups, it suffices to consider a finite collection of infinitesimal linear constraints ${\mathcal{L}}_{\xi_{i}}F=0$ where ${\mathcal{L}}_{\xi}$ is the defined “Lie derivative” defined by

$${\mathcal{L}}_{\xi}F(x)=\left(\left.\frac{\partial\tilde{T}_{g}F(x)}{\partial g}\right|_{g=\operatorname{Id}}-\frac{\partial F(x)}{\partial x}\left.\frac{\partial T_{g}(x)}{\partial g}\right|_{g=\operatorname{Id}}\right)\xi.$$

(2)

Notice that this expression is linear with respect to $\xi$ and with respect to $F$.

The symmetries of a given model $F$ form a subgroup that we seek to identify within a given group of candidates. For continuous Lie groups of transformations, the component of the subgroup containing the identity is revealed by the nullspace of the linear map

$$L_{F}:\xi\mapsto{\mathcal{L}}_{\xi}F.$$

(3)

More generally, the symmetries of a smooth surface in $\mathbb{R}^{n}$ can be determined from data sampled from this surface by computing the nullspace of a positive semidefinite operator. When the surface is the graph of the function $F$ in $\mathbb{R}^{m}\times\mathbb{R}^{n}$, this operator is $L_{F}^{*}L_{F}$ with $L_{F}^{*}$ being an adjoint operator.

Here, we seek to learn a model $F$ that both fits data and possesses as many symmetries as possible from a given candidate group of transformations. Since the nullspace of the operator $L_{F}$ defined in (3) corresponds with the symmetries of $F$, we seek to minimize the rank of $L_{F}$ during the training process. To do this, we regularize optimization problems for $F$ using a convex relaxation of the rank given by the nuclear norm (sum of singular values)

$$\|L_{F}\|_{*}=\sum_{i=1}^{\dim G}\sigma_{i}(L_{F}).$$

(4)

This is convex with respect to $F$ because $F\mapsto L_{F}$ is linear and the nuclear norm is convex.

Data-augmentation, as reviewed by Shorten and Khoshgoftaar (2019); Van Dyk and Meng (2001), is one of the simplest ways to incorporate known symmetry into machine learning models. Usually this entails training a neural network architecture on training data to which known transformations have been applied. The theoretical foundations of these methods are explored by Chen et al. (2020). Data-augmentation has also been used by Benton et al. (2020) to construct equivariant neural networks by averaging the network’s output over transformations applied to the data. Finally, Brandstetter et al. (2022) applied data-augmentation strategies with known Lie-point symmetries for improving neural PDE solvers.

Symmetry can also be enforced directly on the machine learning architecture. For example, Convolutional Neural Networks (CNNs), introduced by Fukushima (1980) and popularized by LeCun et al. (1989), achieve translational equivariance by employing convolutional filters with trainable kernels in each layer. CNNs have been generalized to provide equivariance with respect to symmetry groups other than translation. Group-Equivariant CNNs (G-CNNs) (Cohen and Welling, 2016) provide equivariance with respect to arbitrary discrete groups generated by translations, reflections, and rotations. Rotational equivariance can be enforced on three-dimensional scalar, vector, or tensor fields using the 3D Steerable CNNs developed by Weiler et al. (2018). Spherical CNNs Cohen et al. (2018); Esteves et al. (2018) allow for rotation-equivariant maps to be learned for fields (such as projected images of 3D objects) on spheres. Essentially any group equivariant linear map (defining a layer of an equivariant neural network) acting fields can be described by group convolution (Kondor and Trivedi, 2018; Cohen et al., 2019), with the spaces of appropriate convolution kernel characterized by Cohen et al. (2019). Finzi et al. (2020) provides a practical way to construct convolutional layers that are equivariant with respect to arbitrary Lie groups and for general data types. For dynamical systems, Marsden and Ratiu (1999); Rowley et al. (2003); Abraham and Marsden (2008) describe techniques for symmetry reduction of the original problem to a quotient space where the known symmetry group has been factored out. Related approaches have been used by Peitz et al. (2023); Steyert (2022) to approximate Koopman operators for symmetric dynamical systems (see Koopman (1931); Mezić (2005); Mauroy et al. (2020); Otto and Rowley (2021); Brunton et al. (2022)).

A general method for constructing equivariant neural networks is introduced by Finzi et al. (2021), and relies on the observation that equivariance can be enforced through a set of linear constraints. For graph neural networks, Maron et al. (2018) characterizes the subspaces of linear layers satisfying permutation equivariance. Similarly, Ahmadi and Khadir (2020) shows that discrete symmetries and other types of side information can be enforced via linear or convex constraints in learning problems for dynamical systems. Our work builds on the results of Finzi et al. (2021), Weiler et al. (2018), Cohen et al. (2019), and Ahmadi and Khadir (2020) by showing that equivariance can be enforced in a systematic and unified way via linear constraints for large classes of functions and neural networks. Concurrent work by (Yang et al., 2024) shows how to enforce known or discovered Lie group symmetries on latent dynamics using hard linear constraints or soft penalties.

Early work by Rao and Ruderman (1999); Miao and Rao (2007) used nonlinear optimization to learn infinitesimal generators describing transformations between images. Later, it was recognized by Cahill et al. (2023) that linear algebraic methods could be used to uncover the generators of continuous linear symmetries of arbitrary point clouds in Euclidean space. Similarly, Kaiser et al. (2018) and Moskalev et al. (2022) show how conserved quantities of dynamical systems and invariances of trained neural networks can be revealed by computing the nullspaces of associated linear operators. We connect these linear-algebraic methods to the Lie derivative, and provide generalizations to nonlinear group actions on manifolds. The Lie derivative has been used by Gruver et al. (2022) to quantify the extent to which a trained network is equivariant with respect to a given one-parameter subgroup of transformations. Our results show how the Lie derivative can reveal the entire connected subgroup of symmetries of a trained model via symmetric eigendecomposition.

More sophisticated nonlinear optimization techniques use Generative Adversarial Networks (GANs) to learn the transformations that leave a data distribution unchanged. These methods include SymmetryGAN developed by Desai et al. (2022) and LieGAN developed by Yang et al. (2023b). In contrast, our methods for detecting symmetry are entirely linear-algebraic.

While symmetries may exist in data, their representation may be difficult to describe. Yang et al. (2023a) develop Latent LieGAN (LaLieGAN) to extend LieGAN to find linear representations of symmetries in a latent space. Recently this has been applied to dynamics discovery (Yang et al., 2024). Likewise, Liu and Tegmark (2022) discover hidden symmetries by optimizing nonlinear transformations into spaces where candidate symmetries hold. Similar to our approach for promoting symmetry, they use a cost function to measure whether a given symmetry holds. In contrast, our regularization functions enable subgroups of candidate symmetry groups to be identified.

Biasing a network towards increased symmetry can be accomplished through methods such as symmetry regularization. Analogous to the physics-informed loss developed in PINNs by Raissi et al. (2019) that penalize a solution for violating known dynamics, one can penalize symmetry violation for a known group; for example, Akhound-Sadegh et al. (2024) extends the PINN framework to penalize deviations of known Lie-point symmetries of a PDE. More generally, however, one can consider a candidate group of symmetries and promote as much symmetry as possible that is consistent with the available data. Wang et al. (2022) discusses these approaches, along with architecture-specific methods, including regularization functions involving summations or integrals over the candidate group of symmetries. While our regularization functions resemble these for discrete groups, we use a radically different regularization for continuous Lie groups. By leveraging the Lie algebra, our regularization functions eliminate the need to numerically integrate complicated functions over the group—a task that is already prohibitive for the $10$-dimensional non-compact group of Galilean symmetries in classical mechanics.

Automated data augmentation techniques introduced by Cubuk et al. (2019); Hataya et al. (2020); Benton et al. (2020) are another class of methods that arguably promote symmetry. These techniques optimize the distribution of transformations applied to augment the data during training. For example “Augerino” is an elegant method developed by Benton et al. (2020) which averages an arbitrary network’s output over the augmentation distribution and relies on regularization to prevent the distribution of transformations from becoming concentrated near the identity. In essence, the regularization biases the averaged network towards increased symmetry.

In contrast, our regularization functions promote symmetry on an architectural level for the original network. This eliminates the need to perform averaging, which grows more costly for larger collections of symmetries. While a distribution over symmetries can be useful for learning interesting partial symmetries (e.g. $6$ stays $6$ for small rotations, before turning into $9$), as is done by Benton et al. (2020) and Romero and Lohit (2022), it is not clear how to use a continuous distribution over transformations to identify lower-dimensional subgroups which have measure zero. On the other hand, our linear-algebraic approach easily identifies and promotes symmetries in lower-dimensional connected subgroups.

There are several other approaches that incorporate various aspects of enforcing, discovering, and promoting symmetries. For example, Baddoo et al. (2023) developed algorithms to enforce and promote known symmetries in dynamic mode decomposition, through manifold constrained learning and regularization, respectively. Baddoo et al. (2023) also showed that discovering unknown symmetries is a dual problem to enforcing symmetry. Exploiting symmetry has also been a central theme in the reduced-order modeling of fluids for decades (Holmes et al., 2012). As machine learning methods are becoming widely used to develop these models (Brunton et al., 2020), the themes of enforcing and discovering symmetries in machine models are increasingly relevant. Known fluid symmetries have been enforced in SINDy for fluid systems (Loiseau and Brunton, 2018) through linear equality constraints; this approach was generalized to enforce more complex constraints (Champion et al., 2020). Unknown symmetries were similarly uncovered for electroconvective flows (Guan et al., 2021). Symmetry breaking is also important in many turbulent flows (Callaham et al., 2022).

Lie groups are ubiquitous in science and engineering. Some familiar examples include the general linear group $\operatorname{GL}(n)$ consisting of all real, invertible, $n\times n$ matrices; the orthogonal group

$$O(n)=\left\{Q\in\mathbb{R}^{n\times n}\ :\ Q^{T}Q=I\right\};$$

(5)

and the special Euclidean group

$$\operatorname{SE}(n)=\left\{\begin{bmatrix}Q&b\\ 0&1\end{bmatrix}\ :\ Q\in\mathbb{R}^{n\times n},\ b\in\mathbb{R}^{n},\ Q^{T}Q=I,\ \det(Q)=1\right\},$$

(6)

representing rotations and translations in real $n$-dimensional space, $\mathbb{R}^{n}$, embedded in $\mathbb{R}^{n+1}$ via $x\mapsto(x,1)$. Observe that the sets $\operatorname{GL}(n)$, $O(n)$, and $\operatorname{SE}(n)$ contain the identity matrix and are closed under matrix multiplication and inversion, making them into (non-commutative) groups. They are also smooth manifolds, which makes them Lie groups (Lee, 2013). In general, a Lie group is a smooth manifold that is simultaneously an algebraic group whose composition and inversion operations are smooth maps. The identity element is usually denoted $e$ for “einselement”, which for a matrix Lie group is the identity matrix $e=I$. This section summarizes some basic results that can be found in references such as (Abraham et al., 1988; Lee, 2013; Varadarajan, 1984; Hall, 2015).

The most useful and profound property of a Lie group is the fact that the group is almost entirely characterized by an associated vector space called its Lie algebra. This allows global nonlinear questions about the group — such as which elements leave a function unchanged — to be answered using linear algebra. If $G$ is a Lie group, its Lie algebra, commonly denoted $\operatorname{Lie}(G)$ or ${\mathfrak{g}}$, is the vector space consisting of all smooth vector fields on $G$ that remain invariant when pushed forward by left translation $L_{g}:h\mapsto g\cdot h$. Translating back and forth from the identity element, the Lie algebra can be identified with the tangent space $\operatorname{Lie}(G)\cong T_{e}G$. For example, the Lie algebra of the orthogonal group $O(n)$ consists of all skew-symmetric matrices, and is denoted

$$\mathfrak{o}(n)=\left\{S\in\mathbb{R}^{n\times n}\ :\ S+S^{T}=0\right\}.$$

(7)

A key fact is that the Lie algebra of $G$ is closed under the “Lie bracket” of vector fields¹¹1In $\mathbb{R}^{n}$ a vector field $V=(V^{1},\ldots,V^{n})$ is equivalent to a directional derivative operator $V^{1}\frac{\partial}{\partial x^{1}}+\cdots+V^{1}\frac{\partial}{\partial x^{1}}$. A vector field on a smooth manifold is defined as an analogous linear operator acting on the space of smooth functions. The Lie bracket is defined as the commutator of these operators. See Lee (2013).

$$[\xi,\eta]=\xi\eta-\eta\xi\in\operatorname{Lie}(G).$$

(8)

For matrix Lie groups, this corresponds to the same commutator of matrices $\xi,\eta\in T_{I}G$, as shown by Theorem 3.20 in Hall (2015).

The key tool relating global properties of a Lie group back to its Lie algebra is the exponential map $\exp:\operatorname{Lie}(G)\to G$. A vector field $\xi\in\operatorname{Lie}(G)$ has a unique integral curve $\gamma:(-\infty,\infty)\to G$ passing through the identity $\gamma(0)=e$ and satisfying $\gamma^{\prime}(t)=\xi|_{\gamma(t)}$. The exponential map defined by

$$\exp(\xi):=\gamma(1)$$

(9)

reproduces the entire integral curve $\exp(t\xi)=\gamma(t)$ thanks to Proposition 20.5 in Lee (2013). Such an exponential curve is illustrated in Figure 1. For a matrix Lie group and $\xi\in T_{I}G$, the exponential map is given by the matrix exponential

$$\exp(\xi)=e^{\xi}=\sum_{k=0}^{\infty}\frac{1}{k!}\xi^{k}.$$

(10)

Proposition 20.8 in (Lee, 2013) provides many of the basic properties of the exponential map, such as $\exp((s+t)\xi)=\exp(s\xi)\cdot\exp(t\xi)$, $\exp(\xi)^{-1}=\exp(-\xi)$, and $\operatorname{\mathrm{d}}\exp(0)=\operatorname{Id}_{T_{e}G}$. Perhaps the most important is that it provides a diffeomorphism of an open neighborhood of the origin $0$ in $\operatorname{Lie}(G)$ and an open neighborhood of the identity element $e$ in $G$.

The connected component of $G$ containing the identity element is called the “identity component” of the Lie group and is denoted $G_{0}$. Any element in this component can then be expressed as a finite product of exponentials thanks to Proposition 7.14 in (Lee, 2013), that is

$$G_{0}=\big{\{}\exp{(\xi_{1})}\cdots\exp{(\xi_{N})}\ :\ \xi_{1},\ldots,\xi_{N}\in\operatorname{Lie}(G),\ N=1,2,3,\ldots\big{\}}.$$

(11)

Moreover, the identity component is a normal subgroup of $G$ and all of the other connected components of $G$ are diffeomorphic cosets of $G_{0}$ (Proposition 7.15 in Lee (2013)), as we illustrate in Figure 1. For example, the special Euclidean group $\operatorname{SE}(n)$ is connected, and thus equal to its identity component. On the other hand, the orthogonal group $O(n)$ is compact and has two components consisting of orthogonal matrices $Q$ whose determinants are $1$ and $-1$. The identity component of the orthogonal group is called the special orthogonal group and is denoted $\operatorname{SO}(n)$. It is a general fact that when a Lie group such as $\operatorname{SO}(n)$ is connected and compact, it is equal to the image of the exponential map without the need to consider products of exponentials, see Tao (2011) and Appendix C.1 of Lezcano-Casado and Martınez-Rubio (2019).

A subgroup $H$ of a Lie group $G$ is called a “Lie subgroup” when $H$ is an immersed submanifold of $G$ and the group operations are smooth when restricted to $H$. An immersed submanifold does not necessarily inherit its topology as a subset of $G$, but rather $H$ has a topology and smooth structure such that the inclusion $\imath_{H}:H\hookrightarrow G$ is smooth and its derivative is injective (see Lee (2013)). The tangent space to a Lie subgroup $H\subset G$ at the identity, defined as $T_{e}H=\operatorname{Range}(\operatorname{\mathrm{d}}\imath_{H}(I))\subset\operatorname{Lie}(G)$, is closed under the Lie bracket and thus forms a “Lie subalgebra” of $\operatorname{Lie}(G)$, denoted $\operatorname{Lie}(H)$. Conversely, a remarkable result stated by Theorem 19.26 in Lee (2013) shows that any subalgebra ${\mathfrak{h}}\subset\operatorname{Lie}(G)$, that is, any subspace closed under the Lie bracket corresponds to a unique connected Lie subgroup $H\subset G_{0}$ satisfying $\operatorname{Lie}(H)={\mathfrak{h}}$. Later on, we will use this fact to identify the connected subgroups of symmetries of machine learning models based on infinitesimal criteria. Another remarkable and useful fact is the “closed subgroup theorem” stated as Theorem 20.12 in Lee (2013). It says that if $H\subset G$ is a closed subset and is closed under the group operations of $G$, then $H$ is automatically an embedded Lie subgroup of $G$. Interestingly, while a Lie subgroup $H\subset GL(n)$ need not be a closed subset, it turns out that $H$ can always be embedded as a closed subgroup in a larger $\operatorname{GL}(n^{\prime})$, $n^{\prime}\geq n$ thanks to Theorem 9 in Gotô (1950).

A Lie group homomorphism is a smooth map $\Phi:G_{1}\to G_{2}$ between Lie groups that respects the group product, that is,

$$\Phi(g_{1}g_{2})=\Phi(g_{1})\cdot\Phi(g_{2}).$$

(12)

The tangent map $\phi:=\operatorname{\mathrm{d}}\Phi(e):T_{e}G_{1}\cong{\mathfrak{g}}_{1}\to T_{e}G_{2}\cong{\mathfrak{g}}_{2}$ is a Lie algebra homomorphism by Theorem 8.44 in Lee (2013), meaning that it is a linear map respecting the Lie bracket:

$$\phi\big{(}[\xi_{1},\xi_{2}]\big{)}=\big{[}\phi(\xi_{1}),\phi(\xi_{2})\big{]}.$$

(13)

Moreover, (see Proposition 20.8 in Lee (2013)) the Lie group homomorphism and its induced Lie algebra homomorphism are related by the exponential maps on $G_{1}$ and $G_{2}$ via the identity

$$\Phi\big{(}\exp(\xi)\big{)}=\exp\big{(}\phi(\xi)\big{)}.$$

(14)

Another fundamental result (Theorem 20.19 in Lee (2013)) is that any Lie algebra homomorphism $\operatorname{Lie}(G_{1})\to\operatorname{Lie}(G_{2})$ corresponds to a unique Lie group homomorphism $G_{1}\to G_{2}$ when $G_{1}$ is simply connected. When $G_{2}$ is the general linear group on a vector space, then the Lie group and Lie algebra homomorphisms are called Lie group and Lie algebra “representations”.

A Lie group $G$ can act on a vector space ${\mathcal{V}}$ via a representation $\Phi:G\to\operatorname{GL}({\mathcal{V}})$ according to

$$\theta:(x,g)\mapsto\Phi(g^{-1})x,$$

(15)

with $x\in{\mathcal{V}}$ and $g\in G$. More generally, a nonlinear right action of a Lie group $G$ on a manifold ${\mathcal{M}}$ is any smooth map $\theta:{\mathcal{M}}\times G\to{\mathcal{M}}$ satisfying

$$\theta(\theta(x,g_{1}),g_{2})=\theta(x,g_{1}g_{2})\qquad\mbox{and}\qquad\theta(x,e)=x$$

(16)

for every $x\in{\mathcal{M}}$ and $g_{1},g_{2}\in G$. Figure 1 depicts the action of a Lie group on a manifold. We make frequent use of the maps $\theta_{g}=\theta(\cdot,g)$, which have smooth inverses $\theta_{g^{-1}}$, and the “orbit maps” $\theta^{(x)}=\theta(x,\cdot)$. For example, using a representation $\Phi:\operatorname{SE}(3)\to\operatorname{GL}(\mathbb{R}^{7})$, the position $q$ and velocity $v$ of a particle in $\mathbb{R}^{3}$ can be rotated and translated via the action

$$\theta\left(\begin{bmatrix}Q&b\\ 0&1\end{bmatrix},\ \begin{bmatrix}q\\ v\\ 1\end{bmatrix}\right)=\Phi\left(\begin{bmatrix}Q^{T}&-Q^{T}b\\ 0&1\end{bmatrix}\right)\begin{bmatrix}q\\ v\\ 1\end{bmatrix}=\begin{bmatrix}Q^{T}&0&-Q^{T}b\\ 0&Q^{T}&0\\ 0&0&1\end{bmatrix}\begin{bmatrix}q\\ v\\ 1\end{bmatrix}=\begin{bmatrix}Q^{T}(q-b)\\ Q^{T}v\\ 1\end{bmatrix}.$$

The positions and velocities of $n$ particles arranged as a vector $(q_{1},\ldots,q_{n},v_{1},\ldots,v_{n},1)$ can be simultaneously rotated and translated via an analogous representation $\Phi:\operatorname{SE}(n)\to\operatorname{GL}(\mathbb{R}^{6n+1})$.

The key fact about a group action is that it is almost completely characterized by a linear map called the infinitesimal generator. This map $\hat{\theta}$ assigns to each element $\xi\in\operatorname{Lie}(G)$ in the Lie algebra, a vector field $\hat{\theta}(\xi)$ on ${\mathcal{M}}$ defined by

$$\hat{\theta}(\xi)_{x}=\left.\operatorname{\frac{\operatorname{\mathrm{d}}}{\operatorname{\mathrm{d}}t}}\right|_{t=0}\theta_{\exp(t\xi)}(x)=\operatorname{\mathrm{d}}\theta^{(x)}(e)\xi.$$

(17)

The infinitesimal generator and its relation to the group action are illustrated in Figure 1. For the linear action in (15), the infinitesimal generator is the linear vector field given by the matrix-vector product $\hat{\theta}(\xi)_{x}=-\phi(\xi)x$. Crucially, the flow of the generator recovers the group action along the exponential curve $\exp(t\xi)$, i.e.,

$$\operatorname{Fl}_{\hat{\theta}(\xi)}^{t}(x)=\theta_{\exp(t\xi)}(x).$$

(18)

For the linear right action in (15), this is easily verified by differentiation, applying (14), and the fact that solutions of smooth ordinary differential equations are unique. For a nonlinear right action this follows from Lemma 20.14 and Proposition 9.13 in Lee (2013).

Enforcing symmetry in multilayer percetrons was studied by Finzi et al. (2021). They provide a practical method based on enforcing linear constraints on the weights defining each layer of a neural network. The network uses specialized nonlinearities that are automatically equivariant, meaning that the constraints need only be enforced on the linear component of each layer. We show that the constraints derived by Finzi et al. (2021) are the same as those given by Theorem 3.

Specifically, each linear layer $F^{(l)}:{\mathcal{V}}_{l-1}\to{\mathcal{V}}_{l}$, for $l=1,\ldots,L$, is defined by

$$F^{(l)}(x)=W^{(l)}x+b^{(l)},$$

(28)

where $W^{(l)}$ are weight matrices and $b^{(l)}$ are bias vectors. Defining group representations $\Phi_{l}:G\to\operatorname{GL}({\mathcal{V}}_{l})$ for each layer, yields fundamental operators given by

	$\displaystyle{\mathcal{K}}_{g}F^{(l)}(x)-F^{(l)}(x)$	$\displaystyle=\big{(}\Phi_{l}(g)W^{(l)}\Phi_{l-1}(g)^{-1}-W^{(l)}\big{)}x+\Phi_{l}(g)b^{(l)}-b^{(l)}$		(29)
	$\displaystyle{\mathcal{L}}_{\xi}F^{(l)}(x)$	$\displaystyle=\big{(}\phi_{l}(\xi)W^{(l)}-W^{(l)}\phi_{l-1}(\xi)\big{)}x+\phi_{l}(\xi)b^{(l)}.$		(30)

Let $\{\xi_{i}\}_{i=1}^{q}$ generate $\operatorname{Lie}(G)$ and let $\{g_{j}\}_{j=1}^{n_{G}-1}$ consist of an element from each non-identity component of $G$. Using the fundamental operators and Theorem 3, it follows that the layer $F^{(l)}$ is $G$-equivariant if and only if the weights and biases satisfy

$$\phi_{l}(\xi_{i})W^{(l)}=W^{(l)}\phi_{l-1}(\xi_{i}),\quad\mbox{and}\quad\Phi_{l}(g_{j})W^{(l)}=W^{(l)}\Phi_{l-1}(g_{j}),$$

(31)

$$\phi_{l}(\xi_{i})b^{(l)}=0,\quad\mbox{and}\quad\Phi_{l}(g_{j})b^{(l)}=b^{(l)}$$

(32)

for every $i=1,\ldots,q$ and $j=1,\ldots,n_{g}-1$. These are the same as the linear constraints one derives using the method by Finzi et al. (2021). The equivariant linear layers are then combined with specialized equivariant nonlinearities $\sigma^{(l)}:{\mathcal{V}}_{l}\to{\mathcal{V}}_{l}$ to produce an equivariant network

$$F=\sigma^{(L)}\circ F^{(L)}\circ\cdots\circ\sigma^{(1)}\circ F^{(1)}:{\mathcal{V}}_{0}\to{\mathcal{V}}_{L}.$$

(33)

The composition of equivariant functions is equivariant, as one can easily check using Definition 2.

Enforcing symmetry in neural networks acting on spatial fields has been studied extensively by Weiler et al. (2018); Cohen et al. (2018); Esteves et al. (2018); Kondor and Trivedi (2018); Cohen et al. (2019) among others. Many of these techniques use integral operators to define equivariant linear layers, which are coupled with equivariant nonlinearities, such as the gated nonlinearities proposed by Weiler et al. (2018). Networks built by composing integral operators with nonlinearities constitute a large class of “neural operators” described by Kovachki et al. (2023); Goswami et al. (2023); Boullé and Townsend (2023). The key task is to identify appropriate bases for equivariant kernels. For certain groups, such as the Special Euclidean group $G=\operatorname{SE}(3)$, bases can be constructed explicitly using spherical harmonics, as in Weiler et al. (2018). We show that equivariance with respect to arbitrary group actions can be enforced via linear constraints on the integral kernels derived using the fundamental operators introduced in Section 5. Appropriate bases of kernel functions can then be constructed numerically by computing an appropriate nullspace, as is done by Finzi et al. (2021) for multilayer perceptrons.

For the sake of simplicity we consider integral operators acting on vector-valued functions $F:\mathbb{R}^{m}\to{\mathcal{V}}$, where ${\mathcal{V}}$ is a finite-dimensional vector space. Later on in Section 11.4 we study higher-order integral operators acting on sections of vector bundles. If ${\mathcal{W}}$ is another finite-dimensional vector space, an integral operator acting on $F$ to produce a new function $\mathbb{R}^{n}\to{\mathcal{W}}$ is defined by

$${\mathcal{T}}_{K}F(x)=\int_{\mathbb{R}^{m}}K(x,y)F(y)\operatorname{\mathrm{d}}y,$$

(34)

where the “kernel” function $K$ provides a linear map $K(x,y):{\mathcal{V}}\to{\mathcal{W}}$ at each $(x,y)\in\mathbb{R}^{n}\times\mathbb{R}^{m}$. In other words, the kernel is a function on $\mathbb{R}^{n}\times\mathbb{R}^{m}$ taking values in the tensor product space ${\mathcal{W}}\otimes{\mathcal{V}}^{*}$, where ${\mathcal{V}}^{*}$ denotes the algebraic dual of ${\mathcal{V}}$. Many of the neural operator architectures described by Kovachki et al. (2023); Goswami et al. (2023); Boullé and Townsend (2023) are constructed by composing layers defined by integral operators (34) with nonlinear activation functions, usually acting pointwise. The kernel functions $K$ are optimized during training of the neural operator.

With group actions defined by representations on $\mathbb{R}^{m},\mathbb{R}^{n},{\mathcal{V}},{\mathcal{W}}$, functions $F:\mathbb{R}^{m}\to{\mathcal{V}}$ transform according to

$${\mathcal{K}}_{g}^{(\mathbb{R}^{m},{\mathcal{V}})}F(x)=\Phi_{{\mathcal{V}}}(g)F(\Phi_{\mathbb{R}^{m}}(g)^{-1}x)$$

(35)

for $g\in G$. Likewise, functions $\mathbb{R}^{n}\to{\mathcal{W}}$ transform via an analogous operator ${\mathcal{K}}_{g}^{(\mathbb{R}^{n},{\mathcal{W}})}$.

By changing variables in the integral, the operator on the left is given by

$${\mathcal{K}}_{g}^{(\mathbb{R}^{n},{\mathcal{W}})}\circ{\mathcal{T}}_{K}\circ{\mathcal{K}}_{g^{-1}}^{(\mathbb{R}^{m},{\mathcal{V}})}F(x)=\int_{\mathbb{R}^{m}}{\mathcal{K}}_{g}K(x,y)F(y)\operatorname{\mathrm{d}}y,$$

(37)

where

$${\mathcal{K}}_{g}K(x,y)=\Phi_{{\mathcal{W}}}(g)K\big{(}\Phi_{\mathbb{R}^{n}}(g)^{-1}x,\Phi_{\mathbb{R}^{m}}(g)^{-1}y\big{)}\Phi_{{\mathcal{V}}}(g)^{-1}\det\big{[}\Phi_{\mathbb{R}^{m}}(g)^{-1}\big{]}.$$

(38)

The following result provides equivariance conditions in terms of the kernel, generalizing Lemma 1 in Weiler et al. (2018).

The Lie derivative of a continuously differentiable kernel function is given by

$${\mathcal{L}}_{\xi}K(x,y)=\phi_{{\mathcal{W}}}(\xi)K(x,y)-K(x,y)\phi_{{\mathcal{V}}}(\xi)-K(x,y)\operatorname{Tr}[\phi_{\mathbb{R}^{m}}(\xi)]\\ -\frac{\partial K(x,y)}{\partial x}\phi_{\mathbb{R}^{n}}(\xi)x-\frac{\partial K(x,y)}{\partial y}\phi_{\mathbb{R}^{m}}(\xi)y$$

(40)

The operators ${\mathcal{K}}_{g}$ and ${\mathcal{L}}_{\xi}$ are the fundamental operators from Section 5 because the transformation law for the kernel can be written as

$${\mathcal{K}}_{g}K=\Phi_{{\mathcal{W}}\otimes{\mathcal{V}}^{*}}(g)K\circ\Phi_{\mathbb{R}^{m}\times\mathbb{R}^{m}}(g)^{-1},$$

(41)

where

	$\displaystyle\Phi_{\mathbb{R}^{n}\times\mathbb{R}^{m}}(g)$	$\displaystyle:(x,y)\mapsto\left(\Phi_{\mathbb{R}^{n}}(g)x,\Phi_{\mathbb{R}^{m}}(g)y\right),$		(42)
	$\displaystyle\Phi_{{\mathcal{W}}\otimes{\mathcal{V}}^{*}}(g)$	$\displaystyle:T\mapsto\Phi_{{\mathcal{W}}}(g)T\Phi_{{\mathcal{V}}}(g)^{-1}\det\big{[}\Phi_{\mathbb{R}^{m}}(g)^{-1}\big{]}$

are representations of $G$ in $\mathbb{R}^{m}\times\mathbb{R}^{m}$ and ${\mathcal{W}}\otimes{\mathcal{V}}^{*}$.

As an immediate consequence of Theorem 3, we have the following corollary establishing linear constraints for the kernel to produce an equivariant integral operator.

These linear constraint equations must be satisfied to enforced equivariance with respect to a known symmetry $G$ in the machine learning process. By discretizing the operators ${\mathcal{K}}_{g}$ and ${\mathcal{L}}_{\xi}$, as discussed later in Section 10, one can solve these constraints numerically to construct a basis of kernel functions for equivariant integral operators.

As an immediate consequence of Theorem 4, the following result shows that the Lie derivative of the kernel encodes the continuous symmetries of a given integral operator.

This result will be useful for methods that promote symmetry of the integral operator, as we describe later in Section 8.

We begin by studying the symmetries of smooth submanifolds ${\mathcal{M}}$ of Euclidean space $\mathbb{R}^{d}$ using an approach similar to Cahill et al. (2023). However, we use a different operator that generalizes more naturally to nonlinear group actions on arbitrary manifolds (see Section 12) and recovers the Lie derivative (see Section 7.2). With right action $\theta:\mathbb{R}^{d}\times G\to\mathbb{R}^{d}$ of a Lie group, we define invariance of a submanifold as follows:

The subgroup of symmetries of a submanifold is characterized by the following theorem.

The meaning of this result and its practical use for detecting symmetry are illustrated in Figure 3.

To reveal the connected component of $\operatorname{Sym}_{G}({\mathcal{M}})$, we let $P_{z}:\mathbb{R}^{d}\to\mathbb{R}^{d}$ be a family of linear projections onto $T_{z}{\mathcal{M}}\subset\mathbb{R}^{d}$. These are assumed to vary continuously with respect to $z\in{\mathcal{M}}$. Then under the assumptions of the above theorem, $\operatorname{\mathfrak{sym}}_{G}({\mathcal{M}})$ is the nullspace of the symmetric, positive-semidefinite operator $S_{{\mathcal{M}}}:\operatorname{Lie}(G)\to\operatorname{Lie}(G)$ defined by

$$\big{\langle}\eta,\ S_{{\mathcal{M}}}\xi\big{\rangle}_{\operatorname{Lie}(G)}=\int_{{\mathcal{M}}}\hat{\theta}(\eta)_{z}^{T}(I-P_{z})^{T}(I-P_{z})\hat{\theta}(\xi)_{z}\ \operatorname{\mathrm{d}}\mu(z)$$

(48)

for every $\eta,\xi\in\operatorname{Lie}(G)$. We see in Figure 3 that $(I-P_{z})\hat{\theta}(\xi)_{z}$ measures the component of the infinitesimal generator not tangent to the submanifold at $z$. Here, $\mu$ is any strictly positive measure on ${\mathcal{M}}$ that makes all of these integrals finite. The above formula is useful for computing the matrix of $S_{{\mathcal{M}}}$ in an orthonormal basis for $\operatorname{Lie}(G)$.

Alternatively, when the dimension of $G$ is large, one can compute the nullspace using a Krylov algorithm such as the one described in Finzi et al. (2021). Such algorithms rely solely on queries of $S_{{\mathcal{M}}}$ acting on vectors $\xi\in\operatorname{Lie}(G)$. When $\theta_{g}(z)=\Phi(g^{-1})z$ and $\hat{\theta}(\xi)_{z}=-\phi(\xi)z$ are given by a Lie group representation (see Section 4.2), then the operator defined in (48) is given explicitly by

$$S_{{\mathcal{M}}}\xi=\int_{{\mathcal{M}}}\operatorname{\mathrm{d}}\Phi(e)^{*}\left[(I-P_{z})^{T}(I-P_{z})\phi(\xi)zz^{T}\right]\ \operatorname{\mathrm{d}}\mu(z),$$

(49)

where $\operatorname{\mathrm{d}}\Phi(e)^{*}:\mathbb{R}^{d\times d}\to\operatorname{Lie}(G)$ is the adjoint of $\operatorname{\mathrm{d}}\Phi(e):\operatorname{Lie}(G)\to\mathbb{R}^{d\times d}$. If $G\subset\mathbb{R}^{d\times d}$ is a matrix Lie group and $\Phi$ is the identity representation, then $\operatorname{\mathrm{d}}\Phi(e)$ is the injection $\operatorname{Lie}(G)\hookrightarrow\mathbb{R}^{d\times d}$. When $\operatorname{Lie}(G)\subset\mathbb{R}^{d\times d}$ inherits its inner product from $\mathbb{R}^{d\times d}$, then $\operatorname{\mathrm{d}}\Phi(e)^{*}$ is the orthogonal projection of $\mathbb{R}^{d\times d}$ onto $\operatorname{Lie}(G)$. For example, if $\Phi$ is the identity representation of $\operatorname{SE}(d)$ in $\mathbb{R}^{d+1}$ with the inner product on $\operatorname{\mathfrak{se}}(d)\subset\mathbb{R}^{(d+1)\times(d+1)}$ given by the usual inner product of matrices $\langle M_{1},M_{2}\rangle=\operatorname{Tr}(M_{1}^{T}M_{2})$, then it can be readily verified that

$$\operatorname{\mathrm{d}}\Phi(e)^{*}\left(\begin{bmatrix}A&b\\ c^{T}&a\end{bmatrix}\right)=\begin{bmatrix}\frac{1}{2}(A-A^{T})&b\\ 0&0\end{bmatrix},\qquad A\in\mathbb{R}^{d\times d},\ b\in\mathbb{R}^{d},\ c\in\mathbb{R}^{d},\ a\in\mathbb{R}.$$

(50)