RotEqNet: Rotation-Equivariant Network for Fluid Systems with Symmetric High-Order Tensors

Following [37], fix a vector space $V$ of dimension $n$ over $\mathbb{R}$. A tensor product $V\otimes V$ is a vector space with the property that $\mathbb{R}$-bilinear maps $V\times V\rightarrow\mathbb{R}$ are in natural one-to-one correspondence with $\mathbb{R}$-linear maps $V\otimes V\rightarrow\mathbb{R}$.

The tensor product $V\otimes V$ can be constructed as the quotient vector space $V\times V/C$, where $C$ is generated by vectors of the following types

where $x$ and $y$ are vectors in $V$ and $a$ is a scalar in $\mathbb{R}$. This means any element in $C$ can be written as a linear combination of vectors of the above form. $C$ is not necessarily a vector space of finite dimension. But the quotient space $V\otimes V$ is. Let $g:V\times V\rightarrow V\otimes V$ be the natural projection map, then we use $x\otimes y$ to denote the image of $(x,y)$ under $g$.

Let $\langle e_{1},\cdots,e_{n}\rangle$ be a basis of $V$, then $e_{i}\otimes e_{j}$ for $i=1,...,n$ and $j=1,...,n$ form a basis of $V\otimes V$. This means any vector $p\in V\otimes V$ can be written as

for some $a_{ij}\in\mathbb{R}$.

Here are some relations of tensors which come directly as a consequence of the relations generating $C$:

The representation of a tensor in $V\otimes V$ is similar to the representation of a linear map $V\rightarrow V$, i.e. a matrix. In fact, there is a natural way to think of a tensor as a linear map:

For each element $e_{i}\otimes e_{j}$ in the basis of $V\otimes V$, we can think of it as a linear map $V\rightarrow V$ by defining $e_{i}\otimes e_{j}(v)=e_{i}<e_{j},v>$, where $<,>$ is the natural inner product on $V$. Extend the definition linearly to every element in $V\otimes V$, we obtain a way to identify $V\otimes V$ as the space of linear map $V\rightarrow V$. In fact, the tensor $\sum_{i,j}a_{i,j}e_{i}\otimes e_{j}$ corresponds to the linear map represented by the matrix $[a_{ij}]$.

We have defined the tensor product $V\otimes V$ over $V$. The definition/construction of order $k$ tensor $\overbrace{V\otimes\cdots\otimes V}^{k}$ follows the same course. We will denote order $k$ tensor by $\otimes^{k}V$.

The basis of $\otimes^{k}V$ is given by $e_{i1}\otimes\cdots\otimes e_{ik}$, where $i=1,...,n$ and $j=1,...,k$. With respect to this basis, any order $k$ tensor can be written as $\sum_{i1,\cdots,ik}a_{i1,...,ik}e_{i1}\otimes\cdots\otimes e_{ik}$. Analogous to the order 2 case, we can think of an order $k$ tensor as a $k$-dimensional matrix, the typical way a tensor in physical experiments are represented.

We will use $T^{k}$ to denote a tensor of order $k$, i.e. a vector in $\otimes^{k}V$. $k$ is called the rank of the tensor.

A linear transformation on higher-order tensor is a generalization of a linear transformation on the first-order tensor, i.e. a vector.

Let $g:V\rightarrow V$ be a linear transformation. Use the basis $\langle e_{1},\cdots,e_{n}\rangle$ of $V$, we can represent this expression with the equation

Let $M(g)$ denote the matrix representation of $g$ with respect to the basis $\langle e_{1},\cdots,e_{n}\rangle$. Then

i.e. the transpose of the matrix $[a_{ij}]$

The map $g$ naturally induces a map $\otimes^{k}g$ on $\otimes^{k}V$. On the basis element $e_{i1}\otimes\cdots\otimes e_{ik}$, the action of $\otimes^{k}g$ is defined as

For any tensor $T\in\otimes^{k}V$, we will use $g(T)$ to denote the extension of $g$ on $\otimes^{k}V$

There is a convenient way to represent a linear transformation of 2-tensor as matrix multiplication.

For a 2-tensor $T=\sum_{i,j}b_{ij}e_{i}\otimes e_{j}$, use $M(T)$ be the matrix whose $(i,j)$ term is $b_{ij}$.

The proof here of Lemma 2.8 and Remark 2.9 are left in 5.

Lemma 2.8 and remark 2.9 will be used in the proof of Theorem 3.4. As we have shown in this subsection, one could use a matrix form of rotation operation with certain rules of matrix multiplication to perform a rotation on the tensor. In the following proofs of this paper, we applied this idea to perform rotation operation on tensors via matrix multiplication.

Let $\langle,\rangle$ be the standard inner product on $V$. Using this inner product, we can define the contraction of a tensor. It ”merges” vectors on the specified axes using the inner product and reduces the rank of the tensor by 2. Formally, let $C(a,b)$ denote the contraction along axis-$a$ and axis-$b$. Here, the axis means the ordinal of $V$ in $\otimes^{k}V$. For example, axis-$1$ refers to the first copy of $V$ in $\otimes^{k}V$.

On the element $\otimes_{j=1}^{k}v_{ij}$, $C(a,b)$ acts on it by pairing $v_{ia}$ and $v_{ib}$ via the inner product $\langle,\rangle$, i.e.

where $\check{v}$ means $v$ is not present.

We can then define $C(a,b)$ on $\otimes^{k}V$ by extending linearly. When $k=2$, contraction is nothing other than taking the trace of the corresponding matrix.

Lemma 2.11 shows an interesting connection between rotation operation and contraction. To understand this lemma, it represents that the contraction of a tensor is compatible with a linear transformation if this linear transformation is a rotation. This is an important lemma which is the foundation of the entire analysis in this paper. We would further utilize this lemma for extracting its rotation operation from higher (arbitrary) orders. We show the proof in 5.

Given a symmetric tensor of even order $T^{n}\in\otimes^{n}V$ ($n$ is even). Let $\mathcal{C}$ denote a sequence of contraction along the first two axes until we reach a second-order tensor. Applying $\mathcal{C}$ to $T^{n}$ we get:

Then we find the orthonormal eigen-vectors of $T^{2}$ and use them to form the orthonormal matrix $R$ that diagonalize $T^{2}$

Since $R$ is an orthonormal matrix, we have

We will call $D$ the standard position of $T^{2}$. We write $R(T^{2})=D$ to shorten the notation

Since contraction and rotation are compatible by Theorem 2.11. We can apply $R$ to $T^{n}$ before we apply contraction, and we will have

For the even tensor $T^{n}$, we define

Why would it be different for even order and odd order? Since odd order cannot directly reduce its dimension to 2 by contraction. Due to the fact that each contraction will reduce the dimensionality by 2, the reduced dimension will also be an odd number, which cannot be 2. Involving in this problem, this would further be impossible to extract the rotation matrix, which is impossible to rotate the tensor into a standard position. The following described is the method that we use to solve the problem.

Given a symmetric tensor of an odd order tensor $T^{n}\in\otimes^{n}V$ ($n$ is odd). Let $\mathcal{C}$ denote a sequence of contraction along the first two axes until we reach a third-order tensor. Applying $\mathcal{C}$ to $T^{n}$ we get:

After we obtain $T^{3}$, we could obtain three different order one tensors by contracting it. Name the contracted results, which are first-order tensors i.e. vectors, $V_{1},V_{2},V_{3}$. We could get an order two tensor by concatenating them.

Then, we could perform the a similar process as before. We perform QR factorization to obtain rotation matrix $R$.

For odd tensor, we define:

The pseudocode of our proposed algorithm is documented in Algorithm 1. We evaluate our method of position standardization algorithm in Section 4.

We provide a proof in Appendix 5.4. We call $S(T)$ the standard position of $S(T)$

Using Theorem 3.4 we can automatically have the result on tensor with arbitrary order that the standard position, derived by position standardization algorithm, is rotation invariant.

it has shown that Algorithm 1 is able to preserve rotation information for low dimension, and further extract using diagonalization for matrices. This part is a theoretical guarantee of our position standardization algoriTheorem

Newtonian fluid is a type of fluid such that its viscous stress changes based on its flow. In this experiment, we aim to use simulation data to demonstrate this rule of Newtonian fluid. This would serve as a case study with second-order linear equations. Let $\sigma\in\mathbb{R}^{3\times 3}$ be stress tensor, $p\in\mathbb{R}$ pressure and $S\in\mathbb{R}^{3\times 3}$ strain rate. The rule of Newtonian fluid is an second-order physical equation which satisfies the following condition [43]:

Another definition of the equation for Newtonian fluid would use the velocity vector field, defined as $\nabla v$. $\nabla v$ could be expressed as a $3\times 3$ matrix. Using this definition, the equation of Newtonian fluid could also be written as:

This could also be considered as the definition of strain rate Based on this definition, we could observe that $S=\nabla v+\nabla v^{T}$, and it is symmetric since $S=S^{T}$. Since $S$ is symmetric and $I$ is an identity matrix, $\sigma$ is also symmetric. Therefore, defining an arbitrary rotation matrix $R$, this system is equipped with the property of rotation-equivariant that $R(\sigma)=R(-pI+\mu S)$.

To quantify the stress for Newtonian fluid simulation, it would be useful to be able to predict the Newtonian fluid stress, given the simulation of pressure and velocity vector field. Based on this scenario, in this subsection, we provide a case study for the machine learning model on inputting the shear of Newtonian fluid and prediction of the stress.

Based on Eqn.4.2, we first generate random data to obtain $\nabla v$ and $p$. The generation of random numbers in $\nabla v$ follows a normal distribution from range $(0,1)$. Derived from generated $\nabla v$ and $p$, we could obtain $\sigma$ from Eqn.4.2. Denote the dataset as $D=\{x_{i},y_{i}\}_{i=1}^{N}$. To form a proper dataset $D$ with $N$ elements for a machine learning model for Newtonian fluid, the input $x$ is set up to be a vector where $x\in\mathbb{R}^{10}$. Specifically, $x$ is composed by $p$ and flattened $S$ in Eqn.4.1. The output $y\in\mathbb{R}^{9}$ is a vector which is the flattened result of matrix $\sigma$ derived by $p$ and $S$. The dataset $D$ would set up in the description above. To compare the difference of our method to the baseline method, we trained two models with the same hyper-parameter using different amounts of training data, ranging from $10,000,20,000,...,100,000$. $85\%$ of generated data is used for training and $15\%$ of data is used for testing. A rotation set with 10,000 random rotation matrices is also generated for evaluating the property of rotation-equivariant, denoted by $\{R_{i}\}_{i=1}^{10000}$.

The machine learning model we apply here is neural networks and random forests because of the ability of these two models to approximate arbitrary functions. For Neural Networks, in our implementation, the logistic activation function is used as an activation function for every neuron. The number of neurons for two layers is 512 and 4, respectively. Adam optimizer [44] is applied as the algorithm for optimization, and the learning is set up to be $1\times 10^{-3}$. We also set the batch size to be 64. For random forests, 100 estimators are set up with mean squared error as the criterion. The maximum depth of random forests is 3 to lower the chance of overfitting. We used Sklearn for implementation [45]. The computation environment of this experiment is CPU.

There are two properties to evaluate, including error reduction and rotation-equivariant of RotEqNet. The effect of error reduction is evaluated for the first. A kernel predictor is trained by standard positions derived from training data. Then, the prediction algorithm is applied to both training and testing set to obtain the training and testing performances. The validation error $E$ is defined as the Mean Squared Loss using the formulation that:

In Eqn. 4.3, $N$ is the number of data in dataset $D$, $M$ is the trained machine learning model, $\theta$ is the derived parameter from model $M$, and $(X_{i},y_{i})\in D$ describes input-label pair of the dataset. This evaluation $E$ represents the expected error of model $M$ with dataset $D$.

Fig. 4(a) shows the error reduction property of RotEqNet. This plot consists of three groups of experimental groups. The first experiment group focuses on the accuracy of the baseline model, a single feed-forward Neural Network, on raw data with random rotated positions. As shown in Fig. 4(a) with blue curves, triangle curve represents training error and circle curve represents testing error. The second experiment group is RotEqNet with Neural Network as the kernel predictor. As shown in Fig. 4(a) in red curves, triangle curve represents training error, and the circle curve represents testing error. For 100,000 training samples, the testing error of RotEqNet is 0.0037, and the testing error of the baseline method is 1.333. We could observe a huge error reduction, 99.56% in training, and 99.60% in testing, for RotEqNet compared to the error of using the baseline model. For the last experiment group, as performances marked as black curves in the figure, it reports the performance of kernel predictor with standard position only. This experiment would explain why RotEqNet would improve performance since training with standard positions would be an easier task compared to raw data.

Further, Fig. 4(b) shows the error reduction effect of RotEqNet using Random Forest as a kernel predictor. Similarly, as shown in Fig. 4(b) with blue curves, it represents the performance of the baseline method (Random Forests). The second experiment group is RotEqNet with Random Forests as the kernel predictor. As shown in Fig. 4(b) in red curves, triangle curve represents training error and the circle curve represents testing error. We could observe a huge error reduction, 99.56% in training and 99.72% in testing, for RotEqNet compared to the error of using only the Random Forest predictor. For the last experiment group, as performances marked as black curves in the figure, trains the kernel predictor with standard position only. As stated before, this could also serve as a reason for the error reduction effect for RotEqNet on random forests.

According to the reported results, RotEqNet has a good generalization result without overfitting. For cases training with raw data for baseline models, the testing error is typically higher compared to training error. For example, the difference is training and testing errors are 0.44 for Neural Networks, $1.01$ for Random forests when $N=100,000$. This represents that for both Neural Networks and Random Forests would be easy to overfit this task on Newtonian Fluid. By contrast, RotEqNet would help to reduce this difference in training and testing error. As we could observe from the training and testing error of RotEqNet, the errors are much lower. When $N=100,000$, there are only $0.0002$ for Neural Networks and $0.0078$ for Random Forests. In the case of linear second-order equations, the application of RotEqNet would result in better-generalized results in learning.

Another important property to evaluate for RotEqNet is rotation-equivariant. The experiment is designed on the definition of rotation-equivariant mentioned in Eqn. 2.18. First, we pick a randomly generated data $(X_{0},y_{0})$. Then we apply the rotation set $\{R_{i}\}_{i=1}^{10000}$ with 10,000 random rotation matrices to $(X_{0},y_{0})$. To fully investigate the property of rotation-equivariant, we apply an error evaluation method $E_{D}$ here to evaluate the error compared to real data, which is defined as:

This error evaluation method ($E_{D}$) focuses more on the model’s error on real data for all rotations. As shown in Tab. 2, for both baseline methods, using neural networks and random forests, there are large errors for $E_{M}$ and $E_{D}$. The baseline methods have no theoretical guarantee that it has the property of rotation-equivariant. However, there is an error reduction for both machine learning models when applying with RotEqNet’s architecture. Especially for RotEqNet with Neural Networks as kernel predictor, we could observe that $E_{D}=0.0013$ with $99.8\%$ of error reduction. This could guarantee the rotation-equivariant property of RotEqNet.

In this case, we consider the subgrid model of large eddy simulation (LES) of turbulent flow by Kosovic [46]. In this case study, as formulated previously in [47, 48], we hope to obtain a learned model by simulation data from LES. This would serve as a case study with second-order non-linear equations. LES is defined as:

Here $\tau_{ij}$ is subgrid stress, which is a symmetric traceless 2nd order tensor. $S_{ij}$ and $\Omega_{ij}$ are symmetric and anti-symmetric parts of velocity gradient tensor $G_{ij}$, where $\mathrm{Tr}(G)=0$. Further, $C_{s}$, $\Delta$, $C_{1}$, $C_{2}$ are all constants. The configuration of constants above are reported in the next subsection.

In order to qualify the subgrid stress for LES, this study aims to predict the subgrid stress, given the simulation of velocity gradient tensor. This case study for the machine learning model on inputting the velocity gradient tensor.

Based on Eqn. 4.5, we first generate random data to obtain a simulated velocity gradient tensor $G_{ij}$. The generation of random numbers follows a normal distribution from range $(0,1)$, and $G_{ij}$ is obtained from a random matrix $G_{raw}$ by subtracting $\frac{1}{3}\mathrm{Tr}(G_{raw})$ from diagonal position. This would keep $\mathrm{Tr}(G)=0$. $S_{ij}$ and $\Omega_{ij}$ could be obtained from $G_{ij}$ by getting its symmetric and anti-symmetric parts. For the setup of constants, $C_{s}=0.4,\Delta=0.4,C_{1}=C_{2}=1.0$. $tau_{ij}$ is computed from the above setting with Eqn. 4.2. Denote the dataset as, $D=\{x_{i},y_{i}\}_{i=1}^{N}$. To form a proper dataset $D$ with $N$ elements for a machine learning model for Newtonian fluid, the input $x$ is set up to be a vector where $x\in\mathbb{R}^{9}$. Specifically, $x$ is composed by flattened $G_{ij}$. The output $y\in\mathbb{R}^{9}$ is a vector, which is the flattened result of matrix $\tau$ derived by $G$ and other constants. To compare the difference of our method to the baseline method, we trained two models with the same hyper-parameter using different amounts of training data, ranging from $10,000,20,000,...,100,000$. $85\%$ of generated data is used for training, and $15\%$ of data is used for testing. A rotation set with 10,000 random rotation matrices is also generated for evaluating the property of rotation-equivariant, denoted by $\{R_{i}\}_{i=1}^{10000}$. The model setup is the same compared to Sec. 4.1.2.

The effect of error reduction is evaluated for the first. The validation error $E$ is defined as the Mean Squared Loss using the formulation in Eqn. 4.3. This evaluation $E$ represents the expected error of model $M$ with dataset $D$.

Fig. 5(a) shows the error reduction effect of RotEqNet with Neural Network as a kernel predictor for second-order nonlinear cases with three groups of experimental groups. The first experiment group focuses on the accuracy of the baseline method on raw data with random rotated positions. As shown in Fig. 5 with blue curves, triangle curve represents training error and the circle curve represents testing error. The second experiment group is RotEqNet, with Neural Network as a kernel predictor. As shown in Fig. 5(a) in red curves. For 100,000 training samples, the testing error of RotEqNet is 0.1391, and the testing error of the baseline method is 0.2946, with 52.77% of error reduction. The performances of the last experiment group are marked as black curves in the figure, with only standard position trained for kernel predictor.

Based on the experimental results, firstly, RotEqNet could reach a better learning performance compared to simply applying Neural Networks (baseline method). Training with standard positions could lower the training difficulty, and therefore RotEqNet could obtain better performance. Further, Fig. 5(b) shows the error reduction effect of RotEqNet using Random Forest as a kernel predictor. The general performance of using Random Forests as a kernel predictor is relatively worse compared to using Neural Networks as a kernel predictor. In Fig. 5(b), blue curves represent the performance of training with raw data by Random Forests (baseline method); red curves represent the performance of RotEqNet; black curves represent the performance of kernel predictor trained with standard positions. We could observe an error reduction for 36.63% in training and 57.58% in testing for RotEqNet with Random Forests.

Moreover, RotEqNet has a good generalization result without overfitting. Applying raw data in learning directly on baseline models, the testing error is much higher compared to the training error. For example, the difference is training and testing errors are $0.0068$ for Neural Networks, $0.1068$ for Random forests when $N=100,000$. It is also observable in Fig. 5(a) that the training error of the baseline model with raw data is the lowest, while the testing error of the baseline model is the highest. In this case study, Neural Networks are worse for the sake of overfitting compared to Random Forests. By contrast, introducing the architecture RotEqNet would help to reduce this difference in training and testing error. As we could observe from the training and testing error of RotEqNet, the errors are much lower. When $N=100,000$, there are only $0.0046$ for Neural Networks and $0.0022$ for Random Forests. In this case study of LES, the application of RotEqNet would result in better-generalized results in learning.

To evaluate the rotation-equivariant property of RotEqNet for second-order nonlinear cases, our experimental process is close to the one stated in Sec. 4.1.3. First, we pick a randomly generated data $(X_{0},y_{0})$. Then we apply the rotation set $\{R_{i}\}_{i=1}^{10000}$ with 10,000 random rotation matrices to $(X_{0},y_{0})$. This error evaluation method ($E_{D}$), as defined in Eqn. 4.4, focuses more on the model’s error on real data for all rotations. As shown in Tab. 4, for both baseline methods, using neural networks and random forests, there are large error for $E_{D}$. The baseline methods have no theoretical way to guarantee that it has the property of rotation-equivariant. However, there is an error reduction for both machine learning models when applying with RotEqNet’s architecture. Especially for RotEqNet with Neural Networks as kernel predictor, it is observable that $E_{D}=0.0025$ with $73.55\%$ error reduction. This could guarantee the rotation-equivariant property of RotEqNet for nonlinear second-order cases.

In this section, we study the performance of RotEqNet for tensor with odd order. In this case, we specifically set a third-order test equation. We used a test equation here revised from Newtonian fluid equation from Eqn. 4.2. We name this testing equation as ’testing Newtonian fluid equation’ for simplicity. The testing equation revised from Newtonian fluid equation can be described as:

where $\sigma\in\mathbb{R}^{3\times 3\times 3}$ is testing stress, $p\in\mathbb{R}$ is testing pressure, and $v\in\mathbb{R}^{3\times 3\times 3}$ is testing velocity field. $I\in\mathbb{R}^{3\times 3\times 3}$ is the identity third-order tensor.

Based on this testing equation, we could observe that $(\nabla v+\nabla v^{T})^{T}=\nabla v+\nabla v^{T}$. Since $\nabla v+\nabla v^{T}$ is symmetric, and $I$ is symmetric, we have $\sigma$ is also symmetric. Therefore, defining an arbitrary rotation matrix $R$, this system is equipped with the property of rotation-equivariant that $R(\sigma)=R(-pI+\mu(\nabla v+\nabla v^{T}))$.

In order to qualify stress for testing the Newtonian fluid equation, this study aims to predict the stress, given the simulation of pressure and velocity gradient tensor. This case study for the machine learning model on inputting the pressure and velocity gradient tensor.

Based on Eqn. 4.6, we first generate random data to obtain $\nabla v$ and $p$. The generation of random numbers in $\nabla v$ follows a normal distribution from range $(0,1)$. $\sigma$ could be obtained using the Eqn.4.6, derived from generated $\nabla v$ and $p$. Denote the dataset as , $D=\{x_{i},y_{i}\}_{i=1}^{N}$. To form a proper dataset $D$ with $N$ elements for a machine learning model for Newtonian fluid, the input $x$ is set up to be a vector where $x\in\mathbb{R}^{28}$. Specifically, $x$ is composed by $p$ and flattened $(\nabla v+\nabla v^{T})$ in Eqn.4.6. The output $y\in\mathbb{R}^{27}$ is a vector which is the flattened result of matrix $\sigma$. The dataset $D$ would set up in the description above. To compare the difference of our method to the baseline method, we trained two models with the same hyper-parameter using different amounts of training data, ranging from $10,000,20,000,...,100,000$. $85\%$ of generated data is used for training and $15\%$ of data is used for testing. A rotation set with 10,000 random rotation matrices is also generated for evaluating the property of rotation-equivariant, denoted by $\{R_{i}\}_{i=1}^{10000}$. The model setup is the same as Sec. 4.1.2.

Fig. 6(a) shows the error reduction effect of RotEqNet with Neural Network as a kernel predictor for third-order cases with three groups of experimental groups. The first experiment group focuses on the accuracy of the baseline model (Neural Network) on raw data with random rotated positions as shown in Fig. 6(a) with blue curves. The second experiment group is RotEqNet, with Neural Network as kernel predictor as shown in Fig. 6(a) in red curves. For 100,000 training samples, the testing error of RotEqNet is 1.8759 and the testing error of baseline method is 2.2232 with 15.62% of error reduction. The performances of the last experiment group are marked as black curves in the figure, with only standard position trained for kernel predictor.

Based on the experimental results, for the third-order testing equation, RotEqNet could reach a better learning performance compared to the baseline method. Training with RotEqNet could lower the training difficulty, and therefore RotEqNet could obtain better performance. Moreover, RotEqNet has good generalization capability without overfitting. As shown in the blue curves of Fig. 6, if we apply raw data in learning directly on baseline models, the testing error is much higher compared to the training error. In this case study, introducing the architecture RotEqNet would help to reduce this difference in training and testing error. As we could observe from the training and testing error of RotEqNet, the errors are much lower. When $N=100,000$, there are only $0.0046$ for Neural Networks and $0.0022$ for Random Forests. In this case study of testing the Newtonian fluid equation, the application of RotEqNet would result in better-generalized results in learning.

Further, Fig. 6(b) shows the error reduction effect of RotEqNet using Random Forest as a kernel predictor. The general performance of using Random Forests as a kernel predictor is relatively worse compared to using Neural Networks as a kernel predictor. In Fig. 5(b), blue curves represent the performance of training with raw data by Random Forests (baseline method); red curves represent the performance of RotEqNet; black curves represent the performance of Random Forest trained with standard positions. For the first point, we could observe an error reduction for 0.90% in training and 6.84% in testing for RotEqNet with Random Forests. As another point, RotEqNet is also obtaining a better-learned model for the model’s capability in generalization. The testing error of the baseline method is observably higher than testing error, while the training and testing performance of RotEqNet is approximately the same. As suggested in Fig. 6(a), in second-order nonlinear cases, RotEqNet could reach a generalized learning result with remarkably lower error compared to baseline methods.

To evaluate the rotation-equivariant property of RotEqNet for this third-order case, we designed an experimental process that is close to the one stated in Sec. 4.1.3. As shown in Tab. 6, for baseline method using neural networks, the error is relatively large for $E_{D}$ compared to RotEqNet. In our experiment, we reached an error reduction of $0.1462$. We would further discuss this result in Section 4.4.3.

This case study focuses on a linear relationship of fourth-order tensor. Nye [49] has introduced a fourth-order tensor in modeling elastic compliances and stiffnesses, which has been investigated using machine learning methods [50, 51]. Generally, in the study of the properties of a crystalline and anisotropic elastic medium, a fourth-order tensor coefficient will typically be applied to model the relationship between two symmetric second-order tensors [52]. In this case, we study Electrostriction, a property causing all electrical non-conductors to change their shape under the application of an electric field. The relationship is described as:

Here $T_{ij}\in\mathbb{R}^{3\times 3}$ is a symmetric traceless second-order strain tensor. $S_{kl}\in\mathbb{R}^{3\times 3},S_{kl}=S_{k}S_{l}$ where $V_{k}$ and $V_{l}$ are first-order electric polarization density. Note here $V_{kl}$ is symmetric. $V_{ijkl}\in\mathbb{R}^{3\times 3\times 3\times 3}$ is the electrostriction coefficient.

Based on the formulation above, this system is symmetric. Since $S_{ij}$ is symmetric, $T_{ij}^{T}=(V_{ijkl}S_{ij})^{T}=S_{kl}V_{klij}=T_{ij}$. This could guarantee that $T_{ij}$ is also symmetric. Due to the face that the system is symmetric, applying a random rotation matrix $R$, $R(T)=R(VS)$.

In order to qualify strain for study on Electrostriction, we aim to predict the strain, given the simulation of electrostriction coefficient and electric polarization density.

Based on Eqn. 4.7, we first generate random data to obtain simulated electrostriction coefficient tensor $V_{ijkl}$ and electric polarization density tensor $S_{ij}$. The generation of random numbers follows a normal distribution. $T_{ij}$ is computed from above setting using $V_{ijkl}$ and $S_{ij}$. Denote the dataset as, $D=\{x_{i},y_{i}\}_{i=1}^{N}$. To form a proper dataset $D$ with $N$ elements for a machine learning model for the study on Electrostriction, the input $x$ is set up to be a vector where $x\in\mathbb{R}^{90}$. Specifically, $T_{ij}$ is composed by flattened $V_{ijkl}$ and $S_{ij}$. The output $y\in\mathbb{R}^{9}$ is a vector, which is the flattened result of second-order tensor $T$. To compare the difference of our method to the baseline method, we trained two models with the same hyper-parameter using different amounts of training data, ranging from $10,000,20,000,...,100,000$. $85\%$ of generated data is used for training, and $15\%$ of data is used for testing. A rotation set with 10,000 random rotation matrices is also generated for evaluating the property of rotation-equivariant, denoted by $\{R_{i}\}_{i=1}^{10000}$. The model setup is the same compared to Sec. 4.1.2. We use Numpy to generate this simulated dataset by generating a random symmetric fourth-order tensor $V$, and second-order tensor $S$. $T$ is computed from generated $V$ and $S$ by Eqn. 4.7.

The effect of error reduction is evaluated for the first. The validation error $E$ is defined as the Mean Squared Loss using the formulation in Eqn. 4.3. This evaluation $E$ represents the expected error of model $M$ with dataset $D$. Fig. 7 shows the performance of Neural Networks and Random Forests as kernel predictor separately. It is observable that in high-order cases, Neural Networks have huge superiority to Random Forests. Details will be demonstrated in the following paragraphs.

We are starting with Neural Networks, Fig. 7(a) shows the error reduction effect of RotEqNet with Neural Network as a kernel predictor. As shown in blue curves, the first experiment group focuses on the accuracy of the baseline model on raw data with random rotated positions. The second experiment group is RotEqNet marked with red curves. As shown in black curves, it shows the performance of the kernel predictor trained by standard position. For 10,000 training samples, the testing error of RotEqNet is 4.0106 and the testing error of baseline model is 8.6458 with 53.61% of error reduction. The testing performance of the kernel predictor is only evaluated on the testing set with only standard positions. It will be helpful to explain the reason for the improved performance of RotEqNet.

To interpret the experimental results, firstly, RotEqNet could reach a better learning performance compared to simply applying Neural Networks (baseline method). A dataset with only standard positions has lower training difficulty compared to random positions. This claim is supported by black curves in Fig. 7(a), the performance of the kernel predictor is much better than the baseline model. RotEqNet could obtain better performance for utilizing rotation symmetry as a prior, and training kernel predictor with only standard positions. Moreover, RotEqNet has a good generalization result without clear overfitting. The training error and testing error of RotEqNet is considerably close to each other, and sometimes, the testing error of RotEqNet is even slightly better than its training error. By contrast, applying raw data in learning directly on $M_{baseline}$ would result in an overfitted model. The testing error is much higher compared to the training error. To demonstrate the improved learning result in generalization, for example, when $N=100,000$, the difference between training and testing errors for RotEqNet is only $0.0024$ while the difference of the baseline method is $2.1118$. As a quick conclusion, for Neural Networks as a kernel predictor, the application of RotEqNet would be better compared to the baseline method.

Further, Fig. 7(b) shows the error reduction effect of RotEqNet using Random Forest as a kernel predictor. At first glance, we could find that the curves for Random Forests are quite messy without certain patterns like Fig. 7(a). The general performance of using Random Forests as a kernel predictor is worse in both aspects of performance and generalization. In Tab. 7, we could observe a training error reduction for 0.58% and testing error reduction of 2.96%. Even if we could still see the general error of RotEqNet seems to be slightly lower than the baseline method. This result is not comparable to the error reduction performance with setting Neural Networks as a kernel predictor. As another point, selecting Random Forests as a kernel predictor fails to extract learning rules with the standard position. As we could observe the black curves in Fig. 7(b) is not showing an improved performance as good as using Neural Networks. Finally, the learned model of RotEqNet is also not getting a model with better generalization capability. There is no significant reduction of overfitting error compared to the baseline method.

To evaluate the rotation-equivariant property of RotEqNet for this fourth-order case, we designed an experimental process as stated in Sec. 4.1.3. The error evaluation measurement ($E_{D}$), as defined in Eqn. 4.4, focuses more on the model’s error on real data for all rotations. As shown in Tab. 8, when using neural networks, baseline method has large error for $E_{D}$. RotEqNet helps in keeping the rotation-equivariant property as observing error reduction in $E_{D}$ for $28.86\%$. Considering the case using Random Forests as a kernel predictor, as shown in the previous paragraph, because of the reason that Random Forests are relatively bad in learning fourth-order data, the performance of $E_{M}$ is still affected, which results in a large error in the prediction of RotEqNet with Random Forest.

The large error reduction observed in case studies raised new opportunities in solving the problem of the physical system with rotation symmetry. Most physical systems have the property of rotation symmetry, and currently, there exist few works that could provide a theoretical guarantee to this property for machine learning methods. A key point in this problem is to design a properly defined algorithm to obtain rotation invariant for high-order tensors. This paper has shown RotEqNet with theoretical and experimental results aiming to solve the problem of rotation symmetry.

We first define a standard position as rotation invariant, which is compatible for high-order tensors. It allows us to extract the rotation invariant of high-order tensors using a contraction, diagonalization, and QR factorization. The theoretical guarantee is shown in Thm. 3.14, and the algorithm is shown in Alg. 3.2.2. RotEqNet is built on Alg. 3.2.2 with a kernel predictor which only deals with standard positions (rotation invariants). By setting kernel predictor with Neural Networks and Random Forests, these two methods are compared with baseline methods in four different case studies focusing on second-order linear, second-order nonlinear, third-order linear, and fourth-order linear cases. There are three important points to address from the observation of case studies.

First, the definition of the standard position is successful. The definition of the standard position is not unique. We aim to define a proper version of the standard position to simplify the learning task by removing the effect of rotation symmetry. In our case, the standard position satisfies the definition of rotation-invariants, which selects a representative point from the orbit of an element via diagonalization (or QR factorization). The experimental results are compatible with this definition of the standard position. We could observe in most of the cases, training kernel predictors with only rotation invariants could reach the lowest error. The reduced error means that the rotation invariant in our definition could lower the difficulty of this learning task as we previously discussed the reason in Sec. 1.

Second, RotEqNet is equipped with the property of Rotation-equivariant. As we could observe from the results of case studies, the rotation error $E_{M}$ is typically low compared to baseline methods. The perseverance of the property of Rotation-equivariant shows the successful design of RotEqNet and the correctness of Thm. 3.14. Operating with Alg. 3.2.2, the property of Rotation-equivariant of RotEqNet could be held if and only if Thm. 3.14 is correct. Further, this fact would cause an error reduction for RotEqNet. As stated in the previous paragraph, training with rotation invariants will result in a lower error. Under this situation, adding with the property of Rotation-equivariant, this would cause RotEqNet could process this system with any rotation.

The two reasons above are the main reasons that are causing the error reduction for RotEqNet. There is also one point to mention is the selection of kernel predictor. The model selection of kernel predictor will affect the learning results significantly since the kernel predictor is essential in learning the physical system without the effect of rotation symmetry. Neural Networks is the best model in the design of the data-driven method for physical systems because of its flexibility to approximate arbitrary functions. We only reported the performance of Neural Networks and Random Forests as previous work by Ling [16]. As described in Sec. 4.4.3, the performance of Random Forests is limited compared to Neural Networks. Also, as a general trend in previous experiments, Neural Networks are usually reaching better performance compared to Random Forests. As a quick conclusion, we believe the application of Neural Networks as a kernel predictor has a series of advantages than other machine learning models.

We wish to further discuss about another error evaluation method of rotation-equivariance property that we do not mention in case studies. Consider a type of error evaluation, evaluating rotation error of model itself, the error $E_{M}$ is defined as:

The evaluation of this error is actually trivial since we have already rigorously provided a proof in Theorem 3.14 showing the rotation-equivariance property of RotEqNet. We applied this evaluation in first two case studies, and the estimated error is around $10^{-16}$ for all these cases.

For future work, there are three directions to this paper: a better definition of standard position, application to other groups, and generalization to non-symmetric systems. For the first direction, for the current definition, the rotation invariant of odd-order tensors is not reaching equivalent performance as even-order tensors. It would be a good work for revising the definition of standard position for odd tensors. Second, besides rotation symmetries, there are also physical systems with other group-equivariant properties such as scaling and transaction. This work could provide a method in solving problems with other groups, but the detailed design of an algorithm should differ from case to case. Third, current work could only deal with the symmetric system. However, for a general case, if the system is not symmetric, there are certain methods to use RotEqNet in a symmetric system for solving a non-symmetric system. A good trick to consider, for example, is to deal with $PP^{T}$, where $P$ is a matrix. This is a great intuition to extend our current work into non-symmetric physical systems.