Singular layer PINN methods for Burgers’ equation

We introduce an $L$-layer feed-forward neural network (NN) defined by the recursive expression: $\mathcal{N}^{l}$, $0\leq l\leq L$ such that

where $\boldsymbol{x}$ is an input vector in $\mathbb{R}^{N_{0}}$, and $\boldsymbol{W}^{l}\in\mathbb{R}^{N_{l}\times N_{l-1}}$ and $\boldsymbol{b}^{l}\in\mathbb{R}^{N_{l}}$, $l=1,\ldots,L$, are the (unknown) weights and biases respectively. Here, $N^{l}$, $l=0,\ldots,L$, is the number of neurons in the $l$-th layer, and $\sigma$ is the activation function. We set $\boldsymbol{\theta}=\{(\boldsymbol{W}^{l},\boldsymbol{b}^{l})\}_{l=1}^{L}$ as the collection of all the parameters, weights and biases.

To solve the Burgers equation (1.1), we use the NN structure above with the output $\mathcal{N}^{L}=\hat{u}(x,t;\boldsymbol{\theta})$ and the input $\boldsymbol{x}=(x,t)\in(-1,1)\times(0,\infty)$ where a proper treatment is implemented for the boundary condition $(i)$ or $(ii)$ in (1.1); see (4.5) and (4.11) below. In order to determine the predicted solution $\hat{u}$ of (1.1), we use the loss function defined by

where $\mathcal{T}\subset\Omega_{x}\times\Omega_{t}$ is a set of training data in the space-time domain, $\mathcal{T}_{b}\subset\partial\Omega_{x}\times\Omega_{t}$ on the boundary, and $\mathcal{T}_{0}\subset\Omega_{x}$ on the initial boundary at $t=0$. The function $l(\hat{u})$ in (3.1) is defined differently for different examples below to enforce the boundary conditions $(i)$ or $(ii)$ in (1.1); see (4.5) and (4.11) below.

In order to create our predicted solution for the Burgers equation (1.1), we revisit the interior layer analysis in Section 2 and then, using this analytic result, we develop two training solutions in distinct regions. More precisely, we partition the domain $\Omega_{x}$ into disjoint subdomains designated as $\Omega_{x,L}$ and $\Omega_{x,R}$, which are separated by the shock curve $x=\alpha(t)$. After that, we introduce the output functions $\hat{u}_{*}=\hat{u}_{*}(x,t;\boldsymbol{\theta}_{*})$ for the neural network in the subdomains $\Omega_{x,*}$ where $*=L,R$. The training solution for our sl-PINN method is then defined as:

Here $\tilde{\varphi}_{*}$ is the training form of the corrector $\varphi$ in (2.13), and it is given by

with the trainable parameters,

Note that the $\tilde{b}$ appears in (3.4) is an approximations of $b(t)=u^{\varepsilon}(\alpha(t),t)$ and it is updated iteratively during the network training; see Algorithm 1 below.

As noted in Section 2, the correctors $\tilde{\varphi}_{*}$, $*=L,R$, capture well the singular behavior of the viscous Burgers’ solution $u^{\varepsilon}$ near the shock of $u^{0}$ at $x=\alpha(t)$. Hence, by integrating these correctors into the training solution, we record the stiff behavior of $u^{\varepsilon}$ directly in (3.2).

To define the loss function for our sl-PINN method, we modify and use the idea introduced in the eXtended Physics-Informed Neural Networks (XPINNs), e.g., [17]. In fact, to manage the mismatched value across the shock curve as well as the residual of differential equations along the interface, we define the loss function as

Here the interface between $\Omega_{L}=(-\infty,\alpha(t))$ and $\Omega_{R}=(\alpha(t),\infty)$ is given by $\Gamma:=\{(x=\alpha(t),\,t)\in\Omega_{x}\times\Omega_{t}\,|\,\,t\in\Omega_{t}\}$, and the training data sets are chosen such that $\mathcal{T}_{*}\subset\Omega_{x,*}\times\Omega_{t}$, $\mathcal{T}_{*,b}\subset\partial\Omega_{x,*}\times\Omega_{t}$, $\mathcal{T}_{*,0}\subset\Omega_{x,*}$, and $\mathcal{T}_{\Gamma}\subset\Gamma$. The function $l(\tilde{u}_{*})$, $*=L,R$, in (3.5) is defined differently for different examples below to enforce the boundary conditions $(i)$ or $(ii)$ in (1.1); see (4.6) and (4.12) below.

The solutions $\tilde{u}_{L}$ and $\tilde{u}_{R}$ defined in (3.2) are trained simultaneously at each iteration step based on the corresponding loss functions $\mathcal{L}_{L}$ and $\mathcal{L}_{R}$ defined in (3.5). Note that we define two networks separately for $\tilde{u}_{L}$ and $\tilde{u}_{R}$, but they interact with each other via the loss function (3.5) at each iteration and thus the trainings of $\tilde{u}_{L}$ and $\tilde{u}_{R}$ occur simultaneously and they depend on each other. The training algorithm is listed below in Algorithm 1.

We consider the Burgers’ equation (1.1) supplemented with with the periodic boundary condition $(i)$ where the initial data is given by

For this example, the explicit expression of solution is given by

where

We have observed that even though the analytic solution above has been well-studied (refer to, for example, [1]), any numerical approximation of (4.4) requires a specific numerical method to compute the integration, and the integrand functions become increasingly singular when the viscosity $\varepsilon$ is small. To avoid this technical difficulties for our simulations, we use a high-resolution finite difference method to produce our reference solution.

Now, to build our sl-PINN predicted solution, we first notice that the interior layer for this example is stationary, i.e., $\alpha(t)\equiv 0$, and hence we find that the training corrector (3.3) is reduced to

For both classical PINNs and sl-PINNs, we set the neural networks with size $4\times 20$ (3 hidden layers with 20 neurons each) and randomly choose $N=5,000$ training points inside the space-time domain $\Omega_{x}\times\Omega_{t}$, $N_{b}=80$ on the boundary $\partial\Omega_{x}\times\Omega_{t}$, $N_{0}=80$ on the initial boundary $\Omega_{x}\times\{t=0\}$, and $N_{i}=80$ on the interface $\{x=0\}\times\Omega_{t}$. The training data distribution is illustrated in Figure 4.1. For the optimization, we use the Adam optimizer with learning rate $10^{-3}$ for $20,000$ iterations, and then the L-BFGS optimizer with learning rate $1$ for $10,000$ iterations for further training. The main reason for using the combination of Adam and L-BFGS is to employ a fast converging (2nd order) optimizer (L-BFGS) after obtaining good initial parameters with the 1st order optimizer (Adam).

Experiments are conducted for both PINNs and sl-PINNs where $l(\hat{u})$ in (3.1) and $l(\tilde{u}_{*})$, $*=L,R$, in (3.5) are defined by

and

We repeatedly train the PINNs and sl-PINNs for the Burgers’ equations using the viscosity parameter values of $\varepsilon=10^{-1}/\pi,\ 10^{-2}/\pi$, and $10^{-3}/\pi$. We use the same computational settings mentioned earlier. Below, we present the training processes for the cases of $\varepsilon=10^{-2}/\pi$ and $\varepsilon=10^{-3}/\pi$ in Figures 4.2 and 4.3.

To evaluate the performance of sl-PINNs, we present the $L^{2}$-error (4.2) and the $L^{\infty}$-error (4.3) at a specific time in Tables 1 and 2. We observe that sl-PINN provides accurate predictions for all cases when $\varepsilon=10^{-1}/\pi$, $10^{-2}/\pi$, and $10^{-3}/\pi$ in $L^{2}$-norm, while PINN struggles to perform well when $\varepsilon=10^{-3}/\pi$. Additionally, the $L^{\infty}$-error at a specific time shows that sl-PINN has better accuracy than PINN when the shock occurs as time progresses.

The figures below show the profiles of the PINN and sl-PINN predictions. In particular, Figure 4.4 illustrates the case when $\varepsilon=10^{-2}/\pi$, and Figure 4.5 illustrates the case when $\varepsilon=10^{-3}/\pi$. It is observed that for $\varepsilon=10^{-2}/\pi$, the sl-PINN method is successful in resolving significant errors near the shock that occurred in the PINN prediction. Additionally, when $\varepsilon=10^{-3}/\pi$, the sl-PINN method successfully predicts the solution with a stiffer interior layer which the PINN fails to predict. Furthermore, Figure 4.6 shows the temporal snapshots of the errors of PINN and sl-PINN for $\varepsilon=10^{-2}/\pi$, while Figure 4.7 shows the same for $\varepsilon=10^{-3}/\pi$.

In a study addressing the use of Physics Informed Neural Networks (PINNs) to solve the Burgers equation (referenced as [22]), it was found that increasing the number of layers in the neural network leads to improved predictive accuracy. To further test this, a deeper neural network with a size of $9\times 20$ (8 hidden layers with 20 neurons each) was employed, along with a doubling of the training points to $N=10,000$, $N_{b}=160$, and $N_{0}=160$. As indicated in Table 2, when $\varepsilon=10^{-2}/\pi$, the accuracy of PINN increased. However, there was no improvement in accuracy for the stiffer case when $\varepsilon=10^{-3}/\pi$. Interestingly, despite using a larger neural network, the singular-layer PINNs (sl-PINNs) performe similarly to using smaller neural networks. Consequently, it was concluded that utilizing small neural networks of size $4\times 20$ for sl-PINNs is sufficient to achieve a good approximation, thereby increasing computational efficiency.

Additionally, it’s important to note that a two-scale neural networks learning method for the Burgers equation with small $\varepsilon>0$ was proposed in a recent study (referenced as [25]). This approach improved the accuracy of the predicted solution by incorporating stretch variables as additional inputs to the neural network. However, this method requires appropriate initialization from pre-trained models. Sequential training is necessary to obtain the pre-trained models for $\varepsilon=10^{-1}/\pi$ and $\varepsilon=10^{-2}/\pi$ in order to achieve better initialization of the neural network for training the most extreme case, $\varepsilon=10^{-3}/\pi$. In contrast, our sl-PINN method only employs two neural networks for the case $\varepsilon=10^{-3}/\pi$ and it still achieves better accuracy.

We consider the Burgers’ equation (1.1) supplemented with the boundary condition $(ii)$ in the form,

and the initial condition,

Using the Cole-Hopf transformation; see, e.g, [5, 14, 20], we find the explicit expression of solution to the viscous Burgers’ equation (1.1) with (4.7) and (4.8) as