PIAT: Physics Informed Adversarial Training for Solving Partial Differential Equations

The efficiency of deep learning has recently proven to be significant in various scientific fields 1 , such as computer vision, natural language processing, robotics, and physics simulation. One of the effective applications of deep neural networks is to approximate the solution of a physical system 2 ; 3 ; 4 ; 5 ; 6 .

The idea of using neural networks to solve ordinary and partial differential equations was first studied by researchers in 7 ; 8 ; 9 ; 10 ; 11 , where the approximate solution is parameterized by a fully-connected network that allows for a fully differentiable and closed analytic form. Applications of these techniques in scientific computation and numerical approximation of physical systems has increased due to the widespread interest in neural networks.

In 12 , Raissi introduced a physics-informed neural network (PINN) in which the underlying PDE and boundary conditions are enforced through minimization of the loss functions, and the solution derivatives are obtained using the automatic differentiation. The points on the boundary conditions are also considered as the training set.

Efficiency of different types of PINNs has been shown 14 ; 15 ; 16 to solve different classes of PDEs, such as integro differential equations 17 , fractional equations 17 , surfaces PDEs 18 , and stochastic differential equations 19 . In 20 , a conservative physics-informed neural network (cPINN) has been developed to satisfy various conservation laws while solving the PDEs, where the problem is solved in multiple subdomains and ensure continuity of the flux across subdomains. This method can also be used to solve the weak form (variational) of a PDE. Since the weak form contains natural boundary conditions, the neural network solution should only meet the necessary boundary conditions. In Karumuri2020SimulatorfreeSO , the authors considered the variational form for stochastic PDEs and applied the idea of PINN to obtain the PDE solution. In addition, a variational formulation of PINNs (hp-VPINN) was proposed to deal with PDEs with non-smooth solutions 21 , which is based on the Galerkin method. A similar version was also studied in XPINN 22 ; a domain decomposition approach in the PINN framework that conforms to the conservation laws. The authors in 23 proposed a Bayesian approach to physics-informed neural networks to solve the forward and inverse problems. A parallel framework for the domain decomposition of cPINNs and XPINNs was also developed in 24 . The Python library DeepXDE lu2021deepxde , is an implementation of PINNs, which makes the user’s codes compact, and follow closely the mathematical formulation.

Some other recent works have focused on the development of neural network architecture and training 25 ; 26 ; 27 ; 28 ; 29 that can improve the performance of PINNs in various fields. Furthermore, in 30 ; 31 ; 32 , the error estimates and convergence of PINN methods were discussed.

In this paper, we explore effectiveness of the neural network smoothing methods in the performance of PINNs. Specifically, we first propose weight decay Krogh91 and Gaussian smoothing as possible methods that may help the neural network model smoothness, and hence generalization. Weight decay adds a penalty term to the loss function that promotes smaller weight norms in the model, and reduces potential overfitting of the model. Gaussian smoothing adds a small Gaussian random perturbation to each training sample, and performs similar to an augmentation on the input samples. Next, we propose a more generalized method for solving nonlinear problems based on adversarial training called PIAT. Adversarial training has been proved to be effective against the adversarial examples, where small adversarial perturbations are added to the original input such that model produces incorrect outputs SzegedyZSBEGF13 ; NguyenYC15 ; Madry2018 . By using this method, the fully connected neural network is trained to solve physical problems in various dimensions, which leads to better robustness, and generalization of the neural network on the test data, while PINN might not be effective enough. This model is implemented for three PDE systems, Kuramoto-Sivashinsky equation, Sawada–Kotera equation and high dimensional Allen-Cauhn equation. In all cases, the total number of training data is relatively small and the train and test data are generated using Latin Hypercube Sampling (LHS) strategy. The loss functions are minimized using the Adam optimization algorithm.

In the proposed PIAT method, a single neural network is used for the whole domain, which automatically enforces continuity of the solution and its derivatives over time. By using this method, fewer iterations and collocation points are used to achieve convergence in comparison with the standard PINN. The numerical results show that the PIAT method works for high order and nonlinear PDEs and its performance is significantly better than the PINN or other methods such as the Gaussian smoothing or weight decay.

The remainder of this paper is organized as follows. In section 2, we review the PINN formulation to solve differential equations. Section 3 introduces weight decay and Gaussian smoothing as baselines for the generalization purpose. The proposed adversarial training method (PIAT) is introduced in section 4 and their properties are also presented. In section 5, PIAT is applied to 3 different problems and compared with other methods to demonstrate the efficiency of the proposed technique.

We consider the general form of an $m$-th order initial value problem (IVP), which is as follows:

where $N$ represents a nonlinear differential operator, $\lambda$ is a parameter, $\Omega\subseteq\mathbb{R}^{d}$ and $u$ is the unknown solution with known initial and boundary conditions. Moreover, $u_{t}({\mathbf{x}},t)=\partial u/\partial t$.

By using PINN framework, a fully connected neural network, which is composed of multiple hidden layers, is used to approximate the solution $u(\textbf{x},t)$ of the given nonlinear problem. The inputs of the network are $(\textbf{x},t)$ and the outputs are considered as $\hat{u}(\textbf{x},t)$. Then, each hidden neuron of the neural network in the $l$-th layer can be expressed as

where $x^{(l)}_{i}$’s are the inputs in the $l$-th layer, and $y^{(l)}_{j}$’s are the outputs of each hidden neuron in the $l$-th layer. The output of the last layer, $y^{(L)}_{1}$ is used to approximate the solution. In this formulation $w^{(l)}_{i,j}$ and $b^{(l)}_{j}$ are the trainable weights and biases in the $l$-th layer, and $\sigma(.)$ is the continuous activation function.

The parameters of the neural network are randomly initialized and iteratively updated by minimizing the loss function that consists of the residuals of three error terms including the PDE, initial condition, and boundary conditions:

where $u(\textbf{x},t)$ is the estimated solutions and $N_{f}$, $N_{b}$, and $N_{0}$ denote the number of collocation points, boundary points, and initial points, respectively, which can be chosen arbitrary. By solving the minimization problem, the optimal value of the weights ${\mathbf{w}}^{\star}$ and biases ${\mathbf{b}}^{\star}$ are computed and then the approximate solutions $\hat{u}(\textbf{x},t)=u({\mathbf{x}},t;{\mathbf{w}}^{\star},{\mathbf{b}}^{\star})$ is obtained.

In the PINN formulation, the solution’s derivatives are computed using the automatic differentiation, where combining the derivatives of the constituent operations by the chain rule gives the derivative of the overall composition. The capabilities of automatic differentiation are well-employed in most deep learning frameworks such as TensorFlow and PyTorch, and it allows us to avoid time consuming derivations or numerical discretization while computing derivatives of all orders in space-time.

To improve the generalization of neural network in solving the physics informed equations, we propose to implicitly make the solution robust to small random perturbations. Such robustness is known to cause model generalization, which is also desired in the case of PINNs. To investigate the effectiveness of this idea, we add a random bounded noise to the input as an augmentation in the training, and also apply weight decay in order to reach a stable model that generalizes well on the test data.

Augmentation is a well-known technique in training of the machine learning models, as it prevents the model from overfitting to the training data. For instance, rotation and crop are broadly used in the context of image classification models ShortenK19 ; Luke18 . In this problem, we can not use geometrical transforms due to the data types, but we can add random perturbations to the samples. Therefore, we apply Gaussian smoothing in the training as an augmentation. Perturbing input samples with Gaussian noise can be simply done as:

where Gauussian noise with mean equal to $0$ and variance equal to $\sigma^{2}$ is added to the input $x$.

Furthermore, we use the weight decay Krogh91 as another effective method in preventing overfitting and improving generalization. Weight decay is a regularization term that adds a penalty term to the loss function to keep the model weights small as follows:

where $\lambda$ is the hyper-parameter that controls the impact of the weight decay. It is clear that a large $\lambda$ causes higher regularization effect, which leads to more smooth solutions. Likewise, higher $\sigma$ in the Gaussian noise augmentation have a similar effect.

Using a training set $\chi$ and a neural network $f_{\theta}(.)$, with weights $\theta$, the standard training can be done as:

to minimize the loss $\ell(.)$ of $f_{\theta}(.)$, the estimated PDE solution, on training samples. Note that the loss function $\ell(.)$ is defined based on the Eq. 3:

where $y_{1}=0$, $y_{2}=g({\mathbf{x}},t)$, and $y_{3}=h({\mathbf{x}})$, for the training sample $({\mathbf{x}},t)$, and $\mathcal{C}$, $\mathcal{B}$, and $\mathcal{I}$ denote collocation, boundary, and initial points. While standard training performs well in many tasks, it leads to models that are fragile against adversarial examples. These examples are defined as the addition of an $\ell_{p}$-bounded perturbation to the original input such that model generates outputs with large errors SzegedyZSBEGF13 ; NguyenYC15 . This issue is due to the fact that the model does not learn robust features for solving the problem IlyasSTETM19 ; d2020underspecification . The fact that the training data does not sufficiently cover the input space suggests that the physics-informed neural networks also suffer from the same phenomenon. The reason for using weight-decay and Gaussian smoothing was to mitigate this issue. However, they might not be as effective as they should be.

Another solution for this problem is the adversarial training that is widely explored on the image classification task Madry2018 ; ZhangYJXGJ19 ; ZhangXH0CSK20 ; az2021 . Adversarial training has two main parts. In the first part, a small perturbation $\delta$ is optimized and added to the training sample ${\mathbf{x}}$ such that the network loss is maximized. Next, the neural network weights are optimized based on a single stochastic gradient update of the loss function defined on the perturbed sample $({\mathbf{x}},t)+\delta$. As a result, the neural network can learn more robust features and perform well even against adversarially perturbed samples. Adversarial training is summarized as:

where $\epsilon$ is the $\ell_{\infty}$ bound for the $\delta$. We assume that $\delta$ constitutes two parts $\delta_{x}$ and $\delta_{t}$, which perturb ${\mathbf{x}}$ and $t$, respectively. The minimization part of Eq. 8 is done with the stochastic Gradient Descent (SGD). For the maximization part, different algorithms can be applied but we use Projected Gradient Descent (PGD) Madry2018 . PGD attack is an iterative algorithm that computes gradients of the loss with respect to the input, and uses their sign to generate perturbations that maximize the model loss as the following:

where $\alpha$ is the size that we move along the gradient sign direction in each step.

The pure adversarial training itself does not work well in PIAT due to the fact that perturbing training samples in the physics-informed neural networks changes the correct label, $y$, for the sample much more than perturbing images in the vision tasks. Therefore, in PIAT, we propose to optimize the model on ($({\mathbf{x}},t)+\delta$, $y^{\prime}$), where $y^{\prime}$ is the ground truth label for the ${\mathbf{x}}+\delta$, where

We further assume that the type of a training sample, which could be either collocation, boundary, or initial, is unchanged after perturbing the point. Note that in the pure AT, one uses the original label for the input ${\mathbf{x}}$ while training on the adversarial examples. The objective function for PIAT is summarized as:

where the loss function is set according to Eq. 3. PIAT is represented schematically in Fig. 1.

In this section, the adversarial training of physics informed neural network, weight decay and Gaussian smoothing are applied for solving linear and nonlinear equations. Also, the Kuramoto-Sivashinsky equation, Sawada–Kotara equation, and a high dimensional Allen-Cahn equation are considered to conduct more comprehensive analysis.

The training and test points of the evaluation scheme are chosen randomly using Latin Hypercube sampling strategy on the domain of the problem. In order to show the efficiency and capability of the proposed method, the numerical approximations of the mentioned methods are compared. The convergence of our method is also illustrated numerically.

Let us consider the periodic initial value problem for the fourth-order nonlinear Kuramoto–Sivashinsky (KS) equation as follows:

where $u=u(x,t)$ is real-valued function and $u^{0}:\mathbb{R}\longrightarrow\mathbb{R}$ is sufficiently smooth. $u^{0}$, $f$, and $g$ are $2\pi$-periodic functions and $\nu$ is a positive parameter that is playing the role of viscosity. The solution of Eq. 12 is also $2\pi$-periodic in the space domain, i.e. $u(x+2\pi,t)=u(x,t)$ for all $x\in\mathbb{R}$ and $t\geq 0$. The exact solution of Eq. 12 is given by $u(x,t)=\sin(x+t)$. In all experiments, $T$ is equal to $1$, and $\nu=\frac{1}{2}$. A visual depiction of the problem is shown in right panel of Fig. 2.

By using the proposed methods, the solution is approximated by a deep neural network with 5 hidden layers and 100 neurons per hidden layer and 200 boundary points. The Gaussian smoothing is only applied to the PINN, but the weight decay is applied to both PINN and PIAT. The results are presented in the Table 1. First of all, the results demonstrate the effectiveness of PIAT method over PINN regardless of using the weight-decay method. However, the weight-decay improves the results for both PINN and PIAT that demonstrates the advantages of using regularization methods in both cases. Note that the regularization is originally done to prevent overfitting, but in our case this technique has helped to reduce the training error (see the first row of the Table 1). We hypothesize that the regularization would make the loss function smoother, and increase the gradient-based optimization effectiveness. Gaussian smoothing, on the other hand, tries to achieve the same goal in an indirect and blind way, which does not help with the gradient-based optimization. Adversarial sample augmentation in PIAT, moves the samples purposefully in the steepest direction that causes large changes in the solution. Therefore, similar to the weight decay, this technique enhances the gradient, in an input adaptive manner, which leads to a better optimization.

To check the impacts of the neural network hyper-parameters in the performance of the PIAT method, the training and test errors of the solutions for different numbers of boundary and collocation points have been presented in the Table 2. Moreover, Tables 3 and 4 show the results for various number of layers and neurons, which all support the general superiority of the PIAT in comparison to the PINN. It is notable that PIAT exhibits this improvement on larger network sizes, which is consistent with the fact that adversarial training requires over-parametrization of the base model Madry2018 . The train and validation losses during the training is also available for these methods in Fig. 3, which demonstrates that both of the methods have converged and trained for sufficient number of epochs.

To check the impacts of the KS problem hyper-parameters in the performance of the PIAT method, the methods are compared within a wider ranges of $x$ and $t$ in Table 5. Moreover, as another ablation study in this example, the methods are also compared after replacing the exact solution with $u(x,t)=e^{t}.\cos(x).\sin(1+x)$ and corresponding $f(x,t)$ in Table 6. All these new findings also point out that PIAT has a much lower error than PINN even with changing the ranges of $x$ and $t$ or the exact solution.

The proposed method is applied, and the training and test errors with randomly chosen training and test data have been reported in the Table 7, which shows the efficiency of PIAT for solving this kind of problem. Moreover, the results of standard PINN are shown in the Table 8 and so the efficiency of the PIAT in comparison with the PINN is concluded.

In Tables 9 and 10, the results of PINN and PIAT with and without weight decay have been presented for different numbers of neurons. Similar to the example 1, the weight decay improves the results for both PINN and PIAT. Moreover, the combination of PIAT and weight decay reaches the best results in both Tables.

To perform a visual comparison, a point-wise visualization of absolute errors ($|f_{\theta}(x,t)-y|$) is presented all over the inputs range in Fig. 4 for both PINN and PIAT. The results show that PIAT achieves an approximately uniform error all over the inputs range. Moreover, PIAT achieves a much lower errors than PINN, that shows its effectiveness regardless of the choice for test points.

Finally, as an ablation study, the methods are compared within a wider ranges of $x$ and $t$ in Table 11. These findings also confirms the superiority of PIAT in solving the problem.

Finally, as an ablation study, the methods are compared within a wider ranges of $x$ and $t$ in Table 13. The results are similar to previous examples, and show that PIAT has a lower error.

In this paper, we have introduced the physics informed adversarial training (PIAT) of neural networks for solving nonlinear problems. Compared to previous works like PINN, our method helps reduce the model’s test errors on various PDE examples, ranging from low-dimensional to high-dimensional problems. In our simulations, we studied the effects of the different factors on training and test errors, where the convergence of our proposed method was achieved by choosing the different numbers of layers and neurons or input ranges.

Moreover, we proposed weight decay and Gaussian smoothing which demonstrated the PIAT advantages. The weight decay was applied to both PINN and PIAT, where the obtained results show the efficiency of the PIAT method regardless of the use of the weight decay method compared to the PINN. Also, weight decay improved the results for PINN and PIAT, which demonstrated the benefits of using generalized methods, which PIAT did well. The results of Gaussian smoothing also showed that an adversarial attack works better than a random noise to perturb the samples in PIAT.

Author’s Contributions: S.S. and M.A. were involved in the conceptualization, formal analysis, investigation, methodology, validation, and drafting the manuscript. M.A. also contributed to the software development and visualization. M.H.R. contributed to the conceptualization, investigation, methodology, supervision, and review/editing of the paper.

Competing Interests: The authors have no competing interests to declare.

Funding Sources: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.