Learning Algorithms for Coarsening Uncertainty Space and Applications to Multiscale Simulations

The parameter space $\Omega$ is assumed to be of very high dimension (i.e. large $N$) and consists of very large number of realizations. For a given realization, the idea is to find its representation in the coarse space and use the coarse space to perform the computation. We will use the deep cluster learning algorithm to perform the coarsening. Due to the heterogeneous properties of the proposed problem, fine mesh is used; this will bring difficulties in coarsening the parameter space and in computation of the solution. We hence perform the parameter coarsening locally in the space $D$, i.e., we also coarsen the spatial domain. To coarsen the spatial domain, we use coarse grids and consider the GMsFEM.

In Figure 1, we present an illustration of the proposed coarsening technique. On the left figure, the coarse grid blocks in the space are shown. Each coarse grid has a different clusters in the uncertainty space $\Omega$, which corresponds to the coarsening of the uncertainty space. The main objective in multiscale methods is efficiently finding the clustering of the uncertainty space, which is our main goal.

It is computationally expensive to capture heterogeneous properties using very fine mesh. For this reason, we use GMsFEM to coarsen the spatial representation of the solution. The coarsening of the parameter space will be performed in each local spatial neighborhood. We will achieve this goal by the GMsFEM, which will briefly be discussed. Consider the second order elliptic equations $Lu=f$ in $D$ with proper boundary conditions; denote the the elliptic operator as:

Let the spatial domain $D$ be partitioned by a coarse grid $\mathcal{T}^{H}$; this does not resolve the multiscale features. Let us denote $K$ as one cell in $\mathcal{T}^{H}$ and refine $K$ to obtain the fine grid partition $\mathcal{T}^{h}$ (blue box in Figure 2). We assume the fine grid is a conforming refinement of the coarse grid. See Figure 2 for details.

For the $i$-th coarse grid node, let $\omega_{i}$ be the set of all coarse elements having the vertex $i$ (green region in Figure 2). We will solve local problem in each coarse neighborhood to obtain set of multiscale basis functions $\{\phi_{i}^{\omega_{i}}\}$ and seek solution in the form:

where $\phi_{j}^{\omega_{i}}$ is the offline basis function in the $i$-th coarse neighborhood $\omega_{i}$ and $j$ denotes the $j$-th basis function. Before we construct the offline basis, we first need to derive the snapshot basis.

We present the general methodology in this section. The target is to save the time in computing the GMsFEM basis $\phi_{k}^{\omega_{i}}$ for all $\omega_{i}$ and for all uncertain space parameters. We propose the clustering algorithm to coarsen the uncertain space in each local neighborhood. The key to the success of the clustering is that: the cluster should inherit the property of the solution, that is, the local heterogeneous fields $\kappa(x,s)$ clustered into the same group should have similar solution properties. When the cluster is learnt by the some learning algorithm, the only computation involved is to fit the local neighborhood of the given testing heterogeneous field into some cluster. This is a feed forward process including several convolution operations and matrix multiplications and compared to the direct computing, we save a lot of time in computing the spectral problem (6) and the inverse of a matrix (10). The detailed process is illustrated in the following chart (Figure 3):

1. 1.

   (Training) For a given input local neighborhood $\omega_{j}$, we train the cluster (which will be detailed in next section) of the parameter space $\Omega$ and get the clusters $S_{1}^{j},...,S_{n}^{j}$, where $n$ is the number of clusters and is uniform for all $j$. Please note that we may have different cluster assignments in different local neighborhoods.

2. 2.

   (Training) For each local neighborhood $\omega_{j}$ and cluster $S_{i}^{j}$, define the average $\bar{\kappa}_{ij}$ and compute generalized multiscale basis for $\bar{\kappa}_{ij}$.

3. 3.

   (Testing) Given a new $\kappa(x,s)$ and for each local neighborhood $\omega_{j}$, fit $\kappa(x,s)$ into a $S_{i}^{j}$ by the trained network (step 1) and use the pre-computed GMsFEM basis (step 2) to find the solution.

It should be noted that we perform clustering using the heterogeneous fields; however, the cluster should inherit the property of the solution corresponding to the heterogeneous fields. This makes the clustering challenging. The performance of the standard K-Means algorithm relies on the initialization and the distance metric. We may initialize the algorithm based on the clustering of the heterogeneous fields but we need to design a good metric. In the next section, we are going to introduce a learning algorithm which uses an auto-encoder structure and multiple losses to achieve the required clustering task.

The cluster net is aimed for clustering the heterogeneous fields $\kappa(x,s)$; but the resulting clusters should inherit the properties of the solution corresponding to the $\kappa(x,s)$, i.e., the heterogeneous fields grouped in the same cluster should have similar corresponding solution properties. This similarity will be measured by the adversary net which will be introduced in section (3.3). We hence design the network demonstrated in Figure 4.

The input $X\in\mathbb{R}^{m,d}$ ,where $m$ is the number of samples and $d$ is the dimension of one local heterogeneous field, of the network is the local heterogeneous fields which are parametrized by the random variable $s\in\Omega$. The output of the network is the multiscale basis (first GMsFEM basis) which somehow represents the solution corresponding to the coefficient $\kappa(x,s)$. This is a generative network which has an auto encoder structure. The dimension reduction function $F(X)$ can be interpreted as some kind of kernel method which maps the input data to a new space which is easier to be separated. This can also be interpreted as the learning of a good metric function for the later performed K-mean clustering. We will perform K-means clustering algorithm in latent space $F(X)$. $G(\cdot)$ will then transfer the latent space data to the space of multiscale basis function. This process can be taken as a generative process and we reconstruct the basis from the extracted features. The detailed algorithm is as follow (see Figure 5 for an illustration):

Loss function is the key to the deep learning. Our loss function is consisted of cluster loss and the reconstruction loss.

1. 1.

   Clustering loss $C(\theta_{F},\theta_{G})$: this is the mean standard deviation of all clusters of the learnt basis and $\theta$ is the parameters we need to optimize. It should be noted that the loss here is computed using the learnt basis instead of the input of the network. This loss controls the clustering process, i.e., the smaller the loss, the better the clustering in the sense of clustering the multiscale basis. Let us denote $\kappa_{ij}$ as $jth$ realization in $ith$ cluster; $G(F(\kappa_{ij}))\in\mathbb{R}^{d}$ will then be $jth$ learnt basis in cluster $i$ and let $\theta_{G}$ and $\theta_{F}$ be the parameters associated with $G$ and $F$, the loss is then defined as follow,

   $\displaystyle C(\theta_{F},\theta_{G})=\frac{1}{|A|}\sum_{i}^{|A|}\sum_{j}^{A_{i}}\frac{1}{A_{i}}\|G(F(\kappa_{ij}))-\bar{\phi}_{i}\|_{2}^{2},$

   (11)

   where $|A|$ is the number of clusters which is a hyper parameter and $A_{i}$ denotes the number of elements in cluster $i$; $\bar{\phi}_{i}\in\mathbb{R}^{d}$ is the mean of cluster $i$. This loss clearly serves the purpose of clustering the solution instead of the input heterogeneous fields; however, in order to guarantee the learnt basis are closed to the pre-computed multiscale basis, we need to define the reconstruction loss which measures the difference between the learnt basis and the pre-computed basis.

2. 2.

   Reconstruction loss $R(\theta_{F},\theta_{G})$: this is the mean square error of multiscale basis $Y\in\mathbb{R}^{m,d}$, where $m$ is the number of samples. This loss controls the construction process, i.e., if the loss is small, the learnt basis are close to the real multiscale basis. This loss will supervise the learning of the cluster. It is defined as follow:

   $\displaystyle R(\theta_{F},\theta_{G})=\frac{1}{m}\sum_{i}^{m}\|G(F(\kappa_{i}))-\phi_{i}\|_{2}^{2},$

   (12)

   where $G(F(\kappa_{i}))\in\mathbb{R}^{d}$ and $\phi_{i}\in\mathbb{R}^{d}$ are learnt and pre-computed multiscale basis of $ith$ sample $\kappa_{i}$ separately.

The entire loss function is then defined as $L(\theta_{F},\theta_{G})=\lambda_{1}C+\lambda_{2}R$, where $\lambda_{1},\lambda_{2}$ are predefined weights. We are going to solve the following optimization problem:

for the required training process.

We have introduced the reconstruction loss which measures the similarity between the learnt basis and the pre-computed basis in the previous section. It is the Mean square error (MSE) of the learnt and pre-computed basis. MSE is a smooth loss and easy to train but there is a well known fact about MSE that this loss will blur the image. In the area of image super-resolution and other low level computer vision tasks, the loss is not friendly to inputs with high contrast and the resulting generated images are usually over smooth. Our problem has multiscale nature and is similar with the low level computer vision task, i.e., this is a generative task; hence blurring and over smooth should happen if the model is trained by MSE. To define a great reconstruction loss is important.

Motivated by some works about the successful application of deep fully convolutional network (FCN) in computer vision, we design a perceptual loss to measure the error. It is now clear that the lower layers in the FCN usually will extract some general features shared by all objects like the horizontal (vertical) curves, while the higher layers are usually more objects oriented. This gives people the insight to train the network using different layers. Johnson then proposed the perceptual loss [27] which is the combination of the MSE of selected layers of the VGG model [36]. The authors claim in their paper that the early layers tends to produce images that are visually indistinguishable from the input; however if reconstruct from higher layers, image content and overall spatial structure are preserved but color, texture, and exact shape are not.

We will adopt the perceptual loss idea and design an adversary network to compute an additional reconstruction loss. The network structure can be seen in Figure 6.

The adversary net is fully convolutional with input and output both pre-computed multiscale basis. The network has an auto encoder structure and is pre-trained; i.e., we are going to solve the following minimization problem:

where $\phi_{i}$ is the multiscale basis and $f$ is the adversary net associated with trainable parameter $\theta_{A}$. Denote $f_{j}(\cdot)$ as the output of layer $j$ of the adversary network. The additional reconstruction loss is then redefined as:

where $I$ is the index set which contains some layers of the adversary net. The complete optimization problem can be now formulated as:

We consider solving (1)-(2) for a heterogeneous field with moving channels and changing background. Let us denote the heterogeneous field as $\kappa(x)$, where $x\in[0,1]^{2}$, then $\kappa(x)=1000$ if $x$ is in some channels which will be illustrated later and otherwise,

where $\eta$ follows discrete uniform distribution in $[0,1]$. The channels are moving and we include cases of the intersection of two channels and formation and disappearance of the channels in the fields. In Figure 7, we demonstrate $20$ heterogeneous fields.

It can be observed from the images that, vertical channel (at around $x=30$) (not always) intersects with horizontal channels (at around $y=40$); and the channel at $x=75,y=25$ demonstrates the case of generation and degeneration of a channel.

We train the network using 600 samples using the Adam gradient descent. We find that the cluster assignment of $600$ realizations in uncertain space is stable(fixed) when the gradient descent epoch reaches a certain number, so we set the stopping criteria to be: the assignment does not change for $100$ iteration epochs; and the maximum number of iteration epochs is set to be $1000$. We also find that the coefficients in (16) can affect the training result. We set $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$.

It should be noted that we train the network locally in each coarse neighborhood. The fine mesh element has size $1/100$ and $5$ fine elements are merged into one coarse element.

We will present the numerical results of the proposed method in this section. We are going to show the cluster assignment experiment first, followed by two other experiments which demonstrate the error of the method.

The target of this task is not the learning of multiscale basis; the multiscale basis in this work is just a supervision of learning the cluster. However to demonstrate the effectiveness of the adversary network, we also test the the effect of the adversary net. There are many hyper-parameters like the number of clusters and coefficients of the loss function which can affect the result; so to reduce the influence from the clustering, we remove the clustering loss from the training, so this is a generative task which will generate the multiscale basis from the output of the first network in Figure 6. The loss function now can be defined as:

where $R$ and $A$ are defined in (12) and (15) separately; and $\lambda_{1}$ and $\lambda_{1}$ are both set to be $1$. We compute the relative error (17) first by using the learnt multiscale basis which is trained by (18); and second by using the multiscale basis trained without the adversary loss (15), i.e.,

The $l_{2}$ relative error improves from $41.120439$ to $36.760918$ if we add one middle layer from the adversary net.

We also calculate the MSE difference of two learnt basis (by loss (18) and (19) separately) and real multiscale basis, i.e., we calculate $\|B_{\text{learnt basis}}-B_{\text{real basis}}\|_{MSE}$, where $B_{\text{learnt basis}}$ refers to two basis trained with (18) and (19) separately and $B_{\text{real basis}}$ is the real multiscale basis formed using the input heterogeneous field. The MSE amazingly decreases from $0.9073400$ to $0.748312$ if we use basis trained with the adversary loss (18). This can show the benefit from the adversary net.