Robust training approach of neural networks for fluid flow state estimations

Let us first apply the CNN to a square cylinder wake at $Re_{D}=300$. The present CNN model ${\mathcal{F}}$ estimates an $x-y$ sectional velocity field ${\bm{u}}$ at $z=1.5$ (non-dimensionalized by $D$) from velocity sensors ${\bm{s}}$ collected from the $x-y$ cross-section at $z=2.0$ such that ${\bm{u}}_{z=1.5}={\mathcal{F}}({\bm{s}}_{z=2.0})$. The estimated velocity fields with the $L_{2}$ error norm are presented in figure 3$(a)$. Here, the $L_{2}$ error norm $\epsilon$ is defined as $\epsilon={||{\bm{u}}_{\rm ML}-{\bm{u}}_{\rm DNS}||_{2}}/{||{\bm{u}}^{\prime}_{\rm DNS}||_{2}}$, where ${\bm{u}}_{\rm ML}$ is the estimated velocity field, ${\bm{u}}_{\rm DNS}$ is the reference DNS field, and ${\bm{u}}^{\prime}_{\rm DNS}$ denotes the velocity fluctuations of DNS data. The reconstructed velocity fields are in reasonable agreement with the DNS data. The reasonable reconstruction can also be observed with the streamwise mean velocity profile in figure 3$(b)$.

We then investigate the capability of the present CNN model from a practical viewpoint. Although we used information from all grid points on the cross section as the input, which means we have to arrange sensors without gaps — this is not realistic. Hence, the robustness of the CNN model against the lacked input data is examined here. The sensor placements are randomly determined, and this information is fed into the CNN model already trained with the full data. However, since two-dimensional CNN can only handle sectional data (i.e., not local sensor measurements), an appropriate preprocessing is required to feed the sensor information into the CNN directly. To do this, we consider four methods to treat the lack of input: 1. zero-fill (i.e., substituting zero to the lacked part), 2. linear interpolation, 3. cubic interpolation, and 4. Voronoi tessellation. The Voronoi tessellation (Voronoi,, 1908) can handle random sensor placements, and its usefulness for fluid flow data has been demonstrated by Fukami et al., 2021c . Note that for the linear and cubic interpolation, the edge region not surrounded by the sensors is filled by linear extrapolation.

The error for each ratio of lacked data $r_{\rm lack}=n_{\rm lack}/n_{\rm all}$ is presented in figure 4$(a)$. The number of all sensors in this problem corresponds to the number of grid points in DNS, i.e., $n_{\rm all}=256\times 128=32768$, while $(n_{\rm all}-n_{\rm lack})$ is the number of randomly placed sensors used for the input. As shown, the error of the interpolation methods is quite smaller than that of zero-fill, which implies the effectiveness of interpolation to keep the robustness. The estimated velocity fields from the lacked input are also visualized in figure 4$(b)$. We can observe the superiority of the cubic interpolation method for all of the lack ratios. The estimated fields using cubic interpolation with the lack ratio $r_{\rm lack}$ of $\{0.88,0.97\}$ are in qualitative agreement with the reference DNS data, though the wake structures slightly deform or disappear at $r_{\rm lack}=0.99$. Therefore, the present CNN model for a square cylinder wake can keep the robustness for as much as $r_{\rm lack}=0.97$ when the cubic interpolation is utilized.

Let us then use a turbulent channel flow as a complex flow example. The CNN models ${\mathcal{F}}$ estimate the velocity field $\{u,v,w\}$ at $y^{+}=15.4$ from sensor measurements on the wall such that ${\bm{u}}_{y^{+}=15.4}={\mathcal{F}}({\bm{s}}_{\rm wall})$. From the perspective of saving sensors, it is important to know physical quantities that can contribute to estimations. Hence, we here examine the dependence of the model performance on the input attributes. As the candidates for physical quantities, we consider several combinations of three physical quantities, streamwise and spanwise wall shear stress $\tau_{x},\tau_{z}$ and pressure $p$, which are often used for state estimation in turbulent channel flow (Suzuki and Hasegawa,, 2006; Guastoni et al.,, 2020).

The estimation performance of each input case is summarized in figure 5. Note that the vorticity field $\omega_{y}$ in figure 5$(a)$ is obtained from the estimated velocity fields $u$ and $w$ such that $\omega_{y,{\rm ML}}=f(u_{\rm ML},w_{\rm ML})={\partial u_{\rm ML}}/{\partial z}-{\partial w_{\rm ML}}/{\partial x}.$ Hence, the assessment on $\omega_{y,{\rm ML}}$ can be regarded as tougher than the use of velocities only because the first-order differential needs to be calculated from the machine-learned velocities. The reconstructed velocity and vorticity fields are in reasonable agreement with the reference DNS. The $L_{2}$ error is also summarized in figure 5$(b)$. Among the cases with a single quantity input i.e., $\{\tau_{x}\}$, $\{\tau_{z}\}$, and $\{p\}$, the $L_{2}$ error with the streamwise wall shear stress $\tau_{x}$ is the smallest, which suggests that ${\tau_{x}}$ most significantly contributes to the estimation. This can particularly be found from the vorticity contours $\omega_{y}$ in figure 5$(a)$. The model with $\{\tau_{x}\}$ input can estimate the DNS-like structures well, while the models with the input of $\{\tau_{z}\}$ and $\{p\}$ cannot estimate vortex structures. However, the $L_{2}$ error of $\{\tau_{x}\}$ input only for $v$ and $w$ is larger than that of $\tau_{z}$. This implies that the reasonable estimation for all velocity components requires both $\tau_{x}$ and $\tau_{z}$. The necessity of the spanwise wall shear stress input $\{\tau_{z}\}$ can also be observed with the probability density function in figure 5$(c)$. For $v^{\prime}$ and $w^{\prime}$, the curve of the CNN with $\{\tau_{x}\}$ input is a little shifted from that of the DNS, while that of the CNN with $\{\tau_{x},\tau_{z}\}$ input being in reasonable agreement with the DNS. Although we investigate the influence of the pressure input, it contributes to the estimation a bit, and the performance of the model with $\{\tau_{x},\tau_{z},p\}$ input is not so different from that with $\{\tau_{x},\tau_{z}\}$ input. We also assess the reconstruction in temporal behavior, considering temporal two-point correlation coefficient $R^{+}_{vv}(t^{+})/R^{+}_{vv}(t^{+}=0)$, as presented in figure 5$(d)$. The coefficient is defined as

The curves for the models $\{\tau_{x},\tau_{z}\}$ and $\{\tau_{x},\tau_{z},p\}$ are in good agreement with the DNS. With the other assessments above, the input attribute of $\{\tau_{x},\tau_{z}\}$ is sufficient for estimation from the viewpoint of temporal behavior.

Let us further assess the capability of the model on the wavespace by focusing on the case with $\{\tau_{x},\tau_{z}\}$ input. The normalized $L_{2}$ error map of energy spectrum in the streamwise and spanwise directions is presented in figure 5$(e)$. Here, the two-dimensional energy spectrum is defined as

where $\hat{(\cdot)}$ represents two-dimensional Fourier transformation and ${(\cdot)^{*}}$ denotes the complex conjugate. For both the streamwise and spanwise directions, the error is small up to the wavenumber of $10^{0.5}$. This implies that the CNN model can make physically reasonable estimations although much finer-scale structures cannot be estimated. This is likely due to the less correlation in the dissipation range (Scherl et al.,, 2020; Fukami et al., 2021b, ).

We also investigate the robustness against the lacked input data of the turbulent channel flow example, analogous to the square cylinder example. Following the discussion above, we here use the CNN model with $\{\tau_{x},\tau_{z}\}$ input. The error for each ratio of lack is shown in figure 6$(a)$. The smaller error with the interpolation methods than that with zero-fill indicates the effectiveness of interpolation, similarly to the square cylinder example. We then check the velocity contours to compare the interpolation methods and to see what percentage of lack can be tolerated, as summarized in figure 6$(b)$. The superiority of the cubic interpolation can be again observed. For $13\%$ lack, the estimated flow field is in qualitative agreement with the DNS, whereas the fine structure is lost with $50\%$ lack, and the estimated structure becomes substantially different from that of DNS with $75\%$ lack. Therefore, we can conclude that the cubic interpolation is the best for preprocessing and the present CNN model for the turbulent channel flow estimation can accept at most $50\%$ lacked input data.

As a more complex problem, let us apply the present CNN model to the transitional boundary layer flow. The model $\mathcal{F}$ estimates the velocity field $\{u,v,w\}$ at $y=0.96L$ from sensor information on a flat plate such that ${\bm{u}}_{0.96L}=\mathcal{F}({\bm{s}}_{\rm plate})$. Analogous to the channel flow example, we investigate what input attributes contribute to the transitional boundary layer estimation. We consider seven cases that are combinations of the streamwise and spanwise shear stresses $\tau_{x},\tau_{z}$ and pressure $p$. The estimated velocity fields and the $L_{2}$ error norm are summarized in figure 7. As clearly seen, the models which include the input of streamwise shear stress $\tau_{x}$ show the better performance than that without $\tau_{x}$ input. However, we should note that the input of $\tau_{x}$ only is insufficient for the estimation of $v$ and $w$. Hence, the additional information such as $\tau_{z}$ and $p$ are required to enhance the estimation accuracy for $v$ and $w$, as presented in figure 7.

We also investigate the robustness against the lacked input, as shown in figure 8. The effectiveness for the use of interpolation can be again observed. We cannot observe a significant difference between the interpolation methods in figure 8$(a)$, which makes us compare in figure 8$(b)$. The $L_{2}$ error norm in figure 8$(b)$ indicates that cubic interpolation is slightly better than the other methods. Note that the larger error of cubic interpolation in the range of $r_{\rm lack}\gtrsim 0.75$ in figure 8$(a)$ does not make sense because the error is quite large. The lack acceptability of the models is then examined by observing flow fields in figure 8$(b)$. For $25\%$ lack, the overall trends of the estimated field can be kept comparing to the DNS. On the other hand, we gradually start to see different flow structures compared to the DNS with $50\%$ lack. Therefore, the present CNN model for the transitional boundary layer example can accept at most $25\%$ lacked input data. However, we should note that the error of “no lack" is originally large, which makes us suspect what the origin of the error is.

As an additional assessment to clarify the point above, the maps of estimated values versus the reference DNS values, usually called 45-degree map, are shown in figure 9$(a)$. We here consider the CNN model with $\{\tau_{x},\tau_{z}\}$ input. A large error can be found in the downstream region for the all velocity components. The same trend can be seen from the comparison of joint probability density function in figure 9$(b)$. The difference between the upper and downstream region can also be assessed from a physical viewpoint by introducing intermittency factor $\gamma$ defined as the fraction of time where the flow in a given region is turbulent. Continuous laminar and turbulent flows respectively correspond to $\gamma=0$ and 1. There are some criteria to determine whether the specific region is turbulent or not. We here use a modified turbulent energy recognition algorithm (M-TERA) method (Zhang et al.,, 1995). The M-TERA method determines a region of turbulent when it satisfies the following equation:

where $\overline{(\cdot)}$ represents the mean over a short time-interval and $(\cdot)_{\rm rms}$ denotes the long-time standard deviation. The contours of intermittency factor are presented in figure 9$(c)$. As can be seen, the downstream region estimated by the CNN model is almost determined as laminar, despite that the region is almost turbulent in DNS. One of the candidates to improve the estimation performance in the downstream region is training data addition (so-called data augmentation in the field of machine learning (Shorten and Khoshgoftaar,, 2019; Morimoto et al.,, 2022)). Hence, our next interest is what kind of structures should be contained in the additional training data.

To clarify this point, we here consider two types of training data addition, as presented in 10$(a)$;

1. 1.

   Add 700 snapshots from both laminar and turbulent region to the original 700 snapshots on transient region such that 2100 snapshots in total (Case 1).

2. 2.

   Add 700 snapshots from both upper and lower portion on transient region such that 2100 snapshots in total (Case 2).

We use the streamwise and spanwise wall shear stresses $\tau_{x},\tau_{z}$ as the input attribute for the CNN model. The performance of each model is then assessed using the test data from the regular region, as summarized in figures 10$(b)$ and $(c)$. Both cases show improvements in reconstruction, especially in the $u$ component. In this particular example, the use of case 2 can affect the accuracy more than that of case 1, which suggests that the transient structures are more important than the laminar/turbulent structures as the additional data in the transient flow estimation. In this way, we can learn considerable paths to improve the estimation accuracy with preparing proper data augmentation methods. We also note that the reason why we cannot find the significant improvement for the $v$ and $w$ components is merely because the training data sampling is determined based on the structural distribution of streamwise velocity $u$, as shown in figure 10$(a)$. Hence, the use of the $v$ and $w$ components for the training data sampling or incorporating intelligence data preparation (Sapsis,, 2020) could promote the estimation capability of the model more.

Regularization is often used in the fields of machine learning and statistics as a method of preventing overfitting and gaining robustness (Schölkopf et al.,, 2002). This method adds a penalization term to the loss function as

where $\alpha$ is a hyperparameter to determine the magnitude of the regularization, $A$ and $B$ respectively indicate the norm factor for the loss and the regularization terms. In this study, a set of $\{A,B\}=\{2,[1,2]\}$ is considered, where $B=1$ and $2$ correspond to Lasso ($L_{1}$ regularization) (Tibshirani,, 1996) and Ridge ($L_{2}$ regularization) (Hoerl and Kennard,, 1970), respectively.

The influence on the sparsity parameter $\alpha$ is investigated for both the $L_{1}$ and $L_{2}$ regularizations in figure 11. We have performed a five-fold cross validation for each $\alpha$ (Brunton and Kutz,, 2019), as presented as the error bar in figures 11$(a)$ and $(b)$. As a method for interpolation of lacked inputs, we use the cubic interpolation. The mean value of the $L_{2}$ error norm within the present cross-validation models do not show a clear dependence on the value of $\alpha$. In contrast, focusing on the standard deviation, we can see that there is a large variation in the error depending on the hyperparameter $\alpha$. Especially, the standard deviation with $\alpha=1\times 10^{-6}$ is quite larger than the others, which suggests that one of the models with $\alpha=1\times 10^{-6}$ exhibits a great robustness compared to the others. The reason for the large variance in the error value is likely because the optimization of neural network is carried out in the high-dimensional solution space regarding the updating of the immense number of weights. The same trend can also be found with the $L_{2}$ regularization in figure 11$(b)$. In sum, a cross validation is mandatory to reach the robust model.

The velocity fields for the regularization cases are summarized in figure 11$(c)$. Note that we here only visualize the best case of each $r_{\rm lack}$ since there is a high variation among the cross-validated models. The regular CNN model without regularization is also shown for comparison. With the regular model, the correct structures cannot be recovered with $75\%$ lack; however, this issue can be mitigated capitalizing on the regularization, especially with the $L_{2}$ regularization for the present case. The superiority of the regularized models can also be observed from the $L_{2}$ error norm. Summarizing above, the regularization is effective in obtaining robustness, although care should be taken for the choice of the hyperparameter $\alpha$. Constructing several models with the appropriate parameter $\alpha$ results in a large variation in robustness, and users should choose the most robust model among them.

Let us then consider dropout (Srivastava et al.,, 2014), which is often used in machine learning to prevent overfitting. This method can randomly deactivate a certain percentage of nodes during the training of a model, which essentially enables us to construct multiple models within a single model. We examine the robustness for the lacked input by constructing several models whose dropout rates are different, as shown in figure 12$(a)$. The dropout rate here represents the deactivation rate of nodes in training. The lack portion is interpolated using the cubic interpolation analogous to the investigation of regularization. The models with dropout outperform the regular model, especially with the large lack ratio. Thus, a larger dropout ratio is indeed effective in obtaining robustness; however, we should note that too large dropout ratio deteriorates the model performance when the lack ratio is small since the connection inside the model is too sparse. The velocity contours are also checked in figure 12$(b)$. Overall, we can observe the superiority of the dropout models over the regular one. In particular, the flow field estimated by the model with $20\%$ dropout is in reasonable agreement with the DNS even with $75\%$ input. Therefore, we can evaluate that dropout can make the CNN model robust for up to about $75\%$ lack in this example.

At last, we investigate the effect of noise-addition training. Noise perturbation for training data is one of the data augmentation techniques to gain robustness of machine learning models because test data are generally treated as “noise" against training data (Shorten and Khoshgoftaar,, 2019; Morimoto et al.,, 2022). We here consider a Gaussian noise whose magnitude is defined with signal-to-noise ratio (SNR), ${\rm SNR}={\sigma^{2}_{\rm data}}/{\sigma^{2}_{\rm noise}}$, where $\sigma{{}^{2}}_{\rm data}$ and $\sigma{{}^{2}}_{\rm noise}$ are the variances of input data and noise, respectively. The present study covers three different magnitudes of noise ${\rm 1/SNR}=\{0.01,0.05,0.10\}$. The Gaussian noise is perturbed to 5000 snapshots out of 10000 training snapshots. The machine learning model is constructed for each case and the training data sets (i.e., 5000 noisy snapshots plus 5000 clean snapshots) are also prepared for each case.

The error of each model for the lacked input is summarized in figure 13$(a)$. Comparing to the regular model, the noise-trained models are more robust for the lacked input, and the robustness is strengthened by adding a stronger noise though there is an upper limit. The performance of the models is also verified using the streamwise velocity contours in figure 13$(b)$. The flow fields estimated by the models trained with noise-perturbed data are in good agreement with the DNS data even with the lacked input. Noteworthy here is that the $L_{2}$ error against the $50\%$ lack of the model trained with $10\%$ noise is smaller than that against the $0\%$ lack of the model trained without noise. This suggests that the noise perturbation to training data significantly helps the CNN model to obtain the strong robustness.