Representation Evaluation Block-based Teacher-Student Network for the Industrial Quality-relevant Performance Modeling and Monitoring

In the T-S network, the student network can achieve the similar performance as by mimicking the teacher network [28]. The T-S network first uses supervised learning to train a deep teacher network, then uses unsupervised learning to train a student network with the guide of the teacher network, which is shown in Fig. 1. So that the prediction performance of student network would be similar to that of teacher network. The network structure of teachers and students can be formulated as:

To give the feedback of the features differences between two networks to the teacher network, uncertainty modeling is considered in training the teacher network, which is inspired by UAE. UAE, designed in Ref. [29], is a modification of autoencoder, which takes into account the noise in the training process of autoencoder. UAE models uncertainty in the training process by adding noise in the latent features of the encoder.

As shown in Fig. 2, UAE consists of two parts, i.e. the encoder and decoder. The encoder represents the edcoding process with noise through a parameterized encoding function $f$ from high dimensional signals ${{\mathbf{x}}_{t}}$ to measured data $\mathbf{z}$ , where ${{\mathbf{x}}_{t}}\in{{\mathbf{R}}^{n}}$, $\mathbf{z}\in{{\mathbf{R}}^{m}}$, $f:{{\mathbf{R}}^{n}}\to{{\mathbf{R}}^{m}}$. The encoder can be expressed according:

The objective function of the autoencoder is to minimize the difference between the output of the decoder and the input of the encoder, that is, to maximize the posterior probability of the signal ${{\mathbf{x}}_{t}}$ recovered from the measured data $\mathbf{z}$:

To learn the optimal recovery $\phi\in\Phi$ with fixed uncertainty, the objective function is to maximize the logarithmic posterior probability of recovering the signal ${{\mathbf{x}}_{t}}$ from the measured data $\mathbf{z}$, which is calculated through:

However, estimating mutual information between high dimensional random variables can be challenging, [29] gets lower bound the mutual information by introducing a variational approximation ${{p}_{\theta}}({{\mathbf{x}}_{t}}\mid\mathbf{z})$ to the model posterior $q_{\phi}\mathbf{(x_{t}\mid z)}$, and when the bound is tight, ${{p}_{\theta}}({{\mathbf{x}}_{t}}\mid\mathbf{z})$ is euqal to $q_{\phi}\mathbf{(x_{t}\mid z)}$. Thus, the objective of UAE is given by :

Though the student network intends to mimic the features from the teacher network, the features of the teacher and student network must be different, which will degrade the performance of the student network. Thus the optimal pre-trained model will suffer from unavoidable performance degradation when the student network is substituted for the teacher network. On the contrary, if we can accurately measure the differences as the feedback and consider the feedback in training, the model would be able to show the robustness and ensure the performance when using the mimic feature from the student network to substitute the features of the teacher network.

Therefore, REB is proposed to generate the virtual robust features in the training phase and the mimic student features in the testing phase according to the usage pattern. In the testing phase, the representative features of the student network are used as the imitate feature of the teacher network. In the training phase, robust features that can improve robustness of a system are generated, which represent the features of the teacher network and reflect the feedback of the features differences between two networks. REB is designed as:

To measure the feedback of features differences added in the training phase of the system, uncertainty modeling is applied in REB, in which the feedback of features differences is isotropic Gaussian noise whose volume is the maximum likelihood estimation of the features differences. Therefore, $\mathbf{d}_{f}\sim\mathcal{N}\left(0,{{\sigma}^{2}}\mathbf{I}\right)$ serves as the inconsistency of the features of the two networks, whose volume is reflected from ${{\sigma}^{2}}$ and estimated through maximum likelihood estimation.

Uncertainty modeling on REB with the teacher-student network is shown in Fig. 3, which extracts representative features from inputs and realizes the robust training of the teacher network. The whole model can be described as follows:

When the noise of the teacher-student network is evaluated, the robust teacher features are considered as the mixture of the noise and the features of the teacher network. Then, according to Eq. 6, we maximize the mutual information between the robust teacher features and the supervision information to learn the parameters that are most conducive to recovering input data ${{\mathbf{x}}_{t}}$ from measured data $\mathbf{z}$ in the case of uncertainty. The mutual information between the representative features $\mathbf{z}$ and the supervision information ${{\mathbf{x}}_{t}}$ can be calculated as:

The virtual features $\mathbf{z}$ are used to replace the real features of the student network through REB. In the robust training of the teacher network in the training phase, the virtual robust teacher features are parameterized by the real teacher features $\mathbf{{{z}_{t}}}$ and an isotropic Gaussian noise according to Eq. 10. The features from the student network are used to mimic the features from the teacher network:

In this way, the performance of the student network is given back to the teacher network to train through uncertainty modeling on REB. The objective functions of the whole network are calculated through:

We adopt the parameterized variational approximation of the posterior probability ${{q}_{{{\phi}_{t}}}}({{\mathbf{x}}_{t}}\mid\mathbf{z})$ as ${{p}_{{{\theta}_{d}}}}({{\mathbf{x}}_{t}}\mid\mathbf{z})$, and thus the loss of the teacher network can be calculated as:

As $\mathcal{L}_{t}$ and $\mathcal{L}_{s}$ cannot be optimized analytically, asynchronous iteration is adopted in the training process of optimizing the objective function of the two networks. We asynchronously iterate ${{\phi}_{t}}$ and ${{\phi}_{s}}$ according to Eqs. LABEL:E16 and LABEL:E17 when ${{\sigma}^{2}}$ is evaluated. ${{\sigma}^{2}}$ is the intermediate variable in the iterative training process, which is updated according to Eq. 12 at the end of each iteration. When ${{\mathcal{L}}_{t}}$ and ${{\mathcal{L}}_{s}}$ reach the set value or do not change significantly, it can be considered that the teacher-student network has been trained to extract the maximum manual information, that is, the quality-relevant features most conducive to the recovery of the process variables have been extracted to the maximum extent.

The overall algorithms of TSUAE are described in algorithm 1 and algorithm 2.

The teacher-student network is trained in the training phase through algorithm 1. Specifically, the parameters of the network are initialized and the initial hyperparameter of the standard deviation is set first. In each iteration, we sample the noise from the current noise condition and calculate the robust teacher features according to Eq. 9. Then the teacher network is updated according to the loss function calculated from the robust teacher features according to Eq. LABEL:E16, and the student network is supervised according to the loss function calculated from the representative features of REB according to Eq. LABEL:E17. Then the noise condition is updated according to Eq. LABEL:E13. Thus, the parameters of the teacher-student network can be updated. Finally, the iteration process repeated until the stop condition is satisfied. In the testing phase, TSUAE is used to get the representative student features from the student network when TSUAE has been trained through algorithm 2.

At last, the complexity of the training algorithm is calculated. Assuming that the number of iterations of the teacher-student network in the training process is $N$ and the algorithm complexity of each iteration is $O(T)$, the algorithm complexity is related to the network structure and can be simplified as $O(T)=O(n_{in}n_{z}+n_{z}n_{h}+n_{h}n_{out})$ for the proposed teacher-student structure in the paper, where $n_{in}$ is the number of inputs of the structure; $n_{z}$ is the number of features of the structure; $n_{h}$ is the number of neurons in the hidden layer of the structure; $n_{out}$ is the number of outputs of the structure. Thus, the algorithm complexity of the training phase is $O(NT)$ while the algorithm complexity of the testing phase is $O(T)$. Therefore, our proposed method is effective in computational complexity in the training phase and the testing phase.

Considering the high cost and time delay of measuring the quality-relevant variables, it becomes relatively difficult, even in some cases, impractical to monitoring the fault occurs in quality-relevant subspace. The literature [25] indicated that the fault detection in quality-relevant subspace can be realized through the prediction of quality-relevant variables. Accordingly, TSUAE is used to extract the quality-relevant representative features by mimicking the teacher features with process-relevant variables and quality-relevant variables. Thereby TSUAE can be used to predict the process-relevant variables and the quality-relevant variables:

The threshold is determined by ${{D}_{\mathbf{x}}}$ and ${{D}_{\mathbf{y}}}$ calculated from the training samples, and ${{J}_{\mathbf{x},th}}$ and ${{J}_{\mathbf{y},th}}$ are determined by KDE. For the ${{D}_{\mathbf{x},new}}$ and ${{D}_{\mathbf{y},new}}$ are calculated from the new sample ${\mathbf{x}_{\mathbf{t},new}}$, if ${{D}_{\mathbf{x},new}}$ exceeds ${{J}_{\mathbf{x},th}}$, the fault can be considered to be detected. Furthermore, if ${{D}_{\mathbf{y},new}}$ exceeds ${{J}_{\mathbf{y},th}}$, a quality-relevant fault can be considered to have occurred.

In the offline robust training of the teacher network, process-relevant variables $\mathbf{x}$ and quality-relevant variables $\mathbf{y}$ are used to calculate teacher features $\mathbf{{{z}_{t}}}$ and student features $\mathbf{{{z}_{s}}}$. The outputs $\mathbf{z}$ calculated from the REB are considered as the robust features of the teacher network. The robust features of the teacher network are the mixture of the teacher features and an isotropic Gaussian noise representing the inconsistency of the two networks. In online process monitoring when the quality variables are unavailable, the teacher features $\mathbf{{{z}_{t}}}$ are unknowable. REB is used to get real student features in the testing phase. Thus, the outputs of the testing phase can be used to evaluate the process-relevant variables and quality-relevant variables. After that, the error and corresponding threshold are calculated in the process-relevant and quality-relevant subspace in offline modeling when quality-relevant variables are available.

In the online application, the pre-trained student model is used to calculate student features $\mathbf{{{z}_{s,new}}}$ from online process-relevant variables $\mathbf{{{x}_{new}}}$ first. Next, REB is used to get the corresponding outputs $\mathbf{{{z}_{new}}}$, which are also the student features when the REB in the testing phase. After that, the corresponding outputs are used to calculate the evaluated process-relevant variables $\mathbf{{{\hat{x}}_{new}}}$ and the evaluated quality-relevant variables $\mathbf{{{\hat{y}}_{new}}}$, which are then used to calculate the corresponding threshold ${{D}_{\mathbf{x},new}}$ and ${{D}_{\mathbf{y},new}}$. Finally, the quality-relevant subspace monitoring is realized through the monitoring decision in the process-relevant subspace and generates a subsequent evaluation of the analysis for the impact of fault on quality-relevant variables when the quality-relevant variables are available.

The proposed process monitoring method based on TSUAE mainly contains two parts: offline modeling and online monitoring, which is shown in Fig. 4.

To illustrate the performance of our proposed monitoring method based on TSUAE, a numerical example is designed. The variables in this example are composed of latent variables mapped from latent variables with noise. The quality-relevant variables are related to some of these latent variables and used to validate the ability to extract quality-relevant features. In this example, six input variables are generated from the same standard Gaussian distribution and unit variance. The output variable is set to a nonlinear function of ${{z}_{1}},{{z}_{2}},{{z}_{3}},{{z}_{4}},{{z}_{5}}$, as is shown below:

Fault 1: A step fault increased by 1.5 from the 201st sample of the 10th input variable;

Fault 2: A step fault increased by 0.7 from the 201st sample of the quality-relevant indicator.

To analyze the features of the process-relevant subspace and the quality-relevant subspace, the process-relevant subspace and the quality-relevant subspace are visualized in Fig. 5, respectively. In the process-relevant subspace which contains a lot of process variables, t-distributed stochastic neighbor embedding (t-SNE) [35], a nonlinear dimension reduction algorithm, is used to reduce the dimension of data to the three-dimensional and then visualized the three-dimensional scatterplot in the left of Fig. 5. In the quality-relevant subspace, the relationship between the process variables and the quality indicators are established through PLS according to the normal data. After that, the KDEs of the residual between the quality indicators and the evaluation calculated from the pre-trained PLS model are recorded in the right of Fig. 5.

In the fault 1 and the fault 2, the differences between normal data and fault data cannot be recognized from scatterplot based on t-SNE. However, the process variables of fault 1 have changed according to the process abnormality. This is because t-SNE reduces the change of the process abnormality through dimensional reduction when maintaining the similarity between the data. In the quality subspace, the fault 1 is considered to be a quality-irrelevant fault which reflects the similarity of the KDEs between the normal data and the abnormal data. As the fault 2 is a quality-relevant fault, the KDEs between the normal data and the abnormal data are obviously different. However, the KDEs of the normal data and the abnormal data are overlapped to a certain extent, and thus the detection method based on the residual subspace of PLS contains a large fault missing rate.

The result of fault 1 is shown in Fig. 6, and the result of fault 2 is shown in Fig. 7. The FARs and FDRs of detection results are summarized in Table 1.

The thresholds are determined by KDE with the 99% confidence interval, so the FARs of all methods are small. In this case, we focus on analyzing the FDR of each method. The fault 1 occurs to the process-relevant variables, which has little impact on the prediction of the quality-relevant subspace. Therefore, the fault detection methods based on the process-relevant subspace can detect the fault, while fault detection methods based on quality-relevant subspace cannot detect the fault. It is worth mentioning that the fault detection method based on the statistics of the process variables can detect the abnormality that can not be distinguished by the visualization method, representing the effectiveness of fault detection methods based on the statistics. The fault 2 is a quality-relevant fault. As the process-relevant variables do not contain abnormality, the fault detection methods based on process-relevant subspace cannot detect the fault, but the quality-relevant fault detection methods such as TSSAE and TSUAE can detect the fault. PLS, RR and TSSAE cannot detect faults in the most time, while our proposed TSUAE can effectively detect faults in the most time. The numerical example proved the model of TSUAE describing the relationship between quality information and input variables was more accurate than others.

A lower threshold means higher fault sensitivity. The thresholds of different methods are shown in Table 2, which are determined by KDE with the 99% confidence interval. The subspace of the TSUAE outputs has a lower threshold value, indicating that it better captures the relationship between input and quality-relevant information. Moreover, the method focuses more on the ability to recover the quality-relevant information because it extracts the quality-relevant representative features for process-relevant variables and combination variables of quality-relevant and process-relevant. Although the threshold of the process-relevant subspace is larger than others, it can still accurately detect the faults. Since the process subspace of this simulation is a linear model from the hidden variables, PCA based on the linear model has the lowest threshold of the process-relevant subspace.

To verify the performance of the TSUAE, we record the root-mean-square errors of the teacher network and the student network in the process-relevant subspace and quality-relevant subspace during the training process, respectively, as is shown in Fig. 8. In the process-relevant subspace, because of the large number of variables, the noise feedback in TSUAE improves the robustness of the teacher network, so our proposed TSUAE contains a lower error, which implies that the model of TSUAE has better features extraction capabilities. In the quality-relevant subspace with only one quality-relevant variable, the performance of the teacher network in TSUAE is worse than in TSSAE after introducing perturbation. However, when the features of the student network are used to replace the features of the teacher network, the prediction performance of TSUAE is better than TSSAE. On the whole, replacing the features of the teacher network with that of the student network helps the recovery of the process-relevant variables. At the same time, it will increase the prediction error of the quality-relevant variables. The reason is that the inputs of the student network are process-relevant variables, which are conducive to process recovery, but lack of representability of quality-relevant variables. Thus it can be inferred that the optimal teacher model is conducive to the prediction of quality variables, but the features differences will reduce the predicting performance dramatically. Our proposed TSUAE can reduce performance degradation on the quality-relevant variables.

In order to analyze the role of REB and uncertainty modeling independently, REB is added in TSSAE with a conventional strategy in which the feedback is the features differences scaled by pre-set feedback rate $-k$, where $k$ is non-negative. Thus, the training processes of the REB based on negative feedback and REB based on uncertainty modeling are shown in Fig. 9. $N_{n}$ represents the noise based on the differences from the $Nth$ iteration in uncertainty modeling.

Therefore, the feedback of features differences can be calculated through:

Therefore, the training process of REB based on negative feedback can be expressed as:

To verify the effectiveness of the REB and the proposed uncertainty modeling on REB, the REB with a negative feedback rate is added to the TSSAE training process. In TSSAE, the feedback is zero as the teacher network is pre-trained. Since the fault 1 is a process-relevant fault and the fault 2 is a quality-relevant fault, we mainly discuss the detection results of the corresponding subspace. Table 3 shows the FDRs and thresholds at different feedback rates. Figs. 10, 11 show the FDRs and thresholds of model based on negative feedback and uncertainty under different feedback rates.

As is shown is Table 3, FDRs in the fault 1 are the same for all methods, indicating that the fault 1 can be detected by all the methods. When k=0.05, 0.1, 0.15, the fault detection performance with negative feedback is superior to the one without negative feedback, which is reflected in the higher quality-relevant fault detection rate and lower thresholds of process-relevant subspace and quality-relevant subspace. When k=0.2, the fault detection performance with negative feedback is not as good as the method without negative feedback because the high-frequency oscillation occurs when the negative feedback rate is too high, which will lead to the instability of the system.

The detection performance is better when k=0.1. This indicates that a small negative feedback rate is beneficial to the fault detection performance, but an excessively large negative feedback rate will degrade the fault detection performance. Therefore, choosing an appropriate negative feedback rate can improve the fault detection performance, indicating the effectiveness of REB.

To further analyze the performance differences in detection between REB based on negative feedback and the REB based on uncertainty modeling proposed in the paper, fault detection rates of fault 1 and fault 2, thresholds of process-relevant subspace and quality-relevant subspace are shown in detail in Fig. 10.

In Fig. 11, although a small negative feedback rate can improve the detection performance, REB based on uncertainty modeling contains higher quality-relevant fault detection rate and lower thresholds of process-relevant subspace and quality-relevant subspace, showing the effectiveness of uncertainty modeling.

The urban wastewater treatment process is a long time-delay process with high coupling and complicated nonlinearity when the disturbance occurs in the wastewater treatment process [36, 37, 38]. To ensure the safe operation of the sewage treatment process and make the discharge products reach the wastewater discharge standard, it is essential to develop a monitoring method for the wastewater treatment process.

BSM1 is a widely used wastewater treatment process model proposed by European Co-operation in the field of Scientific and Technical Research (COST) for the institutions of water treatment research around the world. As is shown in Fig. 12 [39], the BSM1 is composed of a bioreactor and a clarifier. The bioreactor includes two anoxic sections and three aerated sections. It combines nitrification with pre-denitrification and is commonly used to achieve bioreactor denitrification in full-scale plants. The water in the biochemical reaction pool flows out of the fifth unit, some flows into the secondary sink from the sixth layer, and the other flows back through the pipe to the inlet of the biochemical treatment unit for internal circulation. The secondary sedimentation tank contains non-reactive units which are artificially divided into ten layers. Part of the units that conform to sewage discharge standards are discharged from the top to the river, while the sludge clarifier at the bottom is divided into two parts, one part is sent back to the first anoxic tank, and the other part is pumped out of the remaining sludge.

COST provides three different weather conditions of the inflow water described in the files (http://www.benchmarkwwtp.org/) for simulating BSM1, namely dry, rain, and stormy weather conditions. The weather file contains 14 days of sampled data at intervals of 15 minutes. In this experiment, the BSM1 model is used to simulate the dry weather input data by 1,344 times. The first 672 samples are selected as the training dataset, and the last 672 samples are selected as the validation dataset. The descriptions of the 15 variables considered by our simulation are shown in Table 4 [40]:

BOD5 is set as the quality-relevant variable, and the remaining 14 variables are set as input variables. The fault data are set from the 673rd sample with the description in table 5.

The first fault is a step-change in the maximum specific growth rate of autotrophic bacteria in the first aerobic pool, which decreased from 0.5 to 0.3. Due to the weakened life activities of the autotrophic bacteria, the complex biochemical reactions of the wastewater treatment process are affected. The second fault is halving of the oxygen transfer coefficient in the fourth bioreactor. The change in the concentration of dissolved oxygen will change the removal performance of organic matter by the decomposition of heterotrophic bacteria and the nitrogen dislodgement of nitrification of autotrophic bacteria and denitrification of heterotrophic bacteria, and thus affect the biochemical reactions of the wastewater treatment process. The third fault case is when the $S_{NO_{2}}$ sensor produces a displacement range of 0.5. As the measurement of the concentration of nitrate and nitrite nitrogen in the second reactor is the input of the controller, the disagreement of the corresponding real value will affect the wastewater treatment process.

Similar to the numerical example, the characteristics of the process-relevant subspace and the quality-relevant subspace are visualized in Fig. 13. In the process-relevant subspace, a three-dimensional scatterplot based on the t-SNE of the process variables is recorded. In the quality-relevant subspace, the KDEs of the errors between the label of the quality indicators and the estimation based on PLS are recorded. Different from the numerical example, the process-relevant subspace in BSM1 shows the correlation between samples, that is the distance between the features of adjacent samples pre-processed by t-SNE is small. This is not surprising as the continuous process show similarities in adjacent samples, which is reflected in small European distance of adjacent samples. In addition, the differences between the normal data and the abnormal data exists, in which the differences in the fault 2 is obvious. In the quality-relevant subspace, the KDEs between the normal data and the fault data of the fault 2 are obviously different, indicating that the fault 2 is an obvious quality-relevant fault. The KDEs between the normal data and the fault data of the the fault 3 are basically the same, indicating that the fault 3 is a quality-irrelevant fault. The KDEs between the normal data and the fault data of the fault 1 contain insignificant differences, which indicates that the quality indicators are influenced by fault 1 slightly. However, due to the accuracy of the modeling of PLS, the change of the quality indicators cannot be detected.

To detect these faults and analyze the influence on the quality-relevant variables, we monitor the fault in the process-relevant subspace and analyze the fault in the quality-relevant subspace, the result of which is shown in Figs. 14, 15 and 16 through logarithmic monitoring graphs.

FARs of all methods are relatively low because the thresholds are determined by KDE, which can guarantee the reliability of FDRs. Thus, we concentrate on analyzing FDRs. In the process-relevant subspace, all the fault detection methods can detect fault 1 and fault 2, which indicating these faults show unignorable changes in process-relevant subspace. In the fault 3, the nonlinearity of the process dominates the prediction error in the statistics, and thus PCA fails to detect the fault as it substitutes the nonlinear process as the linear process. However, the detection performance of neural network-based methods SAE, TSSAE, and TSUAE is better. As a whole, our proposed method has better performance than other methods.

In the quality-relevant subspace, we analyze each fault to show the performance of the methods. SAE and PCA have no mechanisms for the detection of quality-relevant faults, so they will not be included in this part. For the fault 1, PLS, RR, and TSSAE cannot detect the fault, but our proposed TSUAE can detect the fault. However, it should be mentioned that the quality-relevant statistics of TSSAE have been changed. According to the mechanisms of the process, it can be understood that the changes in the maximum growth rate of autotrophic bacteria in the first anoxic section will affect the quality-relevant variables. Since the changes in quality-relevant variables are not large, it is considered an incipient quality-relevant fault.

For the fault 2, all quality-relevant fault detection methods (TSSAE, TSUAE, PLS, RR) can detect the fault. The changes of quality-relevant variables indicated the concentration of dissolved oxygen will affect the quality-relevant variables, which is considered as an obvious quality-relevant fault. Especially, our proposed method has the highest detection performance compared to other methods, which suggests that our method is more sensitive to quality-relevant failures.

Interestingly, for the fault 3, none of the methods can detect the fault. If we further analyze the fault, it can be considered that the fault 3 is caused by the sensor failure instead of the process failure. Therefore, the fault cannot be detected during the process-relevant and quality-relevant subspaces. Thus, the fault is considered as a quality-irrelevant fault.

In summary, compared to other methods, our proposed method can more efficiently detect not only the largely quality-relevant faults but also the incipient quality-relevant fault in most situations. It can be concluded that TSUAE achieves better detection results, compered to TSSAE, PLS and RR.

Finally, as the lower threshold value of detection methods under the same statistics represents more meaningful representative features of data, the thresholds of different methods are recorded, as shown in Table 7. It is obvious TSUAE has a lower threshold value in the process-relevant subspace and the quality-relevant subspace compared to other methods.