Graph neural network-based fault diagnosis: a review

Mathematical notations used in this paper follow [53] and are given in Table I. A simple graph can be represented as

where $V$ and $E$ are the sets of nodes and edges, respectively. Let $v_{i}\in V$ be a node and $e_{ij}=(v_{i},v_{j})\in E$ denote an edge between $v_{i}$ and $v_{j}$. Then, the neighborhood of a node $v$ can be defined as $\mathcal{N}(v)=\{u\in V|(v,u)\in E\}$. Usually, a graph can be described by an adjacency matrix $\mathbf{A}\in\mathcal{R}^{N\times N}$ where $N$ is the number of nodes, that is, $N=|V|$. In particular, $\mathbf{A}_{ij}=1$ if $\{v_{i},v_{j}\}\in E$ and $i\neq j$; $\mathbf{A}_{ij}=1$, otherwise. In undirected graph, $\mathbf{A}_{ij}$ denotes an edge connection between nodes $v_{i}$ and $v_{j}$, while in a directed graph, $\mathbf{A}_{ij}$ represents an edge pointing from $v_{i}$ to $v_{j}$. In practical applications, a graph may have node features (also called attributes) $\mathbf{X}\in\mathcal{R}^{N\times c}$ where $c$ is the dimension of a node feature vector. A degree matrix $\mathbf{D}\in\mathcal{R}^{N\times N}$ is a diagonal matrix, which can be obtained as $\mathbf{D}_{ii}=\sum_{j=1}^{N}\mathbf{A}_{ij}$. A graph can also be represented by the Laplacian matrix $\mathbf{L}$, defined as:

An illustration of the relationship among the above three matrices is shown in Fig. 6.

The input of GNNs receives two contributions[39], i.e., the feature matrix $\mathbf{X}$ and the adjacency matrix $\mathbf{A}$. The output of GNNs can be obtained by the general forward propagation equation (3)

where $gnn$ denotes an operation of GNN and $l$ is the number of layers in the architecture.

1) Graph convolutional networks (GCNs)

A GCN can be regarded as an extension of a CNN architecture, and provides an effective method to extract relational/spatial features from graph structured inputs. The GCN model has undergone many improvements, yielding to [28], which proposed the computation

where $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}_{N}$, $\mathbf{I}_{N}$ is identity matrix of order $N$, and $\tilde{\mathbf{D}}$ a diagonal matrix, whose diagonal elements are $\tilde{\mathbf{D}}_{ii}=\sum_{j=1}^{N}\tilde{\mathbf{A}}_{ij}$. $\sigma$ is an activation function, e.g., the Relu function. $\mathbf{\Theta}\in\mathcal{R}^{c\times c^{\prime}}$ is the parameter matrix of the network to be learned, $c^{\prime}$ represents the dimension of the output, and $\mathbf{Z}\in\mathcal{R}^{N\times c^{\prime}}$ is the output matrix.

The cross-entropy loss function

is generally used to train the parameters. Here, $N_{tr}$ represents the number of the training points, $M$ is the number of categories, $\mathbf{Z}$ is obtained from Eq. (4), and $\mathbf{Y}$ is the corresponding label of the training set. It should be noted that in a classification problem, the dimension of the output equals the number of categories, i.e., $c^{\prime}=M$.

Even though GCNs have shown major improvements w.r.t. preceding non-relational architectures, there still exists some aspects to be improved:

(1) Shallow GCN network cannot spread label information on a large scale, which limits its receptive field [30].

(2) The deep GCN network leads to over-smoothing solutions. A multi-layer GCN will make data become highly similar in the forward transmission process, and cannot be correctly distinguished, which greatly reduces the model performance.

(3) GCN belongs to the transductive learning model. When dealing with new data, new nodes must be introduced to modify the original association graph, and the whole GCN model must be trained again to adapt to the new graph.

(4) For a given node, GCN considers all its neighbors to be equally important, instead of paying selective attention to special neighbors: this limits the performance.

Above deficiencies not only restrict performance, but also constraint applications. To address these limits, several new GNN models have been proposed over time.

2) Graph attention networks

A GAT architecture uses the attention mechanism to assign different weights to a node neighbors. In addition, it is an inductive model, implying that GAT can perform online (fault diagnosis) task after training [54][55].

The GAT architecture is more complex than a GCN one. In detail, the feature matrix $\mathbf{X}=\{\bm{x}_{1},\bm{x}_{2},\dots,\bm{x}_{N}\}\in\mathcal{R}^{N\times c},\bm{x}_{i}\in\mathcal{R}^{c},i=1,2,...,N$. Then, performing a linear transformation on the vector $\bm{x}_{i}$ of the node to obtain its feature vector $\bm{x}^{{}^{\prime}}_{i}\in\mathcal{R}^{c^{{}^{\prime}}}$, $\mathbf{W}$ is a linear mapping matrix:

In GAT, for a given $\bm{x}_{i}$, the attention weight value of its neighbor $\bm{x}_{j}$ is reflected by $\bm{a}_{ij}$ obtained as $\bm{x}^{{}^{\prime}}_{i}$ and $\bm{x}^{{}^{\prime}}_{j}$ of nodes $i$ and $j$ after linear mapping are concatenated together, and then the inner product is calculated with a vector $\bm{a}$ of length $2c^{{}^{\prime}}$. The activation function uses $Leaky\_ReLU$, and a final softmax layer normalizes the weights value.

where $\|$ represents the concatenation operator. Then, without loss of generality, the output of the $(l+1)$th hidden layer become

where $\sigma$ denotes the activation function, and $\bm{h}^{l}$ is the output of the $l$th hidden layer, with $\bm{h}^{0}=\mathbf{X}$. The learnable parameters in GAT include the vector $\bm{a}$ and the parameter matrix $\mathbf{W}$.

3) Graph sample and aggregate

Similar to the GAT model, GraphSage belongs to the inductive learning model. In principle, GraphSage learns an aggregation function, which can aggregate the features of a specific node and its neighbors to obtain its high-order features [56].

Denote the input feature matrix as $\mathbf{X}$. The forward propagation formula of the $l$th hidden layer in a GraphSage model is:

where $Concat$ represents the concatenation operation, $Aggre$ performs an aggregation operation on features, and $\bm{h}^{l}$ the output of the $l$th hidden layer, $\bm{h}^{0}=\mathbf{X}$. Graphsage uses four types of feature aggregation operations, including mean aggregator, GCN aggregator, Long Short-Term Memory (LSTM) aggregator, and a pooling aggregator. GraphSage does not involve the attention mechanism, so it treats all neighbor nodes equally. Due to the inductive learning characteristic, GraphSage can be used for an online fault diagnosis task with a lower computational complexity compared to GCN.

4) Graph auto-encoder (GAE)

A GAE is an unsupervised learning model, similar to the auto-encoder [57] [58] [59]. GAE uses a GCN layer as the encoder, its input includes the association graph and the feature matrix, and the output is the coding value $\hat{\mathbf{A}}$ of the node in the graph.

where $\mathbf{Z}=gcn(\mathbf{X},\mathbf{A})$; the mean-squared error function is used as the loss function,

where $\hat{\mathbf{A}}$ derives from Eq. (11).

5) Spatial temporal graph convolutional network (STGCN)

STGCNs have been widely used to solve traffic forecasting problems [60]. Generally, STGCN is composed both of GCN and CNN architectures, where GCN is responsible for aggregating nodes features in the spatial dimension, while CNN performs temporal convolution in the temporal dimension.

According to the different graph analytic tasks, the networks mentioned above fall into three categories, namely, node-level GNNs, edge-level GNNs, and graph-level GNNs [53]. GCN, GAT, and GraphSage belong to the node-level GNN, which classify the nodes in the association graph. Edge-level GNN, like GAE, can be used for matrix completion, which predicts the correlation edges that do not exist in the input adjacency matrix. In a graph-level GNN, such as STGNN, each graph corresponds to a single feature. The characteristics and taxonomy of the aforementioned GNN models in subsection III.B are summarized in Fig. 7.

Fig. 8 illustrates the efficiency of a graph neural network to exploit information coming from its neighbors. Fig. 8(a) shows the original data of a motor running state after normalization and dimensionality reduction. Principal component analysis (PCA) was used to reduce the original data from 48 dimensions to 2 dimensions for visualization. Nodes with different colors show different categories of the motor running states. There are eight square nodes (the number of training sets) in the figure; the test set is composed of 290 nodes. Fig. 8(b) shows the motor running state data after the graph convolutional layer. Data have been normalized and their dimension reduced too. The meaning of nodes in both subfigures is the same; data processed by GCN show operational clusters. As shown in Fig. 8(a), it is likely that the high model performance cannot be obtained using the insufficient training set with eight nodes. On the other hand, for the data shown in Fig. 8(b), since nodes belonging to the same categories are more concentrated and nodes belonging to different categories are farther apart, the training set after the GCN layer is more representative. It can be expected that GNN-based method can achieve better classification performance. Considering that the data set for fault diagnosis is always composed of a large proportion of unlabeled data (fault-free data) and a small part of labeled data (faulty data), the ability of GNN to obtain information of neighbors becomes essential.

For a given node, a GNN synthetically analyzes the characteristics of a node and its neighbors for fault diagnosis [61].

According to the difficulty of obtaining the association graph, GNNs can be divided into two categories. In the early GNN-based applications, the association graph of obtained data set has a certain physical meaning, which provides great convenience for its construction and verification. For example, in the Karate Club data set mentioned in [28], each individual data represents the club member’s information, and the task of the neural network is to predict which club a member belongs to. In this example, the association graph is constructed according to club members’ interpersonal relationships. The explicit dependency between members simplifies the design of the association graph.

However, in the field of GNN-based fault diagnosis, there is generally no explicit dependency information, so the association graph can only be built from data. How to construct the graph and evaluate the quality of the graph are challenges faced by the GNN-based fault diagnosis method. In this paper, two construction methods for the association graph, as well as their advantages and disadvantages, will be discussed in Section V.

In this subsection, fault diagnosis is transformed into three tasks on graphs, that is node classification, edge classification, and graph classification. In addition, the aforementioned GNN models are integrated into the fault diagnosis algorithms.

1) Exploration of node-level GNNs for FD

The considered node-level GNNs in this paper include GCN, GraphSage, and GAT. These architectures treat each measurement as a node in the association graph. As mentioned in subsection III.A.1, two nodes that have a relationship in the association graph are more likely to be divided into the same category by GNN. The implementation of node-level GNN-based FD algorithm is shown in Table II.

2) Exploration of edge-level GNNs in fault diagnosis

In this paper, edge-level GNN is chosen as a GAE. As its outputs are the reconstruction of the corresponding adjacency matrices of the graph, it cannot be directly used for fault diagnosis. However, it is pointed out in subsection V.A that a ready-made association graph in fault diagnosis is often not accessible, and the accuracy of the graph constructed by various methods cannot be guaranteed. Therefore, the edge-level GNN is used to reconstruct the original association graph, so that nodes with uncertain relationship (the corresponding value in the adjacency matrix is 0) are identified as having association. The above steps construct the association graph that can better reflect dependencies between nodes.

Edge-level GNN can be combined with node-level GNN. First, the original graph is input into the GAE model for reconstruction, then the reconstructed graph is fed into the node-level GNN together with feature matrix $\mathbf{X}$ to realize the final fault diagnosis. The implementation of the edge-level GNN-based FD algorithm is given in Table III (only GAE model is considered in this paper).

3) Exploration of graph-level GNNs in fault diagnosis

STGCN is the commonly used graph-level GNN. Unlike node-level GNN, STGCN does not focus on the dependencies within the data set, but on the complex dependency between components in a single data point [60]. In the task of fault diagnosis, time-series are usually composed of data collected by multiple sensors. Therefore, a sample can be understood as a node, and a series of measurements sampled by multiple sensors together form a graph. The implementation of the graph-level GNN-based FD algorithm is shown in Table IV.

In practice, a possible method to construct the association graph is: first, a large number of time-frequency features of a given data set are extracted. Then, a small number of time-frequency features that can reflect the characteristic of the system are obtained through feature selection. Next the K-nearest neighbor (KNN) method is used to find the $K$ nearest neighbors of the given node according to these features. Finally, the construction of the association graph is realized [62].

Suppose the data set $\mathbf{X}=\{\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{N}\}$ is a feature matrix, the KNN method is used, then the nearest neighbor matrix $\mathbf{P}$ is formed, where $\mathbf{P}_{i}$ stores the $K$ nearest neighbor nodes of the $i$th data in $\mathbf{X}$, $i\in[1,N]$. The composition can be realized by using the nearest neighbor matrix $\mathbf{P}$, and the corresponding adjacency matrix $\mathbf{A}\in\mathcal{R}^{N\times N}$, that is,

The illustration of KNN-based graph construction method is shown in Fig. 10, and there are two factors that affect the quality of KNN-based graph construction method:

1. 1.

   The selection of time and frequency domain features [63]. In fact, the selection of feature types is completely subjective. It is difficult to explain why some feature types are used instead of others, many trial-and-error experiments are necessary.

2. 2.

   The determination of the distance between nodes. A straightforward method is to use Euclidean distance to measure the distance between features. From another point of view, the occurrence of a fault will lead to the deviation of features, but may not affect others. Therefore, various features should be lead to different distance values. However, due to the principle of KNN method, all features are treated equally. In addition, its time complexity is $O(N\times N)$, resulting that the increase of the amount of data $N$ can lead to a dramatic increase of the time consumption for the graph construction.

In general, the KNN-based construction method is favorable due to its simple and clear logic.

In [64], the idea of using prior knowledge to construct the association graph is considered. It uses structural analysis (SA) method to prediagnose the fault, and transforms the results of prediagnosis into a graph to construct the GCN model for the final fault diagnosis. In this method, the graph construction is a bridge that connects the model-based SA and the data-based GCN. The illustration of SA-based graph construction method is shown in Fig. 11.

The introduction of prior knowledge significantly increases the training time, and requires not only the knowledge about the mechanisms of the system, but also data information. On the other hand, it ensures the accuracy of the graph, greatly improves the overall performance of the model, and is one of the best methods to build the association graph when prior knowledge is available. In addition, it should be mentioned that, as a bridge, the association graph can combine the knowledge with the measurements to exploit the strengths of both methods.

The construction process of the graph is to analyze the potential existence of the dependency between nodes. Therefore, the construction of the association graph can also be transformed to the link prediction issue. Inspired by [65], we propose a new method for constructing the association graph. Firstly, an incomplete association graph of a data set is constructed by the KNN method. Then the adjacency matrix of the graph is reconstructed by the GAE model. Finally, the downstream task is completed according to the reconstructed association graph.

The method of using a matrix to complete and adjust the association graph is an extension of the KNN method. Since GAE is a type of unsupervised learning, its purpose is to adjust the original association graph, so the graph reconstructed by GAE is at least no worse than the original one. Subsequent experiments show that the method of reconstructing association graph using GAE model can improve diagnostic performance to a certain extent. The illustration of GAE-based graph adjustment method is shown in Fig. 12.

According to the constructed association graphs, GNNs aggregate the isolated data into a whole. Through various aggregation methods, GNNs enable each node to aggregate the information of its neighbors. Due to this, the performance of the GNN model is better than that of ordinary neural network in many fields.

In [66], it was pointed out that the GNN model has been widely used in graph representation learning, but it remains a issue to measure the quality of the graph. To deal with this, two quality indicators, namely feature smoothness and label smoothness, were proposed. The feature smoothness index $\lambda_{f}$ is:

where $\bm{x}_{v_{i}}$ and $\bm{x}_{v_{j}}$ represent the features of the nodes $v_{i}$ and $v_{j}$, respectively, $d$ is the dimension of $\bm{x}_{v_{i}}$ and $\bm{x}_{v_{j}}$, and $|E|$ represents the amount of the edges $E$ in a association graph. It is assumed that nodes with dissimilar features compared with their neighbors tend to obtain more information from association graph. Therefore, $\lambda_{f}$ is positively correlated with the quality of the association graph.

Another label smoothness index $\lambda_{l}$ is defined as:

where $\mathbb{I}(v_{i},v_{j})=0$ if $Y_{v_{i}}\neq Y_{v_{j}}$, $\mathbb{I}(v_{i},v_{j})=1$ if $Y_{v_{i}}=Y_{v_{j}}$, $Y_{v_{i}}$ and $Y_{v_{j}}$ represent the category of the nodes $v_{i}$ and $v_{j}$, respectively.

The purpose of calculating $\lambda_{l}$ is based on an inductive bias that, in a graph, if most nodes and their neighbors are in the same categories, the graph is beneficial for the training of the model. Therefore, $\lambda_{l}$ is negatively correlated with the quality of the association graph.

In conclusion, $\lambda_{f}$ and $\lambda_{l}$ measure the quality of the association graph from two aspects. They point out that a high-quality graph should have the following characteristics: the feature distributions of two neighbors in the graph are quite different (corresponding to a big $\lambda_{f}$ indicator), but they have the same category (corresponding to a small $\lambda_{l}$ indicator).

Interestingly, [66] implicitly explains the limitation of KNN-based construction method: it tends to connect two nodes with similar feature distribution, which makes nodes obtain less information from their neighbors. From this point of view, both the $\lambda_{f}$ index and the $\lambda_{l}$ index calculated by the graph from KNN-based method are small, which indicate a “correct but useless” association graph.

The detailed architectures of various GNN methods have been discussed in Section III. The corresponding hyperparameters of each models are shown in Table V. The GCN model is composed of a GC layer, three 1dCNN layers, and two fully connected layers; The GAT model is composed of two graph attention layers with a multihead mechanism, where the number of heads is eight; The GraphSage model is composed of two layers, and the GCN aggregator is used; the STGCN model contains two temporal blocks and one spatial block. The spatial block uses traditional GCN for graph convolution, while the temporal blocks perform convolution-max-pooling-convolution for each node in the temporal dimension.

Furthermore, 6 kinds of widely-used baseline models are used, namely the CNN, LSTM, RF, GBDT, LGBM, and SVC models.

1. CNN. CNN is a rather common baseline model, which performs well in dealing with issues in various research fields.

2. LSTM. LSTM does well in processing time-series data. It is kind of an advanced recurrent neural network (RNN).

3. RF. Random forest model (RF) constructs multiple decision trees. To realize the prediction purpose, it counts the prediction results of each decision tree, and obtains the final results through voting methods.

4. GBDT. Gradient boosting decision tree, also known as MART (multiple additive regression tree), is an iterative decision tree method. The algorithm is composed of multiple decision trees, and the conclusions of all trees are accumulated to make the final answer.

5. LGBM. Light gradient boosting machine is a framework for implementing GBDT method. It supports efficient parallel training, and has the advantages of faster training speed, lower memory consumption, better accuracy, distributed support and rapid processing of massive data.

6. SVC. Support vector classification avoids the traditional process from induction to deduction, efficiently realizes the transformation from training process to prediction process, and greatly simplifies the usual problems such as classification and regression.

The experiments in this paper are implemented on three data sets. The first data set is obtained from a hardware-in-the-loop (HIL) simulation platform based on the structure of the pulse rectifier in a traction system (as shown in Fig. 13), named the “rectifier data set”. The second data set is collected from a motor benchmark, named “motor data set”. The last data set is obtained from the Tennessee Eastman chemical process benchmark, named the “TEP data set”¹¹1http://brahms.scs.uiuc.edu.

The dimension of the rectifier data set is 1067 $\times$ 256 $\times$ 6. It is composed of 1067 labeled measurements. Each measurement is composed of six sensor samples, and the dimension of a single measurement is 256 $\times$ 6. The 1067 measurements are divided into one normal condition and five fault types.

The motor data set consists of 1104 labeled measurements, each of which records 48 features of its corresponding motor, and the dimension of the total data set is 1104 $\times$ 48. It contains one normal condition and three fault types. Since the motor data set has no evident sequential characteristic, STGCN is not used for validation in it.

The dimension of TEP data set is 1100 $\times$ 500 $\times$ 52, which is composed of 1100 labeled measurements, each of which is composed of 41 observation variables and 11 control variables. It contains one normal condition and seven fault types. Fault types exist in the three data sets are briefly described in Table VI.

The three data sets are collected from systems either with known structural information (like the pulse rectifier) or without known structural information. Thereby, they are representative in the field of fault diagnosis. Related parameters of the data sets are summarized in Table VII.

As is shown in Table VI, more than ten types of faults are considered in experiments, which can be roughly divided into three categories, including random variation fault, step fault, and slow drifting fault. Their characteristics are depicted as follows [67].

1. Characteristics of a random variation fault. Sampled values with random variation fault will fluctuate randomly. Therefore, there exists irregularly changed deviations between the samples and the actual value. The generation of the random variation faults is affected by many factors including noise and uncertainties, which are ubiquitous in a industrial process.

2. Characteristics of a step fault. The deviation between sampled values and actual values changes significantly in a short period of time. The step fault can lead to great harm to a industrial system, take pulse rectifier for example, step fault may be caused by an open circuit or short circuit.

3. Characteristics of a slow drifting fault. The samples adds an extra signal that is proportional to time. Slow drifting faults may result from aging of components.

The construction of the association graph plays an important role in GNN-based fault diagnosis. Based on SA, KNN, and KNN + GAE methods, three kinds of graphs are constructed.

Among graphs constructed by three kinds of methods, each of them has its own characteristics. For the graph constructed by SA method, all nodes in a graph are clustered respectively. Nodes belonging to the same cluster are connected to each other, and nodes of different clusters are isolated from each other (as is shown in Fig. 14(a)). For the graph constructed by KNN method, all nodes have the potential to be connected to each other. Usually, all nodes in the graph form a connected domain (as is shown in Fig. 14(b)). The graph constructed by KNN + GAE method is slightly different from the KNN-based graph, whose schematic diagram is shown in Fig. 14(c).

Related parameters about each graph are shown in Table VIII. Note that the $K$ value is only applicable to the KNN method.