Machine Learning Approaches for Diagnostics and Prognostics of Industrial Systems Using Open Source Data from PHM Data Challenges: A Review

In a collaborative effort with the Swiss Centre for Electronics and Microtechnology (CSEM), 2021 PHM EU offered a dataset derived from a real-world industrial manufacturing line dedicated to testing electrical fuses. The objective of this competition is to perform fault identification and classification, root cause analysis, and system operation parameter identification. The dataset showed eight unique system failure modes under two distinct operating conditions. The primary challenge with the training data was its class imbalance, as the majority of samples represented healthy conditions. Additionally, the dataset was quite small, with only 70 training samples and 29 testing samples. In the fuse test bench dataset, about 10% of the data was missing, and it was not evenly distributed across variables.

Against this background, the winning solution was a combination of decision tree algorithms and a propagation system [23]. While the decision trees focused on diagnosis issue, the propagation system tackled chronology by incoperating a Kalman-style filter. To address data imbalance, the SMOTE (Synthetic Minority Oversampling Technique) method was utilized. Additionally, [45] incorporated the Leave One Feature Out Importance (LOFO-Importance) package for capturing essential features. Subsequently, they applied linear discriminant analysis (LDA) for dimensionality reduction. During the modeling phase, different ML methods were employed — gradient boosting algorithms such as Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM), LDA classifier, and Gaussian process classifier. Especially for gradient boosting algorithms, they used Genetic Algorithms to optimize hyperparameters. A key observation was XGBoost’s superior performance over the other algorithms. Differing from their approach, [3] showcased a rule-based diagnostic technique, comparing its performance with that of decision trees and random forest methods. Additionally, [7] proposed a regularized LSTM for sifting through vital features and then leveraged an ensemble of binary LSTM classifiers for fault detection and classification.

Moreover, the XGBoost algorithm found favor not just in the aforementioned studies but also in three other distinct research papers [4, 91, 112] outside the competition. The distinctions among these studies were as follows: For instance, [112] relied on a feature importance ranking (FIR) method, targeting enhanced performance and simplification in complex industrial classification scenarios, whereas [91] focused on the challenge of missing value imputation, harnessing Partial Least Squares (PLS-MV). A significant contribution of Ramezani’s framework was its explainability, achieved by pinpointing key performance indicators for each fault family using the SHAP method [74].

The 2022 European PHME Data Challenge, hosted in collaboration with Bitron Spa, focused on a classification issue within an actual industrial Printed Circuit Board (PCB) production line. The challenge’s objectives contain three important tasks: (1) Task 1: predicting Automatic Optical Inspection (AOI) defect detection based on Solder Paste Inspection (SPI) data; (2) Task 2: predicting human-made visual inspection labels (OperatorLabel); (3) Task 3 predicting the human-assigned repair label (RepairLabel). Task 1 and Task 2 are binary classification problems, whereas Task 3 is a multi-class classification problem. This explains why, in Table 3, the ”Failure Mode” is listed as ”1+1+2”. In Table 3, the ”Operating Condition” is listed as ”1+1+1”, which means each task has its own operating condition. Meanwhile, the SPI dataset(2022 PHM EU) had missing information in specific fields and 95% of the SPI data was classified as healthy, highlighting a significant imbalance issue.

In 2022 PHM EU, Gaffet’s team got 1st place using XGBoost method based on encoding and feature engineering [32]. Notably, they harnessed the SHAP method for model interpretation. Similarly, [108] leveraged two tree-based algorithms (LightGBM and XGBoost). Their approach centered on solving classification problems by extracting significant statistical data during the feature engineering phase. The importance of feature engineering was further emphasized by [110]. They introduced a novel statistical feature extraction method coupled with a PinNumber-based technique. This method aimed to compress pin-level data into component-level information. When predicting automatic inspection defects, they integrated a neural network model, factoring in feeding imbalance control to navigate data imbalance challenges. Additionally, a random forest model was developed for both human inspection and repair predictions. Apart from feature engineering, [95] applied a multi-layer perceptron (MLP) neural network for defect predictions in automated inspections. For human inspection outcomes, a random forest algorithm was their choice, while decision trees were favored for predicting human repair labels.

Outside of competition, Mirzaei’s team delved deep into the challenge of imbalanced data. They proposed a data-level technique that integrated recursive feature elimination (RFE) for feature selection and oversampling methods. This ensured balanced representation for minority classes, enhancing the performance of multiple ML algorithms, from decision trees and random forests to SVMs and 1dCNNs [80]. Additionally, [62] introduced a new quality management paradigm termed Stream-of-Quality (SoQ) tailored for multi-stage manufacturing processes. By leveraging this dataset, they showcased the effectiveness of their methodology, offering promising avenues for refining industrial AI algorithms and methodologies in a systematic manner.

Rock drills play a pivotal role in sectors like mining, tunneling, and construction. Due to the potential economic and human costs from work interruptions caused by rock drill faults, ensuring accurate fault diagnosis is necessary. 2022 PHM NA aimed to address fault classification problem under various product configurations. The main challenges is domain-shift problem, caused by data being collected from different rock drill machines. The dataset encompassed training data from five rock drill machines, validation data from another machine, and test data from two additional machines, highlighting potential domain shift issues between training, validation, and test data. This leads to heterogeneous signal distributions, which can negatively impact classification accuracy. The dataset [48] encompasses ten distinct fault modes and one health mode, with training, validation, and testing data sizes of 34,045, 3,184, and 16,396, respectively.

In the competition,  [86] won the first place. They deployed a hybrid strategy, combining data-driven techniques with various signal-processing methods. Their approach harnessed domain adaptation, metric learning, and pseudo-label-based deep learning to construct an ensemble DL model for comprehensive fault classification. For samples that posed challenges for DL, they deployed signal processing methods like Dynamic Time Warping (DTW), and Cross-correlation, and used SVM for supervised learning. The runner-up solution introduced a data-cropping technique, employing a convolutional neural network (CNN) as a feature extractor to bridge data length discrepancies. [53] innovatively addressed the domain-shift issue through a domain-adaptation-based scheme that harnessed a domain adversarial learning neural network for extracting domain invariant features while using maximum mean discrepancy (MMD) minimization for bridging distribution discrepancy, and a soft voting ensemble to reduce model uncertainty. Moreover, Minami’s team proposed an ensemble approach, appending specialized models onto a baseline model. This ensemble incorporated domain adaptation strategies to accommodate domain fluctuations. The baseline model employed conventional ML algorithms like SVM, Random Forest, and XGBoost for whole multi-class classification, whereas the specialized sub-models, leveraging CNN for feature extraction and classification, concentrated on binary classifications targeting specific classes that poorly performed on the baseline model [79].

Outside the competition, [69] deployed an end-to-end fault classification framework derived from X-Vectors [101], achieving integrated optimization of both feature extraction and classification phases. [109] proposed a novel deep forest algorithm that fused multi-grained scanning for feature extraction and a cascade forest for layered predictions to perform failure model classification.

The Japan Aerospace Exploration Agency (JAXA) initiated a competition centered on the advancement of PHM technology for spacecraft propulsion systems. The primary objective was to accurately diagnose various states ranging from normal conditions to bubble anomalies, solenoid valve faults, and unknown faults. Simulation data was sourced from four distinct spacecraft, resulting in a dataset comprising 177 training samples and 46 test samples [114]. The paucity of data posed a challenge for data-driven approaches.

In the competition, the champions [78], introduced a novel two-step approach. Initially, a similarity-based model was proposed for the categorization of data into four distinct states. Subsequently, for data corresponding to the solenoid valve fault, a model incorporating physic-inspired features was employed to pinpoint the fault location and estimate the valve opening ratio. They also deployed DTW [11] on the training dataset which was instrumental in quantifying the variability across various segments of the sensor data. Standing in the second position, [65] devised a hybrid approach combining the XGBoost-based method and the rule-based method. While the XGBoost-based approach primarily addressed comprehensive fault classification, the rule-based method was employed to formulate the solenoid valve opening ratio equation. This was accomplished through polynomial fitting, rooted in intrinsic physical characteristics, enabling a precise estimation of the solenoid valve opening ratio.

Additionally, [51] utilized the K-NN algorithms to classify faults and pinpoint the location of anomalies. Prior to the classification, a differentiation between normal and anomalous data was executed based on a similarity-based approach. Their estimation metrics for valve opening ratio were hinged upon the similarity of time series waveforms. Moreover, [2] employed an ensemble framework, integrating K-means clustering and decision trees. This model, enriched by domain-specific expertise, exhibited good precision in both anomaly detection and fault diagnosis. Various approaches in 2023 PHM AP underscore the importance of similarity-based methods and the extraction of physical features when the dataset size is small.

The objective of 2023 IEEE is to develop ML-based models that can efficiently detect faults in the planetary gearboxes of industrial machinery using vibration signals. The dataset covers four prevalent sun gear faults: surface wear, chipped teeth, cracks, and missing teeth. Vibration signals have been recorded for a duration of five minutes each, with a sampling rate of 10 kHz, under two distinct operational conditions. Given the substantial dataset, comprising 50,000 samples, the deployment of DL techniques is feasible.

In 2023 IEEE. [66] utilized ensemble-based CNN techniques to address fault detection via time-series vibration data. Their findings underscored robust performance of integrating three convolution kernel-based methods such as ROCKET (RandOm Convolutional KErnel Transform) method [24], one dimensional convolutional neural networks (1dCNN) integrated with ResNet, and a fusion of LSTM and Fully Convolutional Network (FCN) in delivering classification results for multivariate time-series data. Similarly, both [97] and [55], designed residual-based CNN models to address fault classification challenges. On the one hand, [97] incorporated the Short-Time Fourier Transform (STFT) [44], transforming frequency domain signals into time-frequency domain signals using Fourier analysis - an useful tool for interpreting time-evolving signals. On the other hand, [55] leveraged data augmentation and regularization techniques, enabling model construction with fewer parameters without sacrificing performance quality. Additionally, [102] harnessed a tree classifier to select valuable features from raw vibration signals, subsequently developing a sequential neural network model tailored for the concurrent detection of multiple gear faults.

The 2018 PHM NA emphasized the analysis of fault behavior within the ion mill etch tool, tasking participants to develop a model from the sensor-derived time series data capable of accurately detecting, diagnosing, and prognosticating the time-to-failure for three principal failure modes—Flowcool leak (F1), Flowcool Pressure Too High Check Flowcool Pump (F2), and Flowcool Pressure Dropped Below Limit (F3). Additionally, the Ion Mill Etching (IME) dataset faced an imbalance issue, where the faulty data is much less than normal operation data.

For this dataset, [125, 42, 129, 38]have each proposed strategies based on random forest algorithms for early degradation mode detection and diagnosis. Extending the exploration of ML techniques, [99] evaluated an assortment of models, including Generalized Linear Models, MLP, Multivariate Adaptive Regression Splines, Support Vector Regression (SVR), random forest, etc. Among these, the random forest model distinguished itself with superior performance. Meanwhile, [131] utilized a knowledge distillation approach towards fault detection across various modes, aiming to improve detection performance of infrequent but critical faults.

Recent advancements in RUL prediction [52] have predominantly hinged on the application of DL algorithms to analyze complex multivariate time series data. [125, 42, 38] have all utilized Long Short-Term Memory (LSTM) networks and their variations, such as Gated Recurrent Units (GRU) and LSTM-based Metric Regression (LSTM-MR), to capture important features from the raw data. [39] expanded upon this approach by integrating a Temporal Convolutional Network (TCN) with LSTM with attention mechanisms, which facilitated refined feature extraction from sensor data for accurate RUL prediction. Moreover, Liu et al.’s two-stage deep transfer learning framework aimed at achieving accurate RUL prediction. In the first stage, the developed model leveraged TCN for initial temporal feature learning, followed by domain adversarial learning for data alignment based on one fault mode. Then in the second stage, the first-stage model was fine-tuned based on other fault mode data to handle multiple fault modes and enhance the RUL prediction performance [70]. Distinct from these methods, [129] explored transformer networks, focusing solely on data from abnormal operation phases, while [72] provided a comprehensive comparison of state-of-the-art methods, including TCN, LSTM, attention-based mechanisms, CNN, and Transformer. Moreover, [99] introduced a novel approach using DTW for RUL estimation, leveraging a library of truncated degradation curves and health score models to refine the final RUL predictions.

2019 PHM NA focused on the task of fatigue crack length estimation and prediction within aluminum structures at different points. The challenge harnessed wave signal data collected by piezoelectric sensors subjected to both static and dynamic tensile stresses. Given that the test dataset was limited to wave signals from several initial loading cycles, participants were challenged to estimate crack lengths where signal data were present and predict future crack growth for specified cycles lacking signal data. The complexity of the problem is that data-driven methodologies were viable when signals existed, whereas scenarios lacking wave data required exploration into physics-based strategies. Therefore, solutions derived from this challenge have predominantly employed a hybrid approach of data-driven and physics-based techniques.

Regarding the estimation of fatigue crack length with accessible wave signals, a variety of data-driven models and feature engineering strategies emerged, both within and beyond the competition’s scope. [50] proposed a neural network architecture reliant on features manually derived from raw signals, such as the Pearson correlation coefficient, phase shifts, energy, and information entropy, to train models for accurate crack length estimation. Similarly, [54] initiated their approach with signal preprocessing, utilizing techniques like band-pass filtering and phase alignment to mitigate noise and uncertainty before applying physically insightful feature extraction methods. Thereafter, they harnessed a random forest algorithm, optimizing it through feature selection and grid search for hyperparameter fine-tuning, to estimate crack lengths. Additionally, [127] also leveraged a band-pass filter for feature extraction from raw wave signals, subsequently constructing an SVR model with hyperparameters optimized via grid search. Rao and collaborators, meanwhile, designed an ensemble learning regression model to improve estimation performance with four useful extracted features (root mean square value, correlation coefficient, first peak value, and the logarithm of kurtosis) [92].

Predictive modeling for scenarios lacking wave signal data required a pivot towards physics-based techniques, often in conjunction with data-driven insights, to forecast crack progression. [50] devised a Particle Filter (PF) strategy, integrating the Paris Law and outputs from the previously developed neural network as observational inputs to refine and update the crack propagation pathway. [54] suggested an ensemble prognostics framework under consistent loading conditions, constructing a probability density function (PDF) for each instance. This was followed by a computation of weights derived from each PDF to output the final crack length prediction. Furthermore, when different loading conditions prevailed, Walker’s equation was utilized to forecast crack lengths. Innovatively, [127] introduced a trans-fitting approach, aimed at extracting the crack growth trend from training data and extrapolating it to the test data predictions. Additionally, Rao’s group advanced a variation version of Paris’ Law, aiming to elucidate the correlation between crack progression and the number of loading cycles [92].

In the domain of industrial maintenance, filtration systems are crucial for helping process pollutants from industrial equipment, ensuring seamless system operation. A predominant challenge encountered within these systems is filter clogging - a phenomenon where accumulated pollutants impede flow rates, thereby disrupting standard industrial workflows. To address this issue, 2020 PHM EU concentrated on the prediction of filtration systems’ RUL. RUL, in this context, is delineated as the time until the pressure differential across the filter breaches a threshold of 20 psi. The challenge provided the PHME20 dataset, collected from a controlled experimental setup designed to simulate filter clogging at various contamination levels. Twelve distinct conditions were established influenced by two operational parameters: solid ratio (%) and particle size (µm). Additionally, the 2020 PHM EU dataset demonstrated domain shift problem, with training data featuring small and large particle sizes, while the test data included medium particle sizes.

The champion solution [71] employed a novel hybrid approach, combining kernel regression with fundamental statistical methodologies. Meanwhile, the runner-up, [10], adopted different cutting-edge feature engineering, and ML methods. Their process began with the extraction of features through a rolling window technique and proceeded with feature selection informed by the linear kernel Support Vector Machine (SVM) coefficients, RFE, correlation matrix, and monotonicity testing. A four-layer sequential neural network was their model of choice, supported by K-fold cross-validation throughout the training and validation stages. Capturing the third position, [47] conducted a comparative study of tree-based algorithms and the Bayesian approach. Specifically, random forest, gradient boosting, and Gaussian process regression were utilized to estimate the RUL of the filtration system, supplemented by a novel fault-based RUL assignment that integrated ”Piecewise RUL Assignment” and ”Linear RUL Assignment”.

Contrasting with the feature engineering and conventional ML strategies, some researchers explored some DL methods. [120] introduced a CNN-based DL methodology for RUL prediction, marked by two architectural innovations: a Parameterized Fully Connected Layer that adjusts network weights in response to operational parameter shifts, and a multi-head predictor tailored to distinct degradation process stages. Additionally, [113] presented a new transfer ensemble learning (TEL) framework, leveraging metric learning alongside domain dissimilarity metric and Kullback–Leibler (KL) divergence, to enhance model generalization from source to target domains. This TEL framework amalgamated with a bidirectional long short-term memory (Bi-LSTM) algorithm, coined as TEL-Bi-LSTM, was offered for RUL estimation under different operating conditions. In another innovative approach, [46] proposed a joint autoencoder-regression network, a deep neural architecture that fused a CNN autoencoder with an LSTM network regressor in an end-to-end training paradigm. Genetic algorithms were instrumental in optimizing hyperparameters for this architecture. Additionally, Lee’s team developed a distinct strategy by first establishing a health assessment criterion [64]. They defined a Health Index (HI) for the filter system and utilized K-means clustering for the categorization of the system’s health stages. Subsequent HI predictions were facilitated by the Bi-LSTM algorithm, thus determining the system’s RUL. Lastly, Falcon et al. introduced an innovative sequence modeling technique termed the Neural Turing Machine (NTM) [30]. Conceptualized as a computational architecture, the NTM leverages available data to interact with an external memory component, an approach that facilitates enhanced accuracy in predictions. This model stands out for its ability to generate more precise outcomes when benchmarked against the prevalent LSTM-based solutions that dominate the field.

2021 PHM NA was primarily focused on the prediction of RUL for turbofan engines [13], specifically under four distinct flight conditions and seven failure modes. Participants were required to create predictive models leveraging the N-CMAPSS dataset, aiming to accurately forecast RUL using complex condition monitoring data. The dataset, a collection consisting of 90 synthetic run-to-failure trajectories for training and an additional 38 truncated datasets for testing, served as a comprehensive foundation for developing robust predictive algorithms to predict RUL accurately.

During the competition, the winning team, led by Lovberg, put forward an innovative approach leveraging a deep convolutional neural network (DCNN) [73]. This network was distinguished by its use of dilated convolutions complemented by gated linear unit activations and integration of residual skip connections. Those techniques were designed to expand the network’s receptive field and enhance flexibility, so as to reduce the complexity of the neural network architecture by using less number of parameters, but still having comparable performance. Additionally, they adopted a strategic sequence sampling method, minimizing less informative samples while retaining enough degradation signals for the network’s input. Moreover, Solis-Martin et al. and DeVol’s team pursued advancements in DCNNs as well. The former developed a two-level DCNN system where the first-level DCNN focused on extracting important features from raw data and the second-level DCNN leverages the output from the first level to accurately estimate the RUL [103]. The latter drew upon established DL architectures, deploying the basic blocks or modules from the VGG, GoogLeNet, and ResNet designs into their DCNN framework. This enabled a comparative analysis of model performances using a variety of well-known architectural features [25, 26].

Outside of the competition, [19] addressed concerns regarding the uncertain and poor interpretability of deep learning models by integrating principal component analysis (PCA) to refine time-domain feature sets and subsequently applying four supervised learning techniques, including artificial neural network, random forest, extreme random forest and XGBoost, to estimate RUL. Their innovative use of a custom loss function in conjunction with traditional ANN models got the best results in both Area Under the Receiver Operating Characteristic (AUROC) and Area Under the Precision-Recall (AUPR) metrics. Moreover, the domain of PHM has witnessed an upsurge in the application of XAI techniques to enhance the interpretability and trustworthiness of ML models. Various methods, including LIME, SHAP, LRP, Image-Specific Class Saliency Maps, and Gradient weighted Class Activation Mapping (Grad-CAM), have been reported in recent literature [18, 20, 104]. These techniques are useful in elucidating the decision-making processes of complex models.

Handling missing data in ML, especially in the context of the PHM is necessary, as incomplete data can significantly impact the model performance and prediction accuracy. The strategies to address missing data in PHM have evolved, encompassing a range of techniques from basic deletion to advanced imputation methods. A straightforward strategy is the deletion of data points with missing values, such as listwise or pairwise deletion. For example, in the 2022 SPI dataset, instances having missing values in crucial identifiers like ”Panel_ID”, ”Figure_ID”, and ”Component_ID” were eliminated [108]. However, this approach can lead to the loss of valuable information to some degree. Regarding imputation methods, several innovative methods were used in 2021 PHM EU. [23] applied the Last Observation Carried Forward (LOCF) method, coupled with backward filling, to address the gaps, while  [91] introduced a novel approach, PLS-MV, a partial least squares-based method for imputing missing values. Interpolation was another technique used to estimate missing values, ensuring that cases with absent data did not skew the results. In addition to various imputation methods highlighted in the competitions, there remains scope for exploration in future research such as mean/median/multiple, K-Nearest Neighbors(KNN), regression imputation, maximum likelihood estimation, and ML-based approaches [43].

Data imbalance arises when the distribution of classes in a dataset is uneven. This issue is evident in PHM due to the rarity of failure events in comparison to data representing normal conditions. Data imbalance can significantly undermine the performance of ML models, particularly in classification tasks. When trained on imbalanced data, models may become biased towards the majority class (normal operation) and may not effectively recognize the minority class (failure). This leads to poor performance in predicting failures, which may bring ineffective maintenance planning and unexpected downtimes.

To counteract data imbalance in PHM, various strategies have been adopted. Resampling techniques are commonly used to adjust the dataset to balance the class distribution, either by oversampling the minority class or downsampling the majority class. Synthetic data generation is another approach, as demonstrated by a team in 2021 PHM EU that used the SMOTE to create synthetic samples of the minority class to balance the dataset [23]. Advanced algorithms also play a crucial role in addressing data imbalance. For instance, Tang’s team in 2022 PHM EU used the FIR to control the imbalance ratio in each mini-batch during neural network training [110]. Liu et al. employed transfer learning and domain adaptation techniques in 2018 PHM NA to facilitate knowledge transfer across different fault modes, addressing the issue of insufficient data in specific faults [70]. Moreover, during the 2018 PHM NA competition, many teams utilized Random Forest, an ensemble learning method known for its proficiency in handling imbalanced data, by constructing a forest of decision trees.

Beyond the techniques showcased in the competitions, new DL approaches are proposed, such as the semi-supervised information maximizing generative adversarial network [124], the integration of deep residual networks with auxiliary classifier generative adversarial networks [14], the combination of DL with SMOTE [22], etc. Furthermore, a standardized experimental framework is proposed by Aguiar’s team in order to evaluate 24 state-of-the-art data stream algorithms across 515 imbalanced data streams  [1]. Going forward, there is a need for the exploration of additional DL-based strategies to enhance the handling of data imbalance in the PHM domain.

Domain shift refers to the changes in the data distribution between the training phase (source domain) and the real-world application phase (target domain) of the ML models. This phenomenon frequently occurs when models, initially trained on data from a specific set of machines or under certain conditions, are subsequently applied to different machines or varied operating conditions. Such a shift can markedly affect the performance and reliability of ML models in PHM, making tackling with domain shift problem essential for maintaining the robustness and effectiveness of PHM systems. Transfer learning has emerged as a primary solution to domain shift challenges, as it can achieve the adaptation of models from one domain to be effective in another through fine-tuning and domain adaptation techniques. For instance, Kim et al. utilized domain adversarial neural networks (DANN) along with the minimization of MMD to tackle domain discrepancies in 2022 PHM NA [53]. Meanwhile, Oh et al. employed the deep CORAL method, calculating coral loss to diminish domain discrepancy effects by aligning the covariance of the six training domains with that of the test domain [86]. Moreover, Tian’s team developed a novel TEL framework, facilitating knowledge transfer from the source to the target domain in 2020 PHM EU [113]. Additionally, robust modeling approaches and the ability to continuously update models with new data are also useful and imperative for mitigating the impact of domain shift.

In PHM data challenge competitions, participants often grapple with the difficult decision of whether to use conventional ML algorithms or advanced DL models. This choice is influenced by various factors: the volume and quality of available data, and a trade-off between computational cost and performance. An analysis of research papers from the past six years reveals insightful trends and preferences in model selection:

(1) The volume and quality of training data play a significant role in determining the choice of modeling approach. Research teams often choose DL for building data-driven models when training data are abundant. Conversely, in scenarios with limited training data, conventional ML methods, augmented by feature engineering, are more commonly employed.

This trend is clear when addressing classification problems within PHM competitions. As depicted in Table 4 and Figure 2, 2022 PHM NA and 2023 IEEE competitions, because of their large datasets, have facilitated the application of DL techniques. In 2023 IEEE, various deep CNN models are proposed, including ensemble-based CNNs [66] and residual-based CNNs [97, 55]. Additionally, in 2022 PHM NA, many teams integrate diverse DL approaches with transfer learning such as metric learning + pseudo label-based DL [86], CNN + DANN [53], X-Vectors [69]. However, the scenario differed for the 2022 PHM EU competition, which provided approximately 2000 samples and encountered data imbalance challenges. Given these constraints, many teams leaned towards tree-based methods known for their robustness and ability to address data imbalance, deploying algorithms like XGBoost [32, 108], LightGBM [108], decision trees [80], and random forest [110, 80]. In competitions with significantly fewer data samples, such as 2021 PHM EU and 2023 PHM AP, where the dataset sizes were around 100-200 samples, the deployment of DL was impractical due to its requirement for a large volume of data. Instead, conventional ML methods proved more effective. In 2021 PHM EU, tree-based methods were predominantly used [23, 45, 4], while in 2023 PHM AP, similarity-based [78, 51], rule-based [65], KNN/K-means [51, 2], and tree-based methods [65, 2] are proposed, emphasizing the adaptability of traditional methods to limited data scenarios.

In the realm of RUL prediction tasks, the inherent long-time series nature of the datasets, even when datasets are smaller (less than 100 samples), allows for different approaches to data utilization and data augmentation. By considering each time point or a sequence of time points (data window) as an independent training sample, the effective size of the dataset can be substantially increased. This is evident from Table 6 and Figure 2, which indicate the application of both DL and conventional ML methods across various competitions such as CNN-based [120, 103, 25], RNN-based [125, 113], transfer learning [70, 113], Autoencoder-based [46], transformer-based [129, 72] methods, and conventional ML with physics-based approaches [54, 127] to tackle RUL prediction challenges.

(2) When selecting DL models, there is often a trade-off between computational cost and performance. In the competitions, participants may choose highly complex DL models that require substantial computational resources to achieve even a small improvement in accuracy. While this approach might secure a higher position on the leaderboard, it may not be the most practical choice for real-world implementation. In practice, the value of such a small accuracy improvement needs to be weighed against the increased computational cost and potential scalability issues.

(3) When datasets become publicly accessible, there is a clear shift towards the adoption of more sophisticated DL approaches in PHM data challenge competitions over time. Taking the 2018 PHM NA competition as a case study, our review of 12 published papers utilizing this dataset reveals that the methodologies initially favored by the competition teams largely comprised tree-based methods, SVMs, and basic LSTM algorithms [42, 99]. However, as time goes by, there is a discernible trend towards the development of more complex DL algorithms. This includes but is not limited to, GRU [125], knowledge distillation [131], a two-stage deep transfer learning framework utilizing TCN and DANN [70], TCN combined with LSTM [39], attention mechanisms [39, 72], and transformers [129, 72]. This trend underscores a growing need for continuous innovation and refinement of DL methods in the PHM domain.

In PHM data challenge competitions, accuracy in prediction and classification is still the primary goal. However, with the advance of ML and DL, emphasizing the importance of model interpretability is becoming as crucial as their accuracy in prediction and classification, because it helps to illustrate the model decision-making process and plays an important role in error analysis and further model refinement in PHM.

While only a few teams in competitions have used model interpretability methods to explain the output of their models, outside of competitions, various techniques have been developed and applied to enhance ML model interpretability [68, 6]. As the complexity of deep neural network models, often referred to as ”black boxes”, some previous research utilized interpretable methods like Layer-wise Relevance Propagation (LRP), Gradient-weighted Class Activation Mapping in CNNs, and attention mechanisms in sequential models to shed light on model decision-making processes [104]. Additionally, model agnostic methods like SHAP [74, 107] and LIME [94]have been instrumental in offering insights into model behavior. For instance, Baptista et al. applied the SHAP model to evaluate the outcomes of three different algorithms (Linear Regression, MLP, and Echo State Network) using the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset (jet engines) [9]. Moreover, Moradi et al. introduced an interpretable artificial neural network designed for the automatic selection and fusion of features to develop optimal health indicators from data gathered through structural health monitoring (SHM) [82]. Furthermore, in an era increasingly focused on ethical and responsible AI, transparent and interpretable models are key to not only enhancing the technical aspects of AI solutions in PHM but also extending to ensuring their successful integration and acceptance in real-world applications [119].

Model robustness is defined as the ability of a model to maintain its performance when facing diverse challenges like noise, outliers, and adversarial examples. Techniques such as data augmentation, where training data can be expanded by creating modified versions of existing data or synthesizing new data, have become commonplace. For instance, in 2023 IEEE competition, Kreuzer’s team employed methods like additive white Gaussian noise, circular shift, and random amplitude scaling to increase the volume of training data [55]. Moreover, regularization techniques like L1 and L2 regularization have been instrumental in preventing overfitting and enhancing stability, making the model less sensitive to small fluctuations in input data. What’s more, ensemble methods have been increasingly recognized for the contribution to model robustness in PHM. A comprehensive analysis of recent studies shows that out of 59 research papers, nine utilized various ensemble techniques - including tree-based methods, ensemble LSTM [7], soft voting ensemble [53], transfer ensemble learning [113], ensemble regression [92], and ensemble CNN-based [66] approaches - across six different PHM data challenge competitions. Additionally, adversarial training, which involves training models on both regular and adversarial data, has been recognized for its potential to fortify models against adversarial attacks [124, 53, 89]. Moreover, it is also important to consider probabilistic machine learning techniques, such as Bayesian networks, Gaussian processes, and probabilistic graphical models, which incorporate probability theory into the modeling process to handle uncertainty issues. They are crucial for dealing with uncertainties and improving the robustness of PHM models [33, 83, 36].

Alongside robustness, model generalization refers to developing models that perform reliably on new, unseen data. One of the techniques for improving generalization is cross-validation, which involves dividing the training data into several subsets and each time using one of the subsets as the validation data and others as the training data for model validation during the training process. In 2021 PHM NA, researchers opted for k-fold repeated random subsampling validation to address its limitation, wherein the size of the validation set diminishes as the number of folds (k) increases [103]. In addition to cross-validation, transfer learning, and domain adaptation methods have been crucial in maintaining model effectiveness and generalization when data from the target domain differs from the source domain. The specific implementations in PHM competitions have been detailed in the earlier subsubsection titled “Domain Shift”. Outside of PHM data challenges, some novel approaches are proposed to deal with model generalization issues. Matthew Russell and Peng Wang adopted a domain adversarial transfer learning method inspired by generative adversarial networks, utilizing a 1D CNN architecture to predict tool wear on unseen domains using NASA’s milling dataset [122]. Ding et al. developed a multi-source domain generalization learning approach (GRU + Transformer) that can effectively learn useful degradation feature representations from various run-to-failure datasets of internal combustion engine journal bearings across different conditions and predict unseen working conditions well [27]. Furthermore, Ding et al. proposed an adversarial out-domain augmentation (AOA) framework for predicting the RUL of bearings under unseen conditions. The effectiveness of this AOA-based RUL prediction was validated using IEEE PHM Challenge 2012 and XJTU-SY run-to-failure datasets, illustrating its robustness in domain generalization for predictive maintenance [28].

Our review of PHM data challenge competitions reveals a reliance on single-modality data in the competitions, such as pressure signals, currents, vibrations, or images, without incorporating multi-modal datasets for fault diagnosis and prognosis. Multi-modal machine learning (MMML) in PHM refers to capturing complementary information from multiple data sources (different types) to achieve a more comprehensive and precise evaluation of PHM tasks [90, 115]. Despite its potential, it is still underexplored in the PHM domain.

Recently, Jiang et al. leveraged two modality data (vibration and current signals) to develop deep belief networks (DBNs) for diagnosing wind turbine gearbox faults [49]. Su et al. proposed an MMML model using parametric specifications, text descriptions, and images of vehicles to predict five vehicle rating scores [106]. Additionally, Fan et al. evaluate several MMML strategies to create a comprehensive PHM system for coolant pumps in commercial heavy-duty vehicles, utilizing data from onboard signals, multi-dimensional histograms, and categorical variables [31]. Wang et al. proposed a novel method for feature fusion in multimodal data (vibration and torque signals), applying it to diagnose the bearings faults [121]. For future PHM data challenges, the provision of open-source multi-modal datasets would empower researchers to investigate and apply more advanced MMML techniques, potentially leading to advancements in the context of PHM.

Our analysis of ML and DL methods across nine open-source industrial datasets has revealed the advantages of ML methods applied in PHM. Nevertheless, the nature of the competition tends to prioritize solutions that chiefly enhance accuracy, potentially at the expense of a systematic approach, reusability, and methodological inheritance. It is therefore vital to pursue systematic methodologies for constructing effective PHM systems that go beyond the competitive framework. This should involve thorough research and analysis of open-source datasets to advance ML and DL strategies, aiming not just for competition success but also for benchmarking and comparative analysis.

Souza et al. devised an ML-based, data-oriented pipeline for constructing a Prognosis and Health Management System (PHMS) focused on RUL prediction, utilizing semi-supervised ML with Autoencoder, XGBoost, and SHAP method [105]. As shown in Figure 5, Hu et al. offered a new perspective on reviewing PHM efforts by proposing a division of the PHM lifecycle into DEsign, DEvelopment, and DEcision ($DE^{3}$) phases, and showcasing the important activities and challenges within these stages [40]. Additionally, Lee et al. introduced a novel SoQ methodology for multi-stage manufacturing processes. It can help to analyze multi-parameter influences on product quality and model inter-process relationships in multi-stage manufacturing systems [62]. Moving forward, developing novel, systematic approaches for PHM systems should be encouraged in future PHM data challenge competitions. These efforts can augment the systematization and applicability of ML and DL approaches, thereby expanding their utility beyond the confines of the competition-centric paradigm. Such advancements promise to narrow the divide between academic research and industrial practice, facilitating the broader adoption of data-driven ML across various industrial contexts.