TFBEST: Dual-Aspect Transformer with Learnable Positional Encoding for Failure Prediction

Condition-based maintenance (CBM) of equipment in large-scale cyber-physical systems (CPS) aims to maximize the reliability of these systems by selectively replacing parts that are predicted to fail[1]. The underlying assumption is that the degradation model these systems follow show up as trends in data that can be effectively used to predict the Remaining Useful Life (RUL) of such systems. Hard disk drives (HDD) are an important storage component of many systems, from personal computers to distributed data-centers. An HDD failure in a data-center can lead to catastrophic data loss if pertinent backup plans are not maintained. It is important to keep in mind that electro-mechanical devices are prone to failure owing to varied operational conditions and aging, therefore HDDs are no exception. However, early sensing of triggers and out-of-distribution signatures in the data can greatly help plan for failure, in the event it occurs. One common measure used for fault quantification in such cases is the Annualized Failure Rate (AFR). AFR represents the likelihood, expressed as a percentage, of a drive experiencing failure within a year. This probability is derived from the performance patterns of comparable drives.

The informed reader might be aware that prognostication techniques, as of this date, are broadly divided into three categories: model-based, data-driven and hybrid. Model-based techniques require precise and thorough dynamic modeling of electro-mechanical machinery or of individual components therein. This becomes an extremely difficult task if there is non-linearity in the system dynamics. Data-driven approaches, on the other hand, exploit the wealth of sensor data features to understand complex interrelationships between these features and use it to estimate the likelihood of failure in future. In the case of HDDs, data-driven approaches look at S.M.A.R.T. sensor features to model the dependencies and look for patterns of interest. Of course, one might look at hybrid approaches as well, where the Physics of Failure is captured by a joint modeling using dynamical system equations and machine learning models [2].

The S.M.A.R.T. monitoring system is essential for protecting HDDs over the course of their useful lives. Temperature, operation hours, on/off cycles, damaged sectors, and read/write errors are just a few of the many variables that it meticulously tracks. Continuous comparisons are made between these metrics and predetermined thresholds established by the HDD manufacturers. The system warns the user proactively when a particular S.M.A.R.T. parameter exceeds its set threshold. A laudable safety measure, this prompt notification enables users to take preventive actions like data backup and prompt drive replacement, preventing potential data loss. But relying on thresholds can lead to very premature drive change. In the literature, machine learning algorithms have been used for RUL prediction, such as Support Vector Machines (SVM) [3], Hidden Markov-models (HMM) [4] and Long short term memory networks (LSTM)[5]. These techniques, however, rely on time-consuming feature engineering. Transformer [6] based approaches, in comparison, can automatically extract useful features from the data and achieve significantly superior prediction performance.

Another notable issue is that more attention should be paid to the essential features providing significant information on degradation. Transformer architectures [6] using the attention mechanism [7] can learn the degradation mechanics without feature selection and can tend to long input sequences. Very recently, the Dual Aspect Self-Attention Transformer (DAST) [8] has been proposed, which adds an additional encoder over the vanilla transformer [6] to extract more information about how different sensors affect the system rather than attending to only the weights of different time steps. However, the DAST network uses an absolute positional encoding which can not effectively model the timesteps. To overcome this, we propose and extensively validate a novel transformer architecture: Temporal-Fusion Bi-Encoder Self-attention Transformer (TFBEST) that uses a learnable positional encoding to encode position of the timesteps in the encoder.

Contributions: The contributions of this work are significant in the following ways:

1. 1.

   A novel transformer architecture for remaining useful life prediction: We propose a new Transformer architecture which employs a learnable positional encoding to encode position of the timesteps in the encoder and two encoders (sensor and time) to extract information in parallel which avoids mutual influence of information from two aspects. To the best of our knowledge, transformers has not been implemented before for HDD RUL prediction using the comprehensive 10-year quarter-by-quarter health data from Backblaze [9]. We show that our new network outperforms state-of-the-art architectures and produces highly accurate point estimates of the RUL.

2. 2.

   Attention Mechanism instead of Feature Engineering: Our contribution also lies in using an attention mechanism for the encoder-decoder netowrk, This removes the need of feature selection as explored in [10].

3. 3.

   Confidence interval and error margin evaluation: We propose a new confidence interval evaluation to provide a robust range of RUL for a hard-drive. This instils confidence around when a hard-drive will fail.

The paper is organized in the following way. In Section II, we go over related work in the field. Section III provides an overview of why feature engineering is tedious and the shift to attention mechanisms. Section IV talks about the details and architecture of the Vanilla Transformer. Section V introduces the proposed method and Section VI divulges into the different configurations and experimentation done with the dataset. Lastly, Section VIII looks at potential avenues for future research.

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T. ) is a monitoring system included in hard disk drives (HDDs). Primarily, it senses and creates logs of various health indicators of drives thereby enabling proactive planning against imminent hardware failures.

Many data-driven approaches use S.M.A.R.T. features as their input data and provide predictions on the RUL of the hard-drives. The architectures based on deep learning can automatically collect the key feature information from the original data to perform end-to-end prediction by modeling the functional relationship between the hard-drive degradation process and the S.M.A.R.T. monitoring data.

To date, many deep-learning architectures have been proposed for RUL predictions: TCNN [11], LSTM [11], Stacked LSTM [12], Bi-directional LSTM [13], Spatio-temporal Anomaly Detection LSTM Networks [1], Encoder-Decoder LSTM [10], and Ensemble Learning approaches [14]. All of these architectures have modeled the dataset in a specific way and have applied feature engineering to sanitize the data. However, the literature does not contain a case where attention mechanisms and transformers have been used for remaining useful life (RUL) prediction of hard disk drives. This opens up an avenue to apply transformers for prognosticating impending faults in HDDs and reporting the same.

Transformers are appealing to time series problems such as RUL estimation as they have demonstrated excellent modeling capability for long-range dependencies and interactions in sequential data. The vanilla transformer has undergone many iterations that have been used for a variety of time series activities to meet unique issues in forecasting.

In recent studies, there have been examples of different variants of Transformers employed for specific tasks like forecasting [15, 16] and anomaly detection [17, 18]. Specifically, seasonality or periodicity is an important feature of time series observations [19] leveraged by transformers for generating predictions[8].

Time-stamped data from sensors are the backbone of fault prognosis in industrial electro-mechanical components. It makes possible the identification of temporal patterns, including seasonality or cyclical behaviors, which are crucial for predicting future conditions or occurrences whilst sensors continuously collect data on various aspects of the physical system in real-time. DAST exploits these two features to extract valuable contextual information. TFBEST is a novel transformer that builds on the DAST architecture by replacing the encoding with a learnable positional encoding based on LSTM.

This section introduces a new dataset format, associated experimental settings, and trials on 10-years of Backblaze data to assess TFBEST performance in comparison to cutting-edge RUL prediction techniques and to confirm the benefit of the new architecture.

We propose TFBEST, a novel transformer architecture for prediction of of hard-drive failures and highlight its advantages over the vanilla Transformer and DAST. To begin with, existing deep learning-based RUL prediction approaches rely heavily on the RNN/CNN architecture. While the use of LSTM has provided very good results in previous works, most architectures have to depend on careful feature engineering to extract the useful S.M.A.R.T. features. In contrast to preceding architectures, our solution is based on the Transformer architecture, which relies solely on the self-attention mechanism to analyze S.M.A.R.T. features. Our architecture is also superior to the DAST architecture by incorporating time-step positions via a learnable encoding technique. This has improved performance of predictions significantly.

We also use a new confidence-margin interval to provide a more reliable RUL estimate. RUL predictions can be very far from real truth if we only rely on a single output from the model. It is also more useful to predict a range of RUL for a particular log so that the datacenter can evaluate when to change the drive in a timeframe. Compared to previous literature, where either deep-learning models are used as a binary classifier or predicting a single value, our metric is inspired by confidence around failure. We rely on RUL sequences which create an overlap across 60 days by a rolling window. Experimental studies with the above RUL prediction method validate the advantage of our method and model.

We propose TFBEST, a novel transformer architecture for HDD RUL estimation. TFBEST uses a learnable positional encoding based on LSTMs and the self-attention mechanism to process the whole sequence of CBM data. It uses a sensor encoder and a time step encoder to simultaneously record the weighted characteristics of thee data. The TFBEST model adaptively learns the significance of various sensors and time steps without the requirement for feature selection through attention mechanism. The performance of our method for RUL prediction is better than state-of-the-art deep RUL prediction methods, according to experimental findings on actual hard-drive health data spanning 10 years. We also propose a new statistic - a confidence margin which produces a point estimate, error-margin and confidence interval that gives a range of RUL values during which the hard-drive may fail. In a future study, we intend to train and understand the transformer’s performance on other manufacturers. Another potential direction we want to explore is to make the transformer more resilient against cyber attacks. Deep learning models are vulnerable to adversarial attacks such as data poisoning attacks or tampering of weights. Architectural hardening is one way to make these models robust against such adversaries.

The research reported in this publication was supported by the Division of Research and Innovation at San José State University under Award Number 23-UGA-08-044. The content is solely the responsibility of the author(s) and does not necessarily represent the official views of San José State University.