A Multi-Stage Automated Online Network Data Stream Analytics Framework for IIoT Systems

There are two broad types of concept drift: sudden and gradual drifts [9]. A sudden drift is a rapid change in the data distribution over a short period of time, while a gradual drift occurs when a new data distribution gradually replaces a historical concept. Different drift detection methods have been designed to detect different types of drift.

Distribution-based methods and performance-based methods are two common types of drift detection methods [9]. ADaptive WINdowing (ADWIN) [10] is a popular distribution-based approach that utilizes an adaptable sliding window to detect concept drift. ADWIN identifies data distribution changes by computing and comparing the characteristic values of the old and new distributions, such as the mean and variance values [11]. A significant change in the characteristic values over time indicates that a drift has occurred. Through the use of an adaptable sliding window, ADWIN works well with gradual drifts and long-term changes. However, changes in the statistics of data windows are sometimes virtual concept drift, resulting in false alarms and unnecessary model updates.

Early Drift Detection Method (EDDM) [12] is a widely-used performance-based drift detection method that tracks model performance changes based on the change rate of a learning model’s error rate and standard deviation by using a drift threshold and a warning threshold. If the error rate of a model increases dramatically, it will indicate model performance degradation and the occurrence of concept drift. Thus, EDDM can detect all real drifts that have degraded model performance, and is effective in detecting sudden drift. However, EDDM is inferior to distribution-based methods for gradual drift detection.

After identifying a concept drift, learning models should be able to adapt to new concepts and enhance model performance. Existing drift-adaptive learning techniques fall into two primary categories: incremental learning and ensemble learning techniques.

Incremental learning is the process of learning each incoming data sample in chronological order and partially updating the learner. Hoeffding Trees (HTs) [9] is a basic incremental learning method that employs the Hoeffding inequality to determine the minimum number of data samples necessary for each split node, thus updating nodes to adapt to new samples. The Extremely Fast Decision Tree (EFDT) [13] is a state-of-the-art incremental learning approach and an improved version of HTs. It selects and deploys each node split as soon as it reaches the confidence value, indicating a useful split. EFDT is able to adapt to concept drift more precisely and efficiently than HTs. Online Passive-Aggressive (OPA) [14] is another incremental learning algorithm that adapts to drift by passively reacting to correct predictions and aggressively responding to any errors.

Several incremental methods are designed based on traditional ML algorithms. K-Nearest Neighbors with ADWIN drift detector (KNN-ADWIN) and Self-Adjusting Memory with KNN (SAM-KNN) [15] are two improved versions of the traditional KNN model for online data analytics. KNN-ADWIN adds an ADWIN drift detector to the traditional KNN model and uses a dynamic window to determine which samples to retain for model updating. SAM-KNN, on the other hand, adapts to concept drift via the use of two memory modules: Short-Term Memory (STM) for the current concept and Long-Term Memory (LTM) for historical concepts [15].

Ensemble online learning models are advanced drift-adaptive learning methods that integrate the output of multiple base learners for performance improvement. Leverage bagging (LB) [16] is a basic ensemble technique that constructs and combines multiple base learners (e.g., HTs) using bootstrap samples and the majority voting strategy. LB is simple to construct, but sensitive to noisy data. Adaptive Random Forest (ARF) [17] and Streaming Random Patches (SRP) [18] are two advanced ensemble online learning methods that train multiple HTs as base models and employs a drift detector (e.g., ADWIN) for each HT to address concept drift. ARF uses the local subspace randomization strategy to construct trees, while SRP uses global subspace randomization to generate random feature subsets for model learning. Using global subspace randomization improves the learning performance of SRP but increases its model complexity and learning time.

Although ensemble online learning methods usually outperform incremental learning methods for dynamic data stream analytics, they are usually computationally expensive. Many deep learning methods are also unsuitable for data stream analytics due to their high complexity. Thus, despite the existence of many promising drift-adaptive learning methods, there is still much room for improvement. The purpose of this study is to propose an ensemble framework capable of balancing model performance and learning speed. Additionally, existing approaches focus only on model learning but ignore other necessary data analytics procedures, such as data pre-processing and feature engineering. Thus, this paper proposes a comprehensive data analytics framework that includes other typical data analytics procedures. Moreover, we explored the automation of various data analytics procedures with the aim of network automation.

The overview of the proposed MSANA framework for automated data analytics in IIoT systems is shown in Fig. 1. It consists of four primary stages: dynamic data pre-processing, drift-based dynamic feature selection, base model learning and selection, and online ensemble model development. At the first stage, to improve data quality, the incoming IIoT data streams are pre-processed using data balancing and normalization methods. The data balancing method is automatically selected from the dynamic undersampling and oversampling methods, depending on the specific performance and efficiency requirements. The details are discussed in Section III-B.1. The data normalization method is automatically selected from the dynamic min-max and z-score scalers, which are discussed in Section III-B.2. Secondly, dynamic feature selection methods are utilized to select the relevant features of data streams for specific tasks. The proposed Drift-based Dynamic Feature Selection (DD-FS) method consists of two basic feature selection methods: the variance threshold for discarding low-variance features and the select-k-best method for removing unimportant features. In DD-FS, the features are re-selected when a concept drift is detected, which is discussed in Section III-C. The third phase consists of online base model learning and dynamic model selection. In online base model learning, six base online learners are trained using the cleaned data streams to perform basic data stream analytics. Specifically, ARF models with two different drift detectors, ARF-ADWIN and ARF-EDDM, are selected as the leader models, while the two best-performing models among four common online learning methods (i.e., EFDT, KNN-ADWIN, SAM-KNN, and OPA) are selected as the follower models. If concept drift occurs, the dynamic model selection module will re-select the base models according to new data distributions, and then update these learning models on the new concept samples in a time window. The details of base model learning and selection are discussed in Section III-D.1. In the last phase, the proposed W-PWPAE method is used to integrate the output of the selected base learners based on their prediction probabilities and real-time error rates for the online model ensemble, which is discussed in Section III-D.2. The learning models are updated with the occurrence of concept drift, which is detected by two common drift detection methods, ADWIN and EDDM. Finally, the prediction results of the ensemble learner on the processed data streams are delivered.

Data pre-processing is mostly used to enhance the quality of data streams in order to increase model learning performance. In IIoT data streams, class imbalance and feature range difference are two potential data quality issues, which can be solved by data balancing and normalization methods, respectively. On the other hand, in dynamic IIoT systems, the class distributions and feature ranges are all dynamic variables that may change significantly over time. Thus, in contrast to the static process in traditional ML pipelines, data pre-processing in online data stream analytics should be dynamic procedures that must be updated on a continual basis.

As the original features are usually not the optimal features for specific tasks, the primary objective of feature selection is to return the updated data with optimal input features for performance improvement. Feature selection can also improve learning efficiency by discarding irrelevant and noisy features. The proposed Drift-based Dynamic Feature Selection (DD-FS) method used in this work consists of two feature selection techniques: variance threshold and select-k-best.

Variance threshold is a feature selection method that aims to eliminate all low-variance features. The variance $\sigma^{2}$ of each feature can be denoted by [21]:

where $n$ is the number of processed samples, $x$ is an input feature, and $\bar{x}$ denotes the mean value of $x$. Using the variance threshold approach can remove the features whose variance is lower than a given threshold. A low variance indicates that the corresponding feature is often uninformative as it has the same values across the majority of data samples. By removing low-variance features, the learning efficiency of models can be increased.

Select-k-best is a popular feature selection method in which the correlations between each input feature and the target variable are calculated as feature importance scores, and then the features with the $k$ highest importance scores are selected [22]. The feature importance scores can be computed using the Pearson correlation coefficient, a commonly-used metric to measure the correlations between two variables. It can be denoted by [22]:

where $x$ is an input feature, $y$ is the target variable, $\bar{x}$ and $\bar{y}$ are the mean values of an input feature and the target variable, respectively.

The Pearson correlation coefficient has a range of -1.0 to 1.0, with -1.0 indicating a perfect negative relationship, 1.0 indicating a perfect positive relationship, and 0 indicating that the two variables are fully uncorrelated. Thus, it can be used to quantify the strength of a relationship between each feature and the target variable, making it easy to assess and compare the importance of different features.

Algorithm 1 illustrates the primary procedures of DD-FS. At the first stage, the initial training set $S_{train}$ is learned by the variance threshold (${MF}_{1}$) and the select-k-best method (${MF}_{2}$) to generate the initial optimal feature set $F^{\prime}$. The initial $F^{\prime}$ is then utilized to update the feature list of each incoming data sample $x_{i}$ from the online test set $S_{test}$. If a drift is detected by the drift detectors, the variance threshold and select-k-best feature selection methods will re-learn the recent data samples in a time window $s$ as the new concept samples to generate an updated optimal feature set $F^{\prime}$. During the entire online data stream analytics process, this drift-based feature re-selection procedure is automatically repeated each time a concept drift is detected. The drift adaptation functionality of DD-FS is based on the assumption that when concept drift occurs and data distribution changes, the best suitable feature set will change as well.

The proposed model learning framework consists of two stages: base model learning & selection and online model ensemble. The base model learned in the first stage is selected to construct an ensemble model in the second stage.

The proposed system was implemented in Python 3.7 by extending the River [21] library on a computer equipped with an i7-8700 CPU and 16 GB of memory, representing an IIoT cloud server machine for large data stream analytics.

The proposed approach is evaluated on two public IIoT security datasets: IoTID20 [7] and CICIDS2017 [8]. IoTID20 is a relatively new IoT dataset that was created by generating IoT network traffic data from both legitimate and malicious IoT devices, including 83 different network features. CICIDS2017 is a public network security dataset that was contributed by the Canadian Institute of Cybersecurity and contained state-of-the-art cyberattack scenarios. Due to the fact that the CICIDS2017 dataset was created by launching a variety of different attack types in different time periods, the attack patterns in the dataset have changed over time, resulting in six concept drifts, as shown in Figure 2. For the purpose of this work, A representative IoTID20 subset with 6,252 data and a sampled CICIDS2017 subset with 28,303 records are utilized for the model evaluation.

The IIoT data analytics use case solved by the proposed system is anomaly detection, which can be regarded as a binary classification problem by labeling each data sample in the two datasets as a normal sample or an attack sample. The proposed framework is evaluated using the combination of hold-out and prequential validations. For hold-out validation, the first 10% of data is utilized for training the initial base models, and the remaining 90% is used for online testing of dynamic data streams. Prequential validation, also known as test-and-train validation, is utilized to evaluate the proposed model for online learning. In prequential validation, each input sample in the online test set is firstly tested by the learners to monitor their real-time performance, and then learned by the learning model for potential model updates [6].

To provide a comprehensive analysis of the experimental results, the proposed framework is evaluated from multiple perspectives. Firstly, from the effectiveness perspective, four performance metrics, including accuracy, precision, recall, and F1-score, are used to evaluate the proposed framework. As the IoT anomaly detection datasets are imbalanced, precision, recall, and F1-score are used with accuracy to provide a comprehensive view of the model performance. Secondly, from the efficiency perspective, two ML and data analytics-related Quality of Service (QoS) parameters, latency and throughput [23], are used to assess the proposed framework’s learning efficiency. Latency indicates the inference time of learning models, and is computed by the average inference/test time per packet/data sample. Throughput refers to the number of processed data samples/packets in one unit of time (e.g., second). Low latency and high throughput are two primary performance requirements for ML and data analytics models [23]. Moreover, the Probability Density Function and contributors (model components) of the latency are also used to evaluate the efficiency of the proposed model. To achieve real-time analytics, efficient learning models should strike a balance between prediction accuracy and latency.

Figures 2 & 3 and Table I illustrate the performance comparison of the proposed MSANA method against other state-of-the-art online adaptive learning methods presented in Section II-B, including ARF-ADWIN [17], ARF-EDDM [17], SRP [18], EFDT [13], KNN-ADWIN [15], SAM-KNN [15], OPA [14], LB [16], and PWPAE [6].

As shown in Fig. 2 and Table I, on the CICIDS2017 dataset, the two leader models, ARF-ADWIN and ARF-EDDM, achieve high accuracy of 98.95% and 99.02%. This demonstrates why they were chosen as leader learners. Among the four follower models (EFDT, KNN-ADWIN, SAM-KNN, and OPA), SAM-KNN and KNN-ADWIN are the two best-performing models on the CICIDS2017 dataset, so they are selected as the two final follower models to construct the initial ensemble model. After using the two leader models and two follower models to build an ensemble learner, the proposed MSANA achieves the highest accuracy of 99.32% and F1-score of 97.95% among all the evaluated online learning models. Additionally, the inference time of MSANA is only 3.53 ms per packet, which is less than for other ensemble techniques (SRP, LB, and PWPAE). The throughput of the proposed MSANA model is also at a high level (283.29 samples per second).

The evaluation results for the online learning models on the IoTID dataset are shown in Fig. 3 and Table I. Similarly, the two leader models, ARF-ADWIN and ARF-EDDM, achieve high F1-scores of 99.15% and 99.13%. For the follower models, EFDT and KNN-ADWIN are selected to build the initial ensemble model due to their better performance when compared with SAM-KNN and OPA. The proposed MSANA method achieves the second-highest accuracy of 98.88% and the second-highest F1-score of 99.41%. Although the proposed MSANA method has a slightly lower accuracy than the existing PWPAE method (98.88% versus 98.90%), it has much shorter inference time (2.81 ms versus 9.08 ms) and much higher throughput (355.87 versus 110.13 samples per second) than PWPAE. This is because, in comparison to the PWPAE method, the proposed MSANA method uses a window-based strategy and selects lightweight base learners with greater computational speeds to build the ensemble model. Thus, the proposed MSANA method is still the best model in terms of balancing model performance and efficiency among the evaluated models.

Moreover, to evaluate the efficiency and the feasibility of the proposed framework, the Probability Density Functions (PDF) of the inference time/latency of the proposed framework on the CICIDS2017 and IoTID20 datasets are shown in Figs. 4 and 5, respectively. Figures 4 and 5 illustrate that the vast majority of data samples were processed within 6 ms and 4 ms, respectively. There are several data packets in the two datasets that took a relatively long time (up to 16 - 38 ms) to be processed, as the learning models were updated after detecting concept drifts. Due to the low frequency of the occurrence of concept drift in real-world IoT data streams, only a tiny percentage of real-world IoT data would take a relatively long time to process. Additionally, even if the concept drift frequency of the experiments is high, the average inference time/latency of the proposed framework is still at a low level (3.53 ms and 2.81 ms on the two datasets), as shown in Table I. Please note that this inference time will be reduced dramatically when using high-performance computing devices that are usually deployed on IIoT cloud servers.

Lastly, to identify the contributors to the inference time/latency, the inference time of each component of the proposed MSANA framework is computed and provided in Table II. As shown in Table II, the inference time of most components/procedures of the proposed framework is shorter than 0.1 ms, and the proposed W-PWPAE ensemble model has low latencies on the two datasets (0.15 ms and 0.12 ms) due to its low computational complexity. The primary contributor to the inference time is the online base model learning procedure, since it takes time for multiple online learners to learn the data streams. Thus, the proposed framework itself has high efficiency, which can be further improved by replacing the base learners with lightweight learning models with lower complexity.