Dependable Intrusion Detection System for IoT: A Deep Transfer Learning-based Approach

Six key motivations studied inspired this research paper. First, most proposed IDSs focus on detecting a limited number of cyber-attacks [1]. In this case, our proposed model’s target is to identify and classify a wide range of cyber-attacks efficiently (e.g., DoS, data injection, MITM, backdoor, PCA, scanning, and ransomware). Such attacks are significantly more challenging to detect at low computational complexity.

There is limited device profiling in the existing research[14]. Therefore, this paper considers the behavior of seven different IoT devices (e.g., fridge, GPS tracker, motion light, garage door, modbus, thermostat, and weather sensor) to increase the integrity properties of our proposed DTL-based P-ResNet model.

Furthermore, most proposed IDSs did not consider automatic attack type classification and dependability [6]. However, our proposed DTL-based P-ResNet model can automatically detect different attacks and efficiently classify different attack scenarios while ensuring dependability.

Most importantly, the existing TML and DL-based IDSs have apparent advantages in detecting various cyber-attacks [28]. However, these models require a large amount of training data, and the size of training datasets affects the accuracy of the model when the training data is insufficient, as has been confirmed by the subsequent experiment [15]. As a result, the TML and DL models need sufficient training data, and it is challenging to train an efficient IDS model only depending on a small-scale of target domain data.

Moreover, IoT-based applications typically consist of lightweight communication protocols with limited computational power and storage capacity [8]. However, traditional security mechanisms require high computational capabilities [15]. As a result, these conventional security mechanisms cannot be directly deployed for IoT-based applications due to their heterogeneous nature and characteristics (e.g., less scalability, inefficiency) [2, 8]. Furthermore, as IoT applications operate inside the network, traditional security mechanisms seem to be insufficient because of their features, which only identify external attacks [10]. Because of the overwhelming rise of IoT threats and vulnerabilities, traditional security mechanisms have already shown their inadequate strength to fight against targeted attacks, and associated anomalies [11]. As a result, the potential threats to the heterogeneous IoT ecosystem are constantly emerging and changing, and the exact nature of these threats is unknown.

To address the last two dependencies of the TML and DL models, we propose a dependable deep transfer learning-based residual neural network (P-ResNet) IDS that can train efficiently using only a small scale of target domain data and understand its impact with low computational capabilities. The proposed P-ResNet model significantly enhances the performance of neural networks with more layers. Generally, our proposed model tries to tackle those dependencies by directly bypassing the input data to the output layer, allowing the output layer to access the raw data. Therefore, we transfer the knowledge in source domain data to the target domain through the proposed P-ResNet model and combine the target domain data with the same model to construct an efficient and dependable IDS for the target domain to improve the detection accuracy for any heterogeneous IoT ecosystem.

Finally, most of the previously proposed IDSs didn’t consider dependability performance analysis [4]. In this case, we have analyzed the dependability performance to ensure the availability, efficiency, and scalability features of our proposed deep transfer learning-based P-ResNet IDS model.

Deep neural network (DNN) depth is an essential aspect of model performance[14, 27]. More complex features may be extracted by a DNN algorithm with more layers [24]. As a result, deeper DNNs may theoretically result in better performance. If the number of DNN layers increases, the degradation problem causes the DNN algorithms’ to become more saturated [6]. The deep residual network (ResNet) tries to tackle this problem by directly bypassing the input data to the output layer, allowing the output layer to access the raw data. Our proposed P-ResNet model consists of multiple residual blocks, each of which may be defined as- $U(x)=V(x)+x$, where x denotes the input data, V(x) denotes the identity residuals mapping function, and U(x) denotes the mapped solution function.

Deep transfer learning is a supervised machine learning technique that uses a small-scale dataset to apply a pre-trained model [24]. The definition of deep transfer learning has been given in terms of source and target domains. A labeled domain IoT dataset $D$ consists of data instances $X$ and a label $Y$. A task $T$ is defined as the connection of label $Y$ with a prediction function $f$ to be learned from the labeled domain data. The source domain and target domain datasets are denoted by $D_{s}=(X_{s},Y_{s})$ and $D_{t}=(X_{t},Y_{t})$, while the source tasks and target tasks are denoted by $T_{s}=(Y_{s},f_{s})$ and $T_{t}=(Y_{t},f_{t}),$ respectively. We assume that $T_{t}$ and $D_{t}$, which represent the ubiquitous computing issue to be solved and the corresponding labeled dataset, are accessible.

Source Domain: The domain where the initial model is located. The source domain data ($D_{s}:(X_{s},Y_{s})$) is the combination of ${(X_{s1},Y_{s1}),(X_{s2},Y_{s2}),...,(X_{sn},Y_{sm})}$, in which the class of the source domain ($Y_{s}$) is 0, and 1, where the normal and attack scenarios are represented by 1 and 0, respectively.

Target Domain: The domain has a new type of attack. The target domain data ($D_{t}:(X_{t},Y_{t})$) is the combination of ${(X_{t1},Y_{t1}),(X_{t2},Y_{t2}),...,(X_{tn},Y_{tm})}$, in which the class of target domain ($Y_{t}$) is 0 and 1, where the normal scenario is represented by 1 and the attack scenario is represented by 0.

The number of source and target domain data is represented by n and m, respectively. Furthermore, the source domain label ($Y_{s}$) and the target domain label ($Y_{t}$) contain only “normal” and “attack” data, but attackers in the source domain and target domain may be different. Although the source domain label ($Y_{s}$) and the target domain label ($Y_{t}$) have the same feature space, their performance in specific features is different. To measure the difference between the source domain and the target domain, we have used the following formula (1), which is called Maximum Mean Discrepancy (MMD) [12].

We propose a deep transfer learning method that transfers DNN weights learned on different attack classification problems on $D_{s}$ to another DNN trained to solve $T_{t}$ on $D_{t}$. Our proposed method in Figure 3 belongs to the category of inductive transfers since the source tasks and target tasks are distinct $(T_{s}\neq T_{t})$. The steps are as follows:

• •

  Definition of source domain $\mathbf{D_{s}}$ and source task $\mathbf{T_{s}}$: $X_{s}$ is firstly built by considering $M_{c}$ multiple channel IoT datasets (e.g. $modbus,motion\_light,weather$ sensor etc.). Each $M_{c}$ sequence in the $n^{th}$ dataset $(1\leq n\leq M_{c})$ is decomposed into single channels, each of which is divided into different segments $S$ of length $L$ using a sliding_window_approach. All the segments are combined to form the source dataset $X_{s}$, which is defined by the following formula (2):

  $\displaystyle X_{s}=\bigcup_{n=1}^{M_{c}}\left\{x_{i}^{(n)}\epsilon\mathbb{R^{L}}|\left(1\leq i\leq S_{n}\right)\right\}$

  (2)

  Here, $x_{i}^{(n)}$ refers to the $i^{th}$ segment of the $n^{th}$ source dataset $X_{s}$, and $S_{n}$ refers to the total number of segments obtained from the $n^{th}$ source dataset $X_{s}$. That is to say, $X_{s}$ is the union of all segments $S$ extracted from the multiple channel $M_{c}$ source datasets $X_{s}$. The source task $T_{s}$ is defined as the classification of different attacks on the source domain $D_{s}$. Source labels $Y_{s}$ are defined by the following formula (3):

  $\displaystyle Y_{s}=\bigcup_{j=1}^{M_{c}}\left\{y_{i}^{(n)}\epsilon\left\{1,2,.,C_{s}\right\}|\left(1\leq i\leq S_{n}\right)\right\}$

  (3)

  Here, $C_{s}$ is the number of classes of the source domain (e.g., normal and attack), and $y_{i}^{(n)}$ indicates the label of $x_{i}^{(n)}\epsilon X_{s}$. Furthermore, $f_{s}$ is the source prediction function which attributes each $x_{i}^{(n)}$ to its corresponding label $y_{i}^{(n)}$, whether normal or attack.

• •

  Learning of source prediction function $\mathbf{f_{s}}$: A single-channel DNN is used to learn the source prediction function $f_{s}$. For the single-channel DNN architecture, a batch normalization layer used to perform a regularization on the segments in the source dataset $X_{s}$ to address the issue of the heterogeneity of the source data. Assuming the single-channel DNN contains $H\epsilon N^{*}$ hidden layers, we denote the weight matrix and bias vector of the $j^{th}$ layer $(1\leq j\leq H)$ as $W_{j}$ and $B_{j}$, respectively. Finally, a softmax layer with $C$ neurons is added, with each neuron of the layer outputting a value that estimates of probability to its corresponding class. This way, the single-channel DNN can classify the segments of source dataset $X_{s}$ using the source labels $Y_{s}$.

• •

  Initialization of a multi-channel DNN: A multi-channel DNN is defined to learn the target prediction function $f_{t}$. It is trained using target dataset $X_{t}$ which contains multi-channel segments $X\epsilon\mathbb{R^{L\times S_{n}}}$, with $S_{n}$ being the number of channels of the target dataset $X_{t}$ and target label $Y_{t}$ which contains associated labels $Y\epsilon\{1,2,...,C_{t}\}$ with $C_{t}$ being the number of classes of the target problem. For the multi-channel DNN architecture, a batch normalization layer is applied to the segments to perform an operation akin to a standard normalization on the input of the network. The $S_{n}$ sensor channels are then separated. The $k^{th}$ sensor channel $(1\leq k\leq S_{n})$ is processed by an ensemble of hidden layers of the same number and type as the hidden layers of the single DNN. We refer to this ensemble of layers as a branch of the multi-channel DNN. The output of each branch is then concatenated and connected to fully-connected layers. A softmax layer with the target class $C_{t}$ neurons is added to output class probabilities for the $C_{t}$ target classes.

• •

  Transfer of weights from single to multi-channel DNN: The weights $W_{j}$ and biases $B_{j}$ of the $H$ hidden layers of the single DNN learned on $\{D_{s},T_{s}\}$ are transferred to the branches of the multi-channel DNN. In other words, the $j^{th}$ layer of the $k^{th}$ branch (for $1\leq j\leq H$ and $1\leq k\leq S_{n})$ has its weight and bias matrices $W_{k}^{(s)}$ and $b_{k}^{(s)}$ initialized as $W_{k}$ and $b_{k}$, respectively.

• •

  Learning of target prediction function $\mathbf{f_{t}}$: The multi-channel DNN is fined-tuned using $(X_{t},Y_{t})$ to learn $f_{t}$, which is the predictive function for the target ubiquitous computing problem.

Figure 2 shows the block diagram of the proposed model for IDS, which predominantly includes two parts: the first is the model training part, and the second is the intrusion detection part. For the training part, after prepossessing the network data, we applied it to our proposed model.

The most significant parameters of this model have been determined through subsequent empirical experiments. The model with the optimal prediction performance on the training set has been selected as the final intrusion detection model for heterogeneous IoT applications. For the intrusion detection part, we have trained the P-ResNet model with a randomly selected training dataset and validated the model with a validation dataset. The detection performance of the models under the discrete types of parameters is compared. We have also analyzed the dependability performance of our proposed model. Finally, the optimal dependable model has been selected as the final detection model in the field of heterogeneous IoT applications. Table II shows the hyper-parameters of the proposed model.

The network architecture of the proposed P-ResNet model is shown in Figure 3. The input of the network is the same shape. There have been four layers in our proposed model without the input layer. The first layer is the combination of 64 filters of kernel length 8, the second layer is the combination of 128 filters of kernel length 8, the third layer is the combination of 256 filters of kernel length 5, and the last layer is the combination of 128 filters of kernel length 3. Each layer is a one-dimensional convolution layer is followed by a BatchNormalization() and Rectified Linear-Unit (ReLU) activation function, which is called the feature smoothing layer (FSL). FSL has been used to keep the tensors’ shape and smooth the pre-trained tensors to learn features from the new training dataset. After the pre-trained residual blocks (transfer pre-trained knowledge), the FSL has been inserted. The feature extraction layer (FEL) used the first three operations to double the number of channels and fatten the feature map into one dimension. The FEL extracted operation-based features and abstracted important information to describe the previous layer, which improves the final fully-connected layer with these abstracted features.

Each block takes the previous block’s outputs as inputs for the current block and performs some non-linearity operations to transform it into a multivariate series whose dimensions are defined by the number of filters in each layer. The fifth layer is the combination of a GlobalAveragePooling1D() operation, which takes the input of the previous block and averages each series. This operation drastically reduces the number of parameters in a deep model while enabling the use of a class activation map [2] which allows an interpretation of the learned features. The output of the gap layer is then fed to a softmax classification layer whose number of neurons is equal to the number of classes in the dataset. To reduce the overfitting problem of the proposed model, we have considered various strategies, including data augmentation to the training set, adding Gaussian noise during training time, decreasing the network size, and the EarlyStopping method. Skip connections enable feature re-usability and stabilize training and convergence. Finally, we have deployed a skip connections technique of our proposed model by using a vector addition method. Then the gradient would be multiplied by one and its value, which was maintained in the earlier layers. This enabled us to identify the effect of deep transfer learning.

The dataset has been generated from various heterogeneous sources, including seven IoT sensors, (e.g., fridge_sensor, GPS_tracker_sensor, motion_light_sensor, garage_door_sensor, modbus_sensor, thermostat_sensor, and weather_sensors). The fridge_sensor measures the fridge_temperature and temperature_condition, whether it is high or low, based on a preset threshold value. The GPS_tracker_sensor determines the geographic position, such as latitude and longitude of an object. The motion_light_sensor includes two characteristics, i. e., motion_status and light_status. The light_status is changed based on the pseudo-random signal generated by the motion_status. The garage_door_sensor observed two-states, door_state and sphone_signal. The first is observed when the door is open or closed and the second is where the signal is true or false when receiving the door signal on a phone. The modbus_sensor simulates the functionality of the Modbus devices for communication with each other with the help of some register (e.g., input_register, discrete_register, holding_register, and a coil_over_register). The thermostat_sensor governs the temperature of a physical system by regulating a heating or cooling system, where current_temperature and thermostat_status are the fundamental features. The weather_sensors produce air_temperature, air_pressure, and air_humidity data.

Each IoT dataset has been combined into one CSV file by a Python script. The combined dataset has 22 features, including the label feature. The label feature contains two quantitative values, i.e., 0 and 1, which indicate normal and attack scenarios, respectively. Nine types of cyber-attacks (denial of service (DoS), distributed denial of service (DDoS), data injection, man-in-the-middle (MITM), backdoor, password cracking attack (PCA), scanning, cross-site scripting (XSS), and ransomware) have been considered against the IoT sensors mentioned above. Figure 4(a) shows the statistics of the abnormal dataset with attacks and figure 4(b) shows the class distribution of the combined dataset.

To achieve optimal performance and improve the learning process, it is necessary to clean and prepare the dataset before applying DTL methods. Data preparation is generally done by removing unnecessary features, checking the variation of independent features, converting non-numerical features, removing outliers, and replacing missing values if they exist. The two fundamental steps apply during the data preparation process. The first is data pre-processing, and the second is data transformation step.

As discussed in the previous section, all the datasets have been converted into a combined dataset by a Python script containing the training and validation data. First of all, we have divided the combined dataset into the training data set (80%) and the test data set (20%) by using the train_test_split method of the scikit_learn library. The training data set has been used to train the selected models. On the other hand, the test data set has been used to further assess the trained classifier of those models.

Furthermore, we again split the training data (80% of the total data) into the new training data (80%) for training the selected model and validation data (20%) for hyperparameters’ optimization. Proportions of 80% for the training dataset and 20% for the validation dataset have been chosen as suggested in [29]. To avoid the overfitting problem, this splitting ratio has been considered the best ratio between the training and the validation dataset [31]. We want to highlight some DTL models that give better performance than others [2]. We have selected Fully Convolutional Networks (FCN), LeCun Network (LeNet), Inception Network (IncepNet), Multi Channel Deep Convolutional Neural Network (MCDCNN), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Multilayer Perceptron (MLP), and our proposed Residual Neural Network (P-ResNet). We have also considered DL algorithms because some of their variants have been successfully applied to solve classification tasks related to intrusion detection [12, 32]. Therefore, we have considered Long Short-Term Memory (LSTM), Neural Network (NN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) algorithms because of their optimal performance. The overall performance of the selected models has been evaluated with a wide range of tested hyperparameters. Figure 6 summarizes the steps involved in evaluating the performances of the selected models.

In recent years, DL algorithms have advanced significantly, and some of the variants of DL algorithms have been successfully applied to solve classification tasks related to intrusion detection [12]. Therefore, we considered DL algorithms in this section because of their optimal performance. In this experiment, we considered LSTM, NN, CNN, and RNN. The LSTM and NN models show the highest accuracy score of 0.82. Dramatically, the other performance metrics of these two models are almost similar. Figure 7 shows the performance comparison between selected DL models and the proposed DTL-based P-ResNet model. For the LSTM model, we used three hidden-layers, where the units of hidden layers were 128, 100, and 64, respectively. The tanh is the hidden layer activation function, and Adam is used as an optimizer. Also, sigmoid is used as a network output activation function, and “categorical_crossentropy" is used as a loss function. The RNN and CNN models outperform most TML algorithms, with an accuracy score of 0.80 and 0.79, respectively. Furthermore, both the NN and CNN models used the same optimizer and activation function but different types of the hidden layer activation function. Particularly for the RNN model, the softmax is used as the network output activation function, and “categorical_crossentropy" is used as the loss function. On the other hand, for the CNN model, the number of hidden layers is four and “binary_crossentropy" is used as the loss function. Finally, considering all the significant parameters of the DL models, the NN and LSTM models have shown the optimal performance, which indicates that most of the predicted labels are correctly classified. However, when we applied our proposed P-ResNet model to the same dataset, the model performed significantly better in most cases. In such cases, accuracy, precision, f1-score, and ROC AUC are better than in other experimental scenarios.

In recent years, the progression of deep transfer learning and its great success in different fields mean it has emerged as a potential solution for dependable network intrusion detection [12]. Besides considering the fundamental evaluation criteria, we also incorporated other deep learning evaluation criteria, for example, the numbers of hidden layers, units in the hidden layers, output layer activation functions, loss functions, etc. These criteria helped to calculate and report the classification results effectively. First, we consider the quantitative performance of DTL models. Table III shows the quantitative performance summary of the DTL models.

The proposed P-ResNet model shows optimal performance compared to the others, with an accuracy score of 0.87, precision score of 0.88, recall score of 0.86, f1-score of 0.86, and ROC AUC score of 0.83. We used four hidden-layers in this model, where ReLu is the hidden layer activation function. Also, softmax is used as a network output activation function, and “categorical_crossentropy" is used as a loss function along with the Adam optimizer. Figure 8 shows the accuracy score of every single epoch for both the training and validation phases of the selected models. Considering epoch numbers 170 to 200 for both phases, the flattening characteristics of the curve and accuracy are not increasing literally. Therefore, we have considered 200 epochs for our analysis. The proposed model shows the highest accuracy score in both the training and validation phases, whereas, the MLP model shows the lowest accuracy score during any setting of the epoch number within 1 to 200 in both the training and validation phases.

In detail, according to Figure 8, the accuracy of the MLP model starts around 0.67 for the training phase and 0.63 for the validation phase in epoch number 10. However, it increased dramatically to around 0.78 in epoch number 65 and 0.74 in epoch number 140 for the training and validation phases, respectively. However, for the training phase, the accuracy score (0.78) remains stable in epoch number 66 to 200.

On the other hand, for the validation phase, the accuracy score (0.74) remains stable between epoch numbers 141 and 200. As shown in Figure 8, the training and validation accuracy of CNN, IncepNet, LeNet, and MCDCNN models remains steady between epochs 1 and 200. The accuracy of LSTM and FCN models starts with a score of 0.82. However, this score rises gradually with the increase of the epoch number and reaches approximately 0.86 when the epoch number is 160 and then remains stable between epoch numbers 161 and 200 for both phases. The remarkable point is that the behavior of the training phase is almost identical to the validation phase. To better understand our proposed model Figure 9 shows the trend of the accuracy score of both phases. The proposed model’s accuracy increases rapidly at epoch number 60, reaching a peak of close to 0.87 at epoch number 169. However, as shown in Figure 9, it remains almost stable up to the early stopping checkpoint.

Next, we analyzed the losses of each model. The training and validation losses of every epoch are shown in Figure 10. The MLP model shows the highest losses in the training and validation phases. The highest loss of this model indicates that the model cannot provide a reliable classification between normal and attack scenarios. On the other hand, our proposed model shows the lowest loss for any epoch setting within 1 to 200. In detail, the loss of the proposed model starts around 0.41 for the training phase and 0.42 for the validation phase, as shown in Figure 11. However, it decreases to 0.27 for the epoch number 140 and 0.28 when the number is 145 for the training and validation phases, respectively. The loss score (0.27) remains stable between epoch number 141 and the early stopping checkpoint, which has been indicated by the red circle in Figure 11. The losses of the CNN, IncepNet, LeNet, and MCDCNN models remain almost steady during both the training and validation phases, between epochs 1 and 200, which is shown in Figure 10. The loss of the LSTM and FCN models is almost 0.42 at the beginning, which declines gradually to approximately 0.31 at epoch number 140 and remains stable for both phases.

During a concept drift scenario, the learning model drifts, which can be understood as changes in the relationship between the input and target output. In an IDS setup, their are significant challenges arising from the ‘concept drift’, especially in the context of streaming IoT data, where the dataset may not always be static but rather an evolving and moving target for detection. Authors in [33] highlight a set of open challenges faced by the modern ML-based IDS technologies and emphasize adopting concept drift within the solution framework. Gama et al. [34] highlight that transfer learning between the different contexts can be a promising solution to deal with the ‘concept drift’ challenges. McKay et al. [35] recently implemented an online transfer learning-based solution for concept drifting data streams. Hence, with confidence, we can argue that the proposed transfer learning-based solution has the potential to deal with concept drifting contexts, which we have scoped for future work.

The training time, testing time, and number of parameters were also calculated for each DL and DTL model. Table IV compares the total parameters, training time, and testing time for each model. In terms of the DL models, NN requires 2.660 seconds for a total of 50,985 trainable parameters, which is less testing time and a lower number of parameters than other DL models. Moreover, the NN model can be explained as other DL models were lazy learners who used the training phase to store the data and then used the data during the test phase to make a prediction, which makes the testing phase slower, though the number of parameters (trainable) was slightly increasing.

On the other hand, in terms of the training and testing time, the FCN model requires the longest training and testing time of 46756.338 seconds and 7.722 seconds, respectively, for a total of 265,986 trainable parameters. Surprisingly, the CNN model has the lowest testing time of 2.718 seconds and training time of 33869.838 seconds, the third-lowest among all the DTL models. Furthermore, the LSTM model has a testing time of 5.721 seconds and training time of 44859.869 seconds, the third-longest testing time after the LeNet model, which has a training time of 38936.911 seconds and testing time of 5.690 seconds. One would be concerned that the LSTM and LeNet models have more parameters than the CNN. Both the MLP and MCDCNN models have almost the same testing time and number of parameters. Our proposed P-ResNet model requires a testing time of 3.014 seconds and training time of 24401.586 seconds, for a total of 511,002 trainable parameters, which is less than most of the selected DTL models. This can be explained as our proposed model is a fast learner that uses the training phase to train efficiently, and then uses the data during the test phase to make a prediction, which makes the testing phase faster. Considering all the performance indicators, we think our proposed model can be used for practical applications to protect Industry 4.0 manufacturing and other critical heterogeneous IoT-based infrastructures where dependable IDS system and secure data processing are the main challenges.

In this section, we explored the dependability performance analysis of our proposed model. Dependability performance analysis includes availability, efficiency, and scalability features [4]. We consider various strategies to select features and then perform our proposed model to accurately classify normal and attack scenarios without any type of failure or undergoing a repair action, which maintains the availability of our proposed model. Moreover, extensive analysis and performance evaluation (e.g., accuracy, precision, ROC AUC, etc.) show that the proposed model is more efficient and demonstrates better performance than several other existing approaches with low computational loss. Figure 12 shows the efficiency and computational loss of the proposed model.

Finally, we observed an increase in the scalability properties of our proposed model by including various heterogeneous trusted data sources in the training dataset with maximum consistency that were acquired from a wider range of IoT sensors. As a result, our proposed model’s accuracy remained almost the same when we increased the epoch number from 125 to 200, indicating its scalability. Figure 13 shows the scalability performance of the proposed model.