XAI-based Feature Selection for Improved Network Intrusion Detection Systems

Many forms of network intrusions are often encountered. In this work strom2018mitre , we examine the primary network attacks inside the standard MITRE ATT&CK architecture. As a result, the following categories are used to group the network traffic:

Benign traffic: Benign traffic is the regular data flow that is gathered from the network.

Network Service Discovery / PortScan (PS) [T1046, MITRE ATT&CK ID]: The goal of the intrusion is for the attacker to identify the victim’s computer. Finding weak spots and potential entry sites is frequently the initial stage of an attack. Without actually completing the connection, the functionality is to send the victim a connection solicitation. But in this attack, requests are sent to a variety of ports, and the ports that respond with a message are mapped as potential sites of entry lee2003detection .

Network Denial of Service [MITRE ATT&CK ID: T1498] / Denial of Service (DoS): The goal of this kind of assault is to make the target inaccessible to the network. A common illustration of this kind of attack is when a hacker persistently sends requests to establish a connection with a server. Nevertheless, the server never hears back from the origin after accepting the solicitation and sending an acknowledgment in the hopes of receiving a response. Consequently, until the server goes down, the memory for these open interactions is kept open and fully utilized. For further information on certain kinds of DoS assaults, see CICIDS ; SENSOR .

Brute Force [T1110, MITRE ATT&CK ID]: This type of attack involves the attacker trying every possible password to gain access to the victim’s network. Frequently, this technique is combined with the ability to guess the most popular passwords. When a person uses a password that is weak or often guessed, it can become effective CICIDS .

Initial Access/Web Attack [MITRE ATT&CK ID: TA0001, T16 59, T1189]: This is a kind of online assault where the attacker takes advantage of security holes in websites. An attacker may, for instance, use a software defect, misconfiguration, or malfunction to exploit a public-facing application and obtain access to the program’s underlying instance or container. Drive-by Compromise assaults are also included in the category of web attacks Web_attack_Ref2 ; however, web attacks (such as SQL injection and cross-site scripting) usually do not grant initial access to a remote server strom2018mitre .

First Access/Infiltration [MITRE ATT&CK ID: TA0001]: This attack happens when someone tries to obtain initial access to a system or application. Attacks of this kind can employ a wide range of strategies, including targeted spearfishing and taking advantage of flaws on web servers that are visible to the public. This assault can build its base by anything from a straightforward password reset to ongoing access via legitimate accounts and third-party remote services.

Infrastructure Compromise/Botnet [MITRE ATT&CK ID: T1 584.005, T1059, T236, T1070]: This type of automated assault is carried out via remotely operated machines that have been taken over by the attacker, where programs, or bots, replicate and imitate human behavior stone2009your . Its scripting approach makes it possible to scale and deploy it easily, which makes it the perfect tool for hitting several assault locations at once. As a result, it is a popular form of network assault on many computer networks.

Critical infrastructure across a range of industries is seriously threatened by the previously described sophisticated network attacks khan2021m2mon ; hussain2021noncompliance . These dangers undermine trust in the security of stored data in addition to jeopardizing service availability hussain2021noncompliance . IDS is used in computer network systems to detect and reduce harmful activity carried out by external and internal intruders mirzaei2021scrutinizer . Traditional IDS are often designed with the understanding that the behaviors of an intruder differ significantly from those of a legal user and that many unauthorized acts are detectable lukacs2015strongly . Researchers have investigated the application of AI models to automate the detection of network intrusions buczak2015survey ; dina2021intrusion , using recent advances in AI.

An important contribution to automating intrusion detection has been made by AI models. Nonetheless, because of the intricate interactions between the features in the models that go into producing their results, they are inherently black-box because of their innate complexity, understanding how and why the AI model produces its outputs is quite difficult. The black-box problem affects many AI models, such as deep neural networks and random forests. Though they have been successfully applied in many areas and have a high prediction accuracy, it is still difficult to understand the reasoning behind the models’ actions, especially when there are faults or blunders. Particularly in safety-critical applications such as network security via intrusion detection systems (IDS), these mistakes might have serious repercussions. To overcome these difficulties, it is important to offer justifications for the choices made by AI models in IDS botacin2021challenges . Extraction and comprehension of the primary features influencing these AI models’ actions in IDS constitute a primary step toward meeting this demand.

The area of explainable AI (XAI) has just come into existence as a means of addressing the black-box character of most AI models. With a particular focus on comprehending the elements that go into a model’s decision-making process in tasks including classification or regression, XAI seeks to shed light on the interpretability of AI models and it includes several subfields, such as methods that provide explanations for the behavior of AI models at the local level (unique to individual instances) or the global level (across several data instances). These methods can be model-agnostic, which means they can explain any kind of AI model, or model-specific (i.e., they are made to explain particular kinds of AI models). Furthermore, XAI uses explanation techniques such as feature value evaluation to provide a feature priority list for an AI model, or building surrogate models to explain complicated models using simpler ones.

The existing AI-based intrusion detection system produces output that is difficult for human analysts to comprehend. Analysts must thus manually review a large number of records in order to spot unusual activity. Furthermore, they encounter difficulties in locating and resolving problems with the AI models themselves. Therefore, in the field of network intrusion detection, obtaining high-accuracy results and also elucidating the behavior and decision-making process of the AI system becomes paramount. Trust in the AI algorithms will be established if security analysts can understand the judgments that the algorithms make. Given this, our XAI-based approach, which selects intrusion features using black-box XAI techniques, represents a significant advancement in the area.

After outlining the essential prerequisites for network intrusion detection and the necessity of improving feature selection via the use of XAI-based methodologies, we go on to describe the key elements of our XAI-based framework for choosing primary intrusion features in network intrusion detection assignments.

The low-level elements of our XAI-based feature selection process are now explained. The various elements (depicted in Figure 1) are elucidated thereafter.

Intrusion Dataset Loading: Our framework’s first step is to load the data from the database. We start with two well-known network intrusion detection datasets: CICIDS-2017 panigrahi2018detailed and RoEduNet-SIMARGL2021 mihailescu2021proposition .

Extraction of Features: The next phase in our framework is to choose a basic set of features from the database’s log files after the database has been loaded. The AI model’s results are significantly influenced by the feature extraction method since some characteristics may represent the nature of network traffic more accurately than others. To extract a basic set of characteristics for each network intrusion dataset, we employed the same methods as in prior works mihailescu2021proposition ; panigrahi2018detailed .

Reduction of Redundancy: After loading the database and finishing the basic feature extraction, the next step is to remove duplicate items from the traffic. Redundancy is frequently present in network traffic kwon2018understand , and eliminating it is essential to avert the intrusion detection AI model’s possible underperformance. We just need to remove every row that is precisely the same to do this. Moreover, we randomize the rows as a preprocessing step before training. This stage is required since attack labels may have been used to arrange the data sequentially. By arranging the rows in a random order, you can make sure that there is no bias introduced into the training data due to entry order.

Data Distribution: We also handle the problem of data imbalance in our architecture. Both of our datasets show an imbalance where the usual traffic greatly surpasses the attack traffic, as seen in Table 3. Remarkably, while developing strong AI models for network security, the issue of imbalanced network traffic types frequently arises barradas2021flowlens . We use the random oversampling technique moreo2016distributional to balance the dataset to address this problem. By oversampling the cases of the minority class, this strategy makes sure that all classes of network traffic are balanced. Our datasets are huge, and the random oversampling approach is successful in balancing them despite its simplicity moreo2016distributional . The AI model may learn from a representative collection of data by creating a more balanced distribution of cases across different classes through data balancing.

Normalization of Features: We use a conventional feature normalization step (min-max feature scaling) to avoid scale differences between distinct traffic features. By bringing all characteristics to a uniform numerical scale, this procedure makes sure that there are no magnitude differences throughout the dataset. We provide a consistent representation that makes training and analyzing the AI model easier by normalizing the characteristics.

Training Black-box AI Models: After the database is ready, we begin training the AI model. To do this, we separated the data into a training set, which comprised 70% of the entire set, and a testing set, which included the remaining 30%. Notably, we have included seven popular AI classification models at this stage: k-nearest neighbors (KNN), LightGBM, Adaptive Boosting (AdaBoost), support vector machine (SVM), random forest (RF), deep neural network (DNN), and multi-layer perceptron (MLP). Every model is trained using a unique set of parameters for optimal outcomes. We have carefully adjusted each model’s parameters to maximize its effectiveness (see A for a comprehensive setup).

Evaluating Black-box AI Models: In our architecture, testing each of the seven AI models with untested data comes after they have been trained. Next, we proceed to use the test data that has not been seen to examine how well these models work. To assess each AI model’s efficacy for this performance study, we first construct a confusion matrix for it. From there, we extract several common indicators. Accuracy, precision, recall, F1-score, Matthews correlation coefficient, balanced accuracy, and the area under the ROC curve are some examples of these measurements. To evaluate the computing efficiency of each AI model, we also provide the training and testing run times.

XAI Global Feature Selection: It is essential to remember that the AI models we have developed and assessed for our framework fall under the category of “black-box” models. Therefore, it becomes imperative to explain these models together with their related characteristics (traffic log data) and labels (attack kinds). Consequently, using XAI approaches for feature selection is the next step in our architecture. For every characteristic, we produce global significance values. This is accomplished by creating a variety of graphs that let us infer how each attribute affects the AI model’s decision-making process. These realizations help to improve comprehension and expectations regarding the behavior of the AI model in IDS. We use SHAP global graphs, which are based on the theory of Shapley values lundberg2017unified , for this purpose. The Shapley value for a given feature is estimated using coalitions, which are graphs that show the average marginal contribution of each feature value over all conceivable feature combinations.

After outlining the primary elements of our system, we go into further depth about the suggested feature selection techniques and the baseline techniques we employed in our current study.

Starting with an XAI methodology termed SHAP lundberg2017unified , we first describe our suggested feature selection techniques.

Model Specific Features: The global rankings of features for the examined datasets were acquired using SHAP. For every AI model, the most important features are taken from this rank. Since we employ seven distinct AI models, we extract the top-ranked attributes for each model that influence the algorithm’s choice of which incursion class to detect.

Common Features by Overall Rank: By ranking each feature independently of other models, this technique produces a single feature rank that applies to all models. To do this, the average rank of every single characteristic across all AI models is determined.

Common Features by Overall Weighted Rank: This approach expands on the earlier one. The distinction is that, as opposed to sequential numbering, it considers the correctness of each AI model and the SHAP values for every feature. The product of the SHAP value and the accuracy of the AI model for that feature determines the feature’s relevance. Next, a weighted sum that takes into account the accuracy of the AI model as well as the SHAP relevance of each feature is averaged to rank the features.

Common Features by Overall Weighted Normalized Rank: With one exception, this approach is identical to the previous one. Every SHAP value is normalized. It was discovered throughout the trials that some models, such as LightGBM, provide SHAP values that are significantly larger than those of other models. To prevent this kind of bias, the normalizing step was included.

Models + Attacks Ranking Score: This method extracts significant intrusion features via selecting the top-$k$ ranked features across all different AI models and all different intrusion types. Suppose the set of AI intrusion detection models is denoted by $\mathcal{M}$ in which each entry $m\in\mathcal{M}$ represents one black-box AI model and that the set of intrusion types be given by $\mathcal{A}$ in which $a\in\mathcal{A}$ represents one intrusion class. We calculate the overall ranking score of each feature (given by $r_{i}$) as follows. $r_{i}=\frac{1}{2}\left(\frac{\sum_{m\in\mathcal{M}}r_{im}}{|\mathcal{M}|}+\frac{\sum_{a\in\mathcal{A}}r_{ia}}{|\mathcal{A}|}\right),$ where $r_{im}$ and $r_{ia}$ are the ranks of feature $i$ for model $m\in\mathcal{M}$ and intrusion $a\in\mathcal{A}$, respectively. The overall ranking score of a feature $i$ ($r_{i}$) is given by the weighted sum of both the feature rank across all AI models and across all intrusion types. We then chose the $k$ features with the lowest rank value. Note that the lower the $r_{i}$ value, the higher the feature rank.

Combined Selection: Using this approach, we assign a weight to each feature based on how frequently it appears in the top-$k$ features across all feature selection techniques that have been suggested. Put differently, the choice of a characteristic in this case is determined by its overall significance when compared to all other suggested approaches.

Next, we provide popular feature selection techniques that serve as benchmarks for our suggested techniques in this study. Most of these techniques do not require prior model training.

Chi-square gajawada2019chi : This is a well-known statistical technique that considers two occurrences and calculates their independence. An attribute is more important when its chi-square value is higher.

Feature Correlation feat_corr_citation : The correlation between each pair of attributes is measured using this approach. When two traits are closely connected, only one of them may take their position.

Feature Importance dutta2018analysing : To assess the significance of features on an artificial classification problem, this approach uses a forest of trees to determine the features’ value for a dataset.

Information Gain CICIDS : This approach evaluates a dataset’s properties according to information theory’s notion of information gain. The entropy of the target variable is measured both before and after the dataset is divided based on a certain characteristic to determine the information gain.

K-best brownlee2019choose : The top-K most informative features from a dataset are to be chosen using this feature selection technique. It is a simple method that chooses the $K$ characteristics with the highest scores after ranking the features according to a given criterion.

Next, we provide the primary feature selection techniques that we employed in our study using Xplique xplique , a recent XAI tool. It can be described as a Python feature selection toolkit package that focuses on explaining complex TensorFlow-based neural network models. Users can choose from a variety of XAI methods in this library to test various approaches or assess how well different methodologies are explained.

• •

  Saliency saliency : Saliency maps draw attention to the key areas in an input image or set of data that affect how the model predicts a certain class. The back-propagation approach is used internally by these maps, and it focuses on determining the class derivatives while taking the input data into account. They aid in visualizing the input components that have an impact on the AI model’s conclusion. A saliency map representing the pixels associated with the item or class in the picture or input data, respectively, is the end product.

• •

  Integrated Gradients IG : One way to link the model’s prediction to its input attributes is to use integrated gradients. The process computes the gradients between the input and the baseline. Afterward, the attributions are obtained by integrating such gradients along this path. It is regarded as one of the deep neural network techniques with the highest degree of reliability IG .

• •

  Occlusion DeconvNet-Occlusion- : The Occlusion approach is concealing or deleting portions of the input in a methodical manner and then monitoring the effect on the model’s prediction. The procedure checks to see if the model bases its prediction on the context—that is, the data around the item. It accomplishes this by methodically applying a gray square or pre-set value to the relevant area of the input data and tracking the correctness. Therefore, it is helpful to know which areas are essential to the model’s conclusion.

• •

  SmoothGrad smoothgrad : Deep neural networks can use the SmoothGrad approach to lower noise in their sensitivity maps. To create a smoother representation, it entails creating many sounds, adding them to the relevant pictures, calculating the saliency map for each, and then averaging them. This procedure enhances the coherence and sharpness of sensitivity maps smoothgrad while lowering visual noise.

• •

  VarGrad vargrad : Similar to SmoothGrad, VarGrad adds a variation term that is used to produce explanations. The explanation variance that results from introducing noise to the input is called VarGrad. VarGrad captures higher-order partial derivatives, which sets it apart from SmoothGrad vargrad .

• •

  SquareGrad squaregrad : An expansion of SmoothGrad, SquareGrad incorporates squared Gaussian gradients into the noise-adding averaging procedure. The primary distinction between the original SmoothGrad approach and SquareGrad is that the former squares each estimate before averaging it. The feature magnitude may matter more than the direction squaregrad .

• •

  DeconvNet DeconvNet-Occlusion- : A method for displaying characteristics that convolutional neural networks (CNNs) learn is called DeconvNet, also known as Deconvolutional Network. It maps the activations back to the input space using a technique akin to deconvolution. Rebuilding feature activations using processes including unpooling, rectification, and filtering allows DeconvNet to recreate the activation of the ConvNet layer. The DeconvNet-obtained visualization sheds light on the roles played by intermediate feature layers and the AI classifier DeconvNet-Occlusion- .

• •

  GradientInput  GradientInput : By multiplying the input to a neuron by the output gradient regarding that neuron, this approach determines the gradient input. In this way, it takes into account both the sign and intensity of the input and gives information about how relevant the input characteristics are to a neural network. It is an effective tool for examining network activity because of its level of detail, which highlights important characteristics and their effects on the network.

• •

  LIME lime : Through the process of introducing perturbations around the sample of interest and abstracting its behavior, an approach known as Local Interpretable Model-Agnostic Explanations (LIME) with a local scope builds a surrogate model around the region of interest. Because of this, such a model is an easier-to-understand version of the original model while maintaining the same behavior in the relevant domain. This has the benefit of being more explainable than the original model.

Drawing on Xplique’s several feature selection techniques, we provide a straightforward voting procedure that applies to each of these approaches, as elaborated below.

• •

  Voting: Our suggested voting approach is based on a frequency analysis of all the earlier feature selection techniques that Xplique was used to assess. The method uses a point system where the least significant feature is assigned a score of one, and each subsequent characteristic’s value is increased by one until it reaches the highest point. Thus, out of 10 attributes, the least significant is assigned a score of one, while the most significant is assigned a value of ten. Because there are nine Xplique approaches, we add up all the points that each feature has received. Next, we arrange the features in descending order to determine which characteristics are the most important using this straightforward voting method.

Our framework’s last part entails taking specific measurements out of the overall feature descriptions. In particular, we concentrate on extracting characteristics that are distinct to the incursion and traits that are specific to the model. The most crucial elements of every AI model are referred to as model-specific characteristics. The relevance of these traits in impacting the model’s decision-making process is what led to their identification. We can learn more about the precise traits that influence the AI model’s predictions by obtaining model-specific attributes. Conversely, intrusion-specific traits are those that are most closely linked to a given kind or class of network intrusion. These characteristics are essential to comprehending what makes the various forms of intrusions unique. We are able to determine the essential characteristics that aid in distinguishing and identifying particular kinds of network intrusions by extracting information specific to intrusions.

Importance of Features for Different Attacks: The creation of a list of the best characteristics for every kind of assault is a significant result of our system. Security analysts may use this information to narrow down certain log data aspects that are indicative of various attack types, which is useful when they are looking into suspicious network traffic. This feature importance analysis may also help modify current intrusion detection systems (IDS) by offering user-friendly dashboards that are customized to the unique attack patterns seen in an organization’s historical data. Remember that the seven major attack types in the two datasets we analyzed in our work—normal traffic, DoS, PortScanning, Brute Force, Infiltration, Web assault, and Bot—are the subject of our present analysis. Using two datasets, CICIDS-2017 and RoEduNet-SIMARGL2021, we want to have a better knowledge of the feature importance for each sort of assault. It is important to note that the CICIDS-2017 dataset includes information on all attack types, but the RoEduNet-SIMARGL2021 dataset only includes information on DoS and Port Scan attacks combined with regular traffic. The distribution of samples across various attack labels and the number of samples available in each dataset are shown in Table 3.

Comprehensive List of Top Network Intrusion Features: We provide the full list of important network intrusion features for the two datasets this study examines, together with the relevant explanations, to help readers better comprehend these aspects. The rest of the article will make frequent references to these feature lists. In particular, a thorough explanation of every feature in the RoEduNet-SIMARGL2021 network intrusion dataset can be found in Table 1. In a similar vein, each feature in the CICIDS-2017 network intrusion dataset is described in Table 2.

RoEduNet-SIMARGL2021 mihailescu2021proposition : This dataset comes from the SIMARGL project, which was a joint venture between the Romanian Education Network (RoEduNet) and the European Union, funded under the Horizon program. It consists of real-time traffic analysis features together with legitimate network traffic data. The information is organized according to a data schema that is similar to Netflow claise2004cisco , a network protocol that CISCO created to record and observe network flows.

CICIDS-2017 panigrahi2018detailed : Created in 2017 by the University of Brunswick’s Canadian Institute for Cybersecurity, this dataset acts as a standard for intrusion detection. It includes six different types of attack profiles, including brute force, heartbleed, botnet, denial of service, portscan, web assault, and infiltration attack. The dataset includes background traffic created by a B-Profile system sharafaldin2018towards , which extracts different user behaviors depending on network protocols, to provide a realistic environment.

Dataset Summary and statistics: Table 3 displays the number of attack kinds (labels), the distribution (%) for each attack type, and the size of each dataset (samples amount).

Coding Resources: For the construction and analysis of our various black-box AI techniques, we employed several open-source toolboxes (including Keras, Scikit-Learn, Pandas, and Matplotlib).

XAI Methods: The following XAI tools were employed in our assessment:

(a) SHAP lundberg2017unified : We employed the widely utilized Shapley Additive exPlanations (SHAP) XAI toolkit lundberg2017unified . For our AI models, we generated global features and associated techniques using that SHAP tool.

(b) Xplique xplique : For feature selection, we additionally utilized the Python toolkit Xplique library xplique , emphasizing explainability and comprehension of complex neural network models. This collection of techniques includes descriptions, examples, and links to official publications for a wide range of approaches (e.g., Saliency, Occlusion, Integrated-Gradients, SmoothGrad, VarGrad, SquareGrad, GuidedBackprop, DeconvNet, among others). It enables the user to assess model explanations based just on TensorFlow or try other approaches.

AI Methods: We assess black-box AI methods and various components of our proposed XAI-based feature selection framework for a better understanding of network intrusion detection tasks by pairing seven popular AI classification algorithms: deep neural network (DNN) tang2020saae , random forest (RF) waskle2020intrusion , AdaBoost (ADA) yulianto2019improving , k-nearest neighbor (KNN) li2014new , support vector machine (SVM) tao2018improved , multi-layer perceptron (MLP) MEBAWONDU2020e00497 , and light gradient-boosting machine (LightGBM) jin2020swiftids .

Hyper-parameters: We list the primary hyper-parameter selections in A for each of the many AI models that were employed in this study.

Black-box AI Metrics: We use a commonly used set of criteria for assessing intrusion detection and classification issues to evaluate the performance of the black-box AI models. The measures that fall under this category include balanced accuracy (Bacc), Matthews correlation coefficient (MCC), accuracy (Acc), precision (Prec), recall (Rec), and F1-score (F1). The effectiveness of the AI model is also determined by calculating the AucRoc (area under the ROC curve) score.

Selecting Features for XAI Techniques Evaluation: We produce a variety of feature selection techniques for evaluating XAI approaches to offer performance insights. Global summary charts, which provide a summary of feature rank throughout the dataset, are one of these techniques. Furthermore, we ascertain the significance of each feature for every form of assault, pinpointing the primary attributes that serve as indicators of the existence of particular attack kinds. Additionally, we examine the overall feature rank for each dataset, emphasizing the top characteristics that are shared by several AI models. These techniques offer a thorough assessment of the XAI-based feature selection techniques put forward in this study.

Superiority of our Feature Selection Methods: The comparison results between all feature selection methods (i.e., baseline methods and our proposed feature selection methods) are displayed in Table 4. All performance metrics (Acc, Prec, F1, MCC, Bacc, and AucRoc) for the CICIDS-2017 and RoEduNet-SIMARGL2021 datasets are analyzed. It displays which feature methods—here, 5 and 10—performed better overall for the various values of the top characteristics. The way we calculate scores is as follows. The best technique receives three points, the second best receives two, and the third best receives one. Which feature selection techniques work best overall are determined by adding together these scores, or weights. The most accurate techniques are the best ones; in the event of a tie in accuracy, we use the next metric (such as precision, recall, etc.). Our feature selection methods outperform baseline methods, as shown in Table 4, which has the best overall score in 11 out of 12 based on the previously mentioned ranking, which is based on the best three methods for each of the four setups that are taken into consideration ($k=5$ for CICIDS-2017, $k=10$ for CICIDS-2017, $k=5$ for RoEduNet-SIMARGL2021, and $k=10$ for RoEduNet-SIMARGL2021).

Overall Black-box AI Model Performance: The overall effectiveness of our various black-box AI models for IDS is then displayed utilizing various feature selections (all features, top-15, top-10, and top-5). For the RoEduNet-SIMARGL2021 and CICIDS-2017 datasets, respectively, these performances are displayed in Tables 5-6. They display the many metrics that we gathered for various AI models. We provide the top results for several AI models in bold language. The RF, KNN, and SVM models consistently exhibit the greatest performance over most settings in both datasets. Conversely, MLP performs the poorest overall across various trial outcomes for both datasets. Lastly, for both datasets, LightGBM and ADA perform moderately.

Impact of Feature Selection on Black-box AI Models: We next go over the thorough performance analysis we conducted on our various black-box AI models for IDS using the four feature selection setups discussed earlier. In particular, the RoEduNet-SIMARGL2021 dataset performance metrics are presented in Table 5. We find that after being trained on all characteristics, five of the AI models—ADA, SVM, KNN, RF, and DNN—perform worse. Furthermore, we note that the MLP and LightGBM models perform identically when given varying feature sets; that is, the feature selection does not affect these models’ results.

Table 6 for the CICIDS-2017 dataset demonstrates that three AI models—ADA, SVM, and KNN—perform better when feature selection is used during training. However, the DNN and MLP do marginally worse when it comes to feature selection. This experiment demonstrates how crucial it is to carefully choose characteristics to improve AI model performance. It also demonstrates how the AI model and the dataset’s properties affect this kind of feature selection (for example, DNN and MLP models require all attributes to perform better in our configuration).

It is crucial to emphasize that these broad performance measures do not offer in-depth information on how well an AI model functions in various attack kinds, which we cover next.

Variation of Best AI Model across Attack Types: The accuracy of each AI model for distinct attack types in both datasets is shown in Tables 7-8. Tables show that most AI models do not always perform better than other models in all attack classes in the datasets. This underlines how crucial it is to comprehend each AI model’s decision-making process for various attack scenarios, which is the main focus of our ongoing research.

Enhancement of Attack Detection under Feature Selection: The quantification of AI model enhancements in detecting assaults (provided by many AI models with optimal performance) under feature selection is presented in Table 9. For instance, we have two models that performed better with $k=5$ and five models that performed better with $k=15$ for CICIDS-2017 in the ‘Brute Force’ assault. We have two models that perform better when $k=5$, three models that perform better when $k=10$, one model that performs better when $k=15$, and one model that performs better when $k=all$ for RoEduNet-SIMARGL2021 in the ‘Port Scan’ attack.

False Positive Rates: The analysis of the false positives for the CICIDS-2017 and RoEduNet-SIMARGL2021 datasets, Table 10, clearly shows differences in the performances of the different methods. The DNN approach has a notable false positive rate of 34.00% in the RoEduNet-SIMARGL2021 dataset, indicating a large room for improvement. It is noteworthy to observe that, with a much reduced false-positive rate of 2.46%, our technique performs noticeably better in the CICIDS-2017 dataset. In the CICIDS-2017 dataset, the LightGbm technique shows a false positive rate of 2.50%; in comparison, the RoEduNet-SIMARGL2021 dataset records an incredibly low false-positive rate of just 0.0033%. Moreover, there is a discernible performance gap between the two datasets for the ADA algorithm, suggesting that different datasets require different AI techniques for intrusion detection.

The importance of Features via Explainable AI: We now demonstrate the significance of the top features that influence each AI model’s choice for the two network incursion datasets.

Global Summary Plot: First, we show the global summary charts for several AI models, which highlight the key factors that influence the algorithms’ decisions. These plots give important insights into the decision hierarchy by showing the elements in declining order of relevance. For the RoEduNet-SIMARGL2021 and CICIDS-2017 datasets, respectively, Figures 2 and 3 illustrate the feature significance across various AI models. By averaging the mean of each Shapley value, the feature significance values are obtained. Notably, by assigning different colors to each assault label, this representation also emphasizes the importance of certain attributes for each sort of attack (or intrusion).

Feature Importance based on XAI: The next stage is to extract the most important features from all intrusion detection AI models for both network intrusion datasets after analyzing the feature importance of each model. The top eight features for each dataset are shown in Table 11. For instance, TCP_WIN_SCALE_IN is a prominent characteristic in the RoEduNet-SIMAR GL2021 dataset that affects all AI models’ judgments. Similar to this, every AI model in the CICIDS-2017 dataset relies heavily on the Destination Port attribute to make judgments. When the precise attack type is unclear, security analysts can monitor these features in network traffic thanks to this overall feature rating.

Outlining RoEduNet-SIMARGL2021’s Principal Features: Understanding the nature of the “TCP_WIN_SCALE_IN” characteristic and its possible influence on network intrusion detection tasks is crucial when evaluating its significance in AI models. The inbound TCP Window Scale, a parameter in the Transmission Control Protocol (TCP) for internet communication, is referred to as the “TCP_WIN_SCALE_IN” feature. This option allows the window size that controls the data flow to be adjusted. As a result, by increasing the TCP window size, an attacker can use this parameter to attack the network and regulate data movement.

Explaining CICIDS-2017’s Top Features: The port number on the destination device or server in a network connection is referred to as the “Destination Port” in the context of intrusion detection systems and network security. Remember that ports are only numbers that are used to assist in identifying which application or service on a device is receiving network data. Therefore, the port that a packet is meant to be received at is indicated by the ‘Destination Port’. Monitoring and examining destination port addresses in the context of IDS might be essential for identifying specific kinds of network assaults or unwanted activity.

Feature Importance for Each Attack Type: The intrusion-specific feature importance, which lists the top five attributes for every kind of assault, is next shown. It should be noted that by going back to the Figures 2 and  3, the attack-specific characteristics are produced. Next, we use frequency analysis to examine each assault (intrusion) separately and extract the most important attributes of each attack. This data is shown for the RoEduNet-SIMARGL2021 dataset in Table 12, and for the CICIDS-2017 dataset in Table 13. Interestingly, we find that some common aspects across the two datasets—such as Destination Port and Init_Win_Bytes_Bwd for CICIDS-2017, and TCP_WIN_SCALE_IN and FLOW_DURATION_MS for RoEduNet-SIMARGL2021—are shared across various attack types. These results point out the most important factors that analysts have to consider while looking into certain network invasions. Furthermore, by understanding the key indicators (features) of these attacks in the network flow, this knowledge can help develop attack-specific solutions for intrusion detection systems (IDS) and adapt already-existing AI models for IDS based on these identified features for more accurate detection of specific network attacks.

Our approach to feature extraction is compared to five baseline methods: information gain CICIDS , feature importance dutta2018analysing , feature correlation feat_corr_citation , K-best SENSOR , and Chi-square gajawada2019chi . The entire list of the top features retrieved by our suggested approaches is shown along with these various baselines in Table 15. The comparison shows that our methodologies and the baselines for both datasets do not agree on the feature significance rankings. It is important to remember, nevertheless, that several feature extraction techniques have discovered similar top incursion characteristics.

Effect of Different Top Features: We now juxtapose the AI models’ performance under our framework’s top five significant characteristics with those chosen by information gain CICIDS and K-best SENSOR . Table 14 demonstrates that, for the two datasets, our framework performs better in 22 out of 28 AI models (bold text indicates superior performance). Additionally, we present the optimal feature selection strategies for various configurations of the CICIDS-2017 and RoEduNet-SIMARGL202 datasets. Table 16 demonstrates that, when compared to baselines, our suggested techniques perform better at choosing the best features for various AI models.

Xplique Experiment: For this experiment, we do feature selection using nine alternative XAI approaches for the DNN model, utilizing the Xplique toolbox. Table 20 (B) displays the experiment’s findings. In Table 20, the ’Voting’ column aggregates all nine Xplique methods to produce an overall feature ranking result for feature selection, similarly to Table 15 ‘Combined Selection’ column does. Subsequently, we observed that ‘Combined Selection’ and ‘Voting’ selected six features in common out of ten for the CICIDS-2017 dataset and eight features in common out of ten for the RoEduNet-SIMARGL2021 dataset. These results were obtained by comparing both approaches from Table 15 and Table 20. This shows some convergence when taking into account both approaches. However, since all of Xplique’s XAI techniques attempt to explain such design, Xplique is limited to only taking neural network-based AI models (DNNs in this case). However, as our framework demonstrates, our approaches are independent of any particular model, and every AI model that is taken into consideration for this study has been tested using our suggested selection techniques.

Lastly, we display the computational effectiveness of the various AI models that our system makes use of. We also demonstrate how runtime metrics are impacted by the quantity of features. Now, we get into the setup and key results for the runtime analysis.

Computing Resources: A renowned public university’s high-performance computer (HPC) was used for the tests. Four NVIDIA A100 GPUs, 64 GPU-accelerated nodes with 256 GB of RAM apiece, and a single 64-core AMD EPYC 7713 CPU with a clock speed of 2.0 GHz and 225 watts are all included in the HPC system. Maximum performance with this arrangement is about 7 petaFLOPs. This supercomputer is meant to support academics in carrying out complex AI and machine learning tasks at a high level.

(a) AI Model Training Runtime: The training runtimes of the AI models on the CICIDS-2017 dataset are shown in Table 17. As can be shown, most of the AI models in our study simply require a few minutes to train these models on millions of data. The chart also shows that using all characteristics for training, as opposed to just the top 15, leads to longer training periods, with an average increase of 2.1X for all AI models. For this dataset, the individual models with the quickest training runtimes are KNN and RF. In contrast, out of all the AI models under consideration, LightGBM has the slowest training duration. Table 17 shows that SVM has the slowest training runtime while RF and MLP have the fastest training runtimes for the RoEduNet-SIMARGL2021 dataset.

(b) AI Classification Runtime: The prediction runtimes of the AI models are displayed in Table 17. The traffic test data from the CICIDS-2017 dataset shows that most AI models have quick prediction speeds, finishing the task in less than two minutes. RF has the quickest prediction runtime of all the AI models for all feature combinations. Table 17 compares the two datasets and shows that the RoEduNet-SIMARGL2021 dataset has a longer classification runtime than the CICIDS-2017 dataset. The RoEduNet-SIMARGL2021 dataset is around 15 times bigger than the CICIDS-2017 dataset, which is the cause of this discrepancy. Furthermore, Table 17 illustrates that, despite its effective training, KNN has the slowest prediction runtime, whereas RF exhibits the quickest prediction runtime. It is significant to remember that these timeframes might change based on the computer power used.

(c) SHAP Feature Importance Runtime: We also present the efficiency (duration in hours) of SHAP explanation generation for various AI models and sample counts. The Appendix’s Table 18 illustrates the time required to generate the importance of the SHAP feature. In SHAP, the global feature significance is generated quickly by the RF, DNN, and LightGBM models, regardless of the quantity of data used for the explanation. For instance, generating feature significance using SHAP with 10,000 samples takes 18.36 minutes with DNN, 2.76 minutes with RF, and 0.36 minutes with LightGBM. However, ADA and KNN models take many hours to run and are not scalable for a higher number of samples. All things considered, SHAP outperforms the other models in terms of time performance when using RF, DNN, and LightGBM.