Machine Learning for Detecting Data Exfiltration: A Review

According to several studies [1, 13, 14, 15], Data Exfiltration Life Cycle (DELC), also known as cyber kill-chain [16], has seven common stages. The first stage is Reconnassance; an attacker collects information about a target individual or organization system to identify the vulnerabilities in their system. After that, an attacker pairs the remote access malware or construct a phishing email in the Weaponization stage. Rest of the stages use various targeted attack vectors [1, 13, 17] to exfiltrate data from a victim. Fig 1 shows these stages and their mapping with attack vectors. The weapon created in the Weaponization stage such as an email or USB, is then delivered to a victim in the Delivery stage. In this stage, an attacker uses different attack vectors to steal information from a target. These include: Phishing [18], Cross Site Scripting (XSS) and SQL Injection attacks. In Phishing, an attacker uses social engineering techniques to convince a target that the weapon (email or social media message) comes from an official trusted source and fool naive users to give away their sensitive information or download malware to their systems. In XSS, an attacker injects malicious code in the trusted source which is executed on a user browser to steal information. SQL Injection is an attack in which an attacker injects SQL statements in an input string to steal information from the backend filesystem such as database. Delivery stage can result in either stealing the user information directly or downloading malicious software on a victims system. In Exploitation stage, the downloaded malware Root Access Toolkit (RAT) [19] is executed. The malware gains the root access and establishes a Command and Control Channel (C&C) with an attacker server. Through C&C, an attacker can control and exfiltrate data from a victim system remotely. C&C channels use Malicious domains [20] to connect to an attacker server. Different Overt channels [21, 13] such as HTTP post or Peer-to-Peer communication are used to steal data from a victim in C&C stage. Exploration stage often follows C&C stage in which an attacker uses remote access to perform lateral movement [22] and privilege escalation [23] attacks that aid an attacker to move deeper into the target organization cyberinfrastructure in search of sensitive and confidential information. The last stage is Concealment; in this stage, an attacker uses hidden channels to exfiltrate data and avoid detection. The attack vectors Data Tunneling, Side Channel, Timing Channel and Stenography represent this stage. Data Tunneling attacks use unconventional (that are not developed for sending information) channels such as Domain Name Server (DNS) to steal information from a victim, while in Side Channel [24], and Timing channel [25] an attacker gains information from physical parameters of a system such as cache, memory or CPU and time for executing a program to steal information from a victim respectively. Lastly, in Stenography attack, an attacker hides a victim’s data in images, audio or video to avoid detection. Two attack vectors: Insider attack and Advanced Persistent Attack (APT) uses multiple stages of DELC to exfiltrate data. In Insider attack, an insider (company employee or partner) leaks the sensitive information for personal gains. APT attacks are sophisticated attacks in which an attacker persists for a long time within a victim system and uses all the stages of DELC to exfiltrate data.

Fig 2 shows the Machine Learning Life Cycle. Each stage is briefly discussed below (refer to [26] for details). Data collection: In this phase, a dataset composed of historical data is acquired from different sources such as simulation tools, data provider companies, and the Internet to solve a problem (such as classifying malicious versus legitimate software). Data cleaning [20] is performed after data collection to ensure the quality of data and to remove noise and outliers. Data cleaning is also called pre-processing. Feature Engineering: This phase purports to extract useful patterns from data in order to segregate different classes (such as malicious versus legitimate) by using manual or automated approaches [27]. In Feature selection, various techniques (e.g., information gain, Chisquare test [28]) are applied to choose the best discriminant features from an initial set of features. Modelling: This phase has four main elements. Modelling technique: There are two main types of modelling techniques: anomaly detection and classification [29]; Each modelling technique uses various classifiers (learning algorithms) that learns a threshold or decision boundary to discriminate two or more than two classes. Anomaly-based detection learns the normal distribution of data and detects the deviation as an anomaly or an attack. The common anomaly classifiers include: K-mean [30] (Kmean), One Class SVM [31] (OCSVM), Gaussian Mixture Model [32] (GMM) and Hidden Markov Model [33] (HMM). Classification algorithms learns data distribution of both malicious and benign to model a decision boundary that segregates two or more than two classes. ML classifiers can be categorized into two main types: traditional ML classifiers and deep learning classifiers.The common Traditional ML classifiers include: Support Vector Machines [34] (SVM), Naïve Bayes [35] (NB), K-Nearest Neighbor [36] (KNN), Decision Trees [37] (DT) and Logistic Regression [38] (LR). Most popular traditional ensemble classifiers (combination of two or more classifiers) include Random Forest Trees [39] (RFT), Gradient Boosting (GB) and Ada-boost [40] classifiers. The Deep Learning (DL) classifiers include Neural Network [41] (NN), NN with Multi-layered Perception [42] (MLP), Convolutional NN [43] (CNN), Recurrent NN [44] (RNN), Gated RNN [45] (GRU) and Long Short Term Memory [46] (LSTM). In the rest of this paper, we will use these acronyms instead of their full names. Learning type [26] represents how a classifier learns from data. It has four main types: supervised, unsupervised, semi-supervised and re-enforcement learning [47]. The training phase is responsible for building an optimum model (the model resulting in best performance over the validation set) while a validation method is a technique used to divide data into training and validation (unseen testing data) set [48]; and Performance Metrics [26] evaluate how well a model performs. Various measures are employed to evaluate the performance of classifiers. The most popular metrics are Accuracy, Recall, True Positive Rate (TPR) or Sensitivity, False Positive Rate (FPR), True Negative Rate (TNR) or Specificity, False Negative Rate (FNR), Precision, F1-Score, Error-Rate, Area Under the Curve (AUC), Receiver operating characteristic Time to Train, Prediction Time and confusion matrix (please refer to [26, 49] for details of each metric).

Compared to previous reviews, we focus on ML-based data exfiltration countermeasures. The bibliographic details of reviewed papers are given in Appendix LABEL:appendix. Our review included papers, most of which have not been reviewed in previous reviews. The summary of the comparison against previous works is shown in Table I.

All of the previous reviews focus on identifying the challenges (e.g., encryption and human factors) in detecting and preventing data exfiltration. Unlike previous reviews, our review focuses only on data exfiltration countermeasures. Some of the existing reviews [1, 3, 4], highlight data exfiltration countermeasures such as data identification and behavioral analysis. Unlike the previous reviews, our review mainly focuses on ML-based data exfiltration countermeasures.

Our review included papers most of which have not been reviewed by the previous reviews; for example, there were only 3, 4, 1, and 6 papers that have been previously reviewed in [3], [4],[53] and [1] respectively Unlike the previous reviews, we used the Systematic Literature Review (SLR) method guidelines. There are approximately 61 papers published after the latest review (i.e., [1]) on data exfiltration.

Our literature review covers various aspects of ML-based data exfiltration countermeasures as compared to the previous literature reviews that included a variety of topics such as the data states, challenges, and attack vectors. Whilst all of the previous reviews have identified the areas for future research, e.g., accidental data leakage and fingerprinting, the future research areas presented in our paper are different from others. Similar to our paper, one review [1] has also identified response time as being an important quality measure for data exfiltration countermeasures. However, unlike [1], our paper advocates the need for reporting assessment measures (e.g., training and prediction time) for data exfiltration countermeasures

In comparison to previous reviews, our paper makes the following unique contributions:

• •

  Presents an in-depth and critical analysis of 92 ML-based data exfiltration countermeasures.

• •

  Provides rigorous analyses of the feature engineering techniques used in ML-based countermeasures.

• •

  Reports and discusses the datasets used for evaluating ML-based data exfiltration countermeasures.

• •

  Highlights the evaluation metrics used for assessing ML-based data exfiltration countermeasures.

• •

  Identifies several distinct research areas for future research in ML-based data exfiltration countermeasures.

The findings of our review are beneficial for both researchers and practitioners. For researchers, our review has identified several research gaps for future research. The identification of the most widely used evaluation datasets and performance metrics along with their relative strengths and weaknesses, serve as a guide for researchers to evaluate the ML-based data exfiltration countermeasures. The taxonomies of ML approaches and feature engineering techniques can be used for positioning the existing and future research in this area. The practitioners can use the findings to select suitable ML approaches and feature engineering techniques according to their requirements. The quantitative analysis of ML approaches, feature engineering techniques, evaluation datasets, and performance metrics guide practitioners on what are the most common industry-relevant practices.

Since machine learning is a data-driven approach [64] and the success of ML-based systems is highly dependent on the type of data analysis performed to extract useful features. We classify the countermeasures presented in the reviewed studies into two main types based on data type and data analysis technique: Data-driven and Behavior-driven approaches.

Table VI shows the mapping of the proposed taxonomy with respect to DELC. It can be seen that 48% of the reviewed papers report data-driven approaches. These approaches are most frequently (i.e., 30/48 studies) used in detecting the Delivery stage of DELC. In this stage, they are primarily (i.e., 12/32 and 5/5 studies respectively) employed in detecting Phishing and SQL injection attacks. Furthermore, these approaches are also applied in detecting Concealment stage with 9/21 studies classified under this category. However, they are rarely (i.e., 5/44) used in detecting other stages of DELC.

Among data-driven approaches, Direct inspection approach is only applied to detect the Delivery stage in the reviewed studies, in particularly Phishing (12/24) attacks. One rationale behind it can be the presence of common guidelines identified by domain experts that differentiate normal website from phishing ones [18]. For example, in [S5, S37] examined the set of URLs and identified the presence of “@” symbol as one of the feature with an argument that legitimate URLs don’t use “@” symbol. Subsequently, Context Inspection approach is applied by 17/48 data-driven studies to detect Phishing (9/24), SQL Injection (3/5), Malicious Domain (3/5) and more recently in Data tunneling. For example, the authors in [S67] used bytes to represent the DNS packets and trained CNN using the sequence of bytes as input. This representation captured the full structural and sequential information of the DNS packets to detect DNS tunnels. Distribution Inspection on the other hand is commonly used to detect the Concealment stage (7/21) specifically Steganography (3/3) and Data tunnelling (3/12) attacks. The reason behind it is that both of these attacks use a different type of data encoding or wrapping to evade detection. The normal and encoded data have different probability distributions that are imperceptible in direct or context inspection [65]. For instance, [S20] used inter and intra-block correlation among Discrete Cosine Transform (DCT) of the JPEG coefficient of the image to detect Steganography.

52% of the primary studies are based on Behavior-driven approaches. In contrast to data-driven approaches, these approaches are recurrently used in detecting all the other stages of DELC except Delivery. This suggests that once the data exfiltration attack vector is delivered the analysis of the behavior exhibited by it is more significant than its content analysis. An interesting observation is that these approaches are capable to detect multi-stage and sophisticated attacks like APT (5/5) and Insider threat (10/11). One reason behind it is that they comprehend the behavior of a system in a particular context to create a logical inference that can be extended to manage complex scenarios, e.g., how a user interacts with a system or how many resources are utilized by a program? Among behavior-driven approaches, Event-based approach is most evident in detecting RAT (4/7) and insider threat (8/11) detection. This is because both of these attacks execute critical system events such as plugging a USB device or acquiring root access. The time or sequence of the event execution is leveraged by this approach to detect data exfiltration. For example, [S7] analyses the “logon” event based on working days, working hours, non-working hours, non-working days to detect Insider attack.

Flow-based Approach is notable in detecting data tunnelling (7/12) and Overt channels (6/10) attacks. The reason behind it is both of these attacks are used for exfiltrating data through the network. The only difference is the nature of the channel used. Overt channels transfer data using the conventional network protocols such as Peer to Peer or HTTP post while Data tunneling channels use channels that are not designed for data transfer such as DNS. Additionally, in contrast to Distribution inspection which is also used to detect these attacks, this approach manually monitors the relationship between the inbound and outbound flow. For example, [S43] used the ratio of the size of outbound packets with inbound packets to detect Overt channels attack. 6/48 behavior-driven studies utilize Hybrid-based approach. This approach is used in detecting APT and Overt Channels. It captures both the system and network behavior to detect data exfiltration. For example, [S30] detected an APT attack by monitoring both the system and network behavior using the frequency of system calls and the frequency of IP addresses in a session. The authors claimed that using the hybrid approach can be used to prevent long term data exfiltration. Propagation based approach is an interesting countermeasure because it examines multiple stages of DELC across multi-host to detect data exfiltration. For instance, [S2] identifies multiple stages of DELC to detect the APT attack. First, the approach detects the exploitation stage by using a threat detection module, then it uses malicious SSL certificate, malicious IP address, and domain flux detection to discover the Command and Control (C&C) communication. After that, it monitors the inbound and outbound traffic to disclose lateral movement, and finally it uses scanning and Tor connection [66] detection to notice the asset under threat and data exfiltration respectively. Lastly, Resource usage-based approach is used by side-channel (3/3) and Insider (1/1) attack detection. A reason behind it is that for Side-channel attacks, we only found studies that were detecting cache-based side-channel attacks. In this type of attack, an attacker exploits the cache usage pattern [67] to exfiltrate data. Hence, this approach monitors the cache usage behavior to detect Side-channel attacks. For example, [S52] used a ratio of cache misses and hit as one of the features to detect Side-channel attack.

This section discusses the type of features and feature engineering methods used by the primary studies. The features used by the studies can be classified into six types: statistical, content-based, behavioral, syntactical, spatial and temporal. Table VII briefly describes these base features and enlist their strengths and limitations. While Table VIII describes the feature engineering methods [27] with their strength and weakness. These features are mined single-handedly or collectively as hybrid features by different studies, as shown in Fig 7(a). Furthermore, Fig 7(b) shows the distribution of these features types with respect to ML countermeasures and feature engineering method.

Feature engineering process extracts features from data. Feature engineering methods are classified as manual, automated, and semi-automated as described in Table 10. 51 studies used manual, and four studies used a fully automated. In comparison, eight studies used semi-automated feature engineering method to extract features. Fig 10 shows the mapping of feature type with feature engineering method. It is evident that the studies based on syntactical features either used an automated or semi-automated feature engineering method. It was predictable as syntactical features are based on text structure and NLP techniques can be utilized to extract them automatically. However, it was unanticipated that one study utilizing behavioural features used this technique. In [S48], a technique named malicious sequential mining is proposed to automatically extract features from Portable Executable (PE) files using instruction set sequence in assembly language. Other studies that used automated feature engineering method used semantic inspection countermeasure but extracted statistical or Content-based features to train the classifiers.

Feature selection is important, but an optional step in feature engineering process [79]. Table IX describes, and provides strengths and weaknesses of five different methods used by our reviewed study to select the discriminant features from the initial set of features.

The feature selection techniques used by the reviewed studies with respect to ML-based countermeasure, feature-engineering technique and the number of the features used are shown in Fig 8. This analysis is helpful to understand the relationship of feature selection techniques with data analysis and feature engineering process. It shows that 52% studies utilized feature selection methods. Among this filter is the most popular (16 studies) method. Almost all the approaches apply it except for hybrid, propagation-based and resource usage-based countermeasures. Additionally, this approach is not only applied to features extracted manually but also applied to automatic and semi-automatically (3 studies each) extracted features. For example, in [S1], the authors used semi-automatically extracted features i.e., Opcodes n-grams (automated)and gray-scale images (manual), and the frequency of Dynamic Link Library (DLL) in import function (manual) to classify known versus unknown malware families. The feature set dimensionality is then reduced and used information gain [87] a filter method to reduce the dimensions of the feature vector. Similarly, [S5] extracts 16 features manually based on frequency analysis assessment criteria to detect phishing websites. Chi-Square [87] feature selection algorithm a filter method is incorporated to select the features.

The second most popular technique is Dimensionality reduction is used by six studies. It is applied by all approaches except for context inspection and hybrid approaches. Five studies employ this technique over manually extracted features while only one study [S61] used it for semi-automatically extracted features. In [S61], two types of features were extracted n-grams from executable file and Window API Calls. Principle Component Analysis (PCA) is used to reduce the n-gram feature dimension, while class-wise document frequency [88] is utilized to select the top API call. Wrapper method has been reported in six studies. It is mostly used (i.e., 5/6) in data-driven countermeasures and is employed by 4 and 2 studies that extract features manually and semi-automatically, respectively. For instance, the study [S60] used two types of features webpage features and website embedded trojan (7) features. For feature selection, two wrapper methods: RFT based Gini Coefficient [89] and stability selection [90] are employed. Another interesting but relatively new feature selection method is Attention [86]. Although, it is not primarily known as a feature selection method because this algorithm is used to select most relevant features while training DNNs, we have treated it as a feature selection method, as also suggested by [86]. Attention is used by three studies, out of which two [S87, S88] used it on automatically extracted features as a part of DNN training whereas [S74] used it on manually extracted features that are fed as input to DNN classifier. In [S74], users action sequence are manually extracted from log files and feed into Attention-based LSTM that help a classifier to pay more or less focus to individual user actions. Lastly, the embedding method is just explored by two flow-based studies [S11, S65]. In [S11], an ensemble anomaly detector of Global Abnormal trees was constructed based on weights of feature computed by information gain (filter method). We believe that one reason for not using the embedding method is the need for fine-grain understanding of how classifier is constructed [91]. Most of the selected studies only apply ML classifiers as a black box.

In Fig 8b, Fig 8c, we analysed the number of features used viz-a-viz the ML countermeasure and types of features selection methods. The y-axis shows the log of the number of features used by the reviewed studies. We have used this because the total number of features used by the studies have high variance (ranging from 2 to 8000) and hence useful visualization was not possible with direct linear dimensions. The circular points show the outliers in the distribution while the solid line shows the quartile range and median based on the log of the number of features used by the studies. Fig 8b analysis is based on 69/92 studies that reported the total number of features used for training the classifier. Among the remaining 23 studies, 16 studies either extracted feature automatically or semi-automatically and 12 studies belong to context inspection ML countermeasure. It may be because these studies mostly rely on Natural Language Processing (NLP) techniques that produce high dimensional feature vectors [92]. Furthermore, the low median of context inspection approach is just based on the four studies that reported the number of features used. However, for other ML countermeasures, it can be seen that the distribution inspection studies use high feature dimensions (ranging from 258-8000). While other countermeasure use relatively low number of features, e.g., (11/16) event-based studies utilized only $<100$, all the direct inspection approaches use $<80$ features while flow, hybrid, resource-usage and propagation-based approaches used $<25$ features. Fig 8c shows the relationship between the number of features with feature selection methods. From the analysis it is inferred that those studies which did not use any feature selection technique and reported features dimensions (41 studies) already had low feature dimensions, i.e., having a log median of $<12$. Among these only 4/41 of these studies used $>250$ features. Furthermore, the studies that apply dimensionality reduction methods and wrapper still result in a relatively high dimensional space (log median is approximately equal to 100).

Several studies (49%) use real datasets, which contain real traces of actual attack data shared through various public repositories. A study [S51] obtained malicious XSS-based webpages dataset from XSSed [148] and legal web pages dataset from ClueWeb09 [107]. Another study [S10] used a query dataset from the actual web application to detect insider threat. It is evident from Fig 9a that all the direct inspection studies used real datasets. However, the majority (11/15) of them were not publicly available, and only ten studies used publicly available real datasets. Real datasets are also prominent in context inspection (i.e., 13/17) and event-based (i.e., 8/16) studies, but most of them are not publicly available. However, none of the resource usage-based studies used real dataset. Fig 9b depicts the relationship of datasets with data exfiltration attacks, All the studies for detecting phishing, malicious domain and XSS attack use real datasets. Additionally, four studies detect RAT attacks based on real datasets. Table XI provides a list of publicly available datasets used by the studies along with a brief description. Other studies used real but private datasets. We group them into three types: (i) Collected privately: Some studies, e.g., [S10, S11, S16, S25, S35, S42], collected real data, e.g., by monitoring network traffic. However, these datasets are not publicly available for research reproducibility. For example, a study [S16] used dataset obtained from Microsoft Telemetry reports and Virus Total [130]. (ii) A random subset from threat intelligence feeds: 24 studies selected a random subset of data from publicly available threat feeds; however, the exact instances selected are not publicly available. For example, some studies [S5, S37, S46, S47, S53, S58, S40] used different and random instances from Phish-Tank [93]. Table XII provides the list of public attack threat intelligence feeds used by the studies. (iii) Hybrid dataset: Five studies [S13, S19, S30, S48, S61] combined data from the public corpus or other two types of private data to formulate their datasets. For example, A study [S13] used 10,400 deceitful files from VX-Heaven [109] public corpus consisting of 2600 instances of each worm, trojan, backdoor and virus and 1100 normal files samples were privately collected from Windows 7 system and Ninite.com.

32% of the studies used simulated datasets. These datasets are mostly used by Flow-based approaches (11 studies) and distribution inspection studies (7) mainly to detect data tunnelling (10 studies) and overt channel (6) attacks. All of these datasets are not publicly available. For example, [S39] used dataset collected for almost 83 days from five legitimate peer to peer communication applications ( Skype, eMule, eDonkey, FrostWire, µTorrent, BitTorrent, and Vuze) and three malicious software tools (Storm [135], Zeus [133] and Waledac [133]. Table XIII enlists the list of simulation tools used by the studies to collect malicious data.

One recent study [S80] used both simulated tools (Iodine, ReverseDNShell) and two real-world malware: FrameworkPOS [124] and Backdoor.Win32 Denis [123] to test their proposed anomaly detector (Isolation Forest Model). It is an interesting study illustrating the use of malware to exfiltrate data. However, this dataset is also not publicly available.

17% of the studies used synthetic datasets. Data is usually synthesized by using mathematical models or injecting malicious patterns in legitimate data to synthesis an attack. A study [S24] obtained legitimate network flow dataset from “Comprehensive, Multi-Source Cyber-Security” [149] and incorporates lateral movement attack traces “LANL NetFlow” logs and “Susceptible Infected Susceptible” (SIS) virus spread model [150]. Seven event-based studies use this dataset type, especially to detect attacks, e.g., insider threat (8), APT (2), Lateral Movement (1) and SQL injection (3). We can infer two key points: (i) multi-stage DE scenarios, especially, APT, is too complex to be simulated by using single or multiple software tools and require additional domain expertise. (ii) these attack vectors are subjective, i.e., they are driven by additional sophisticated factors such as persistence in case of APT [75], the role of an employee in case of insider threat [14], weak passwords in case of lateral movement [151]. Furthermore, seven studies used publicly available datasets. Among these six studies used CERT a publicly available synthetic datasets for detecting insider threat [110], while one study used DARPA 1998 [111] for detecting insider threat.

Fig 10a and Fig 10b analyse ML countermeasures and attack vectors with the type of modelling technique and their learning type used by the reviewed studies. Whilst most of the studies (74/92) utilised supervised, only three studies used both (supervised and unsupervised) learning types and classification as a modelling technique to train the models. Twelve studies employed anomaly-based detection; among these ten studies utilise supervised learning mechanism and trained an anomaly detectors using one class either legitimate or attack. However, two studies [S72, S91] employed unsupervised anomaly detection to detect insider threat. For instance, in [S72], the authors periodically trained an anomaly detector by using the psychometric score of users and their actions over time to detect insider threat. Anomaly-based detection is mostly applied by event-based and flow-based approaches. One reason behind it is the malicious behavior of an insider is dependent on multi-factors, such as their job role, psychometric factors which are difficult to define to perform classification. Three studies [S10, S24, S63] used both classification and anomaly-based methods to detect data exfiltration attacks. This combination is applied to complex attack scenarios like lateral movement, insider threat and side-channel attacks. For instance, one study [S24] applied unsupervised learning to detect lateral movement by using a propagation-based approach. In this study, two classifiers were trained to detect lateral movement. First uses k-mean [30] along with PCA to classify a host that behave like an infected host; the second uses extreme-value analysis to identify the host infected by the malicious lateral movement. Interestingly, four of the reviewed studies [S1, S4, S10, S18] used a combination of both learning types. All these studies divided a complex problem into sub-problems and solved them sequentially by using suitable learning types. For instance, in [S1], the authors performed a two-tier classification to detect malware. Firstly, seven supervised classifiers (SVM, DT, RFT, NB, GB, LR and KNN) were trained to detect known vs unknown malware family using a weighted voting scheme. Subsequently, for unknown malware detection, Shared Nearest Neighbour (SNN) clustering [152] was used to cluster unknown malware.

This section describes the classifier selected by the reviewed primary studies. We represent our analysis in terms of classifier structure, classifier type and classifier used by the studies. We have used acronyms of classifiers instead of their full name (please refer to section II-A2 for full names). Table XIV shows the classifiers used by the reviewed studies, along with their mapping with the type of ML countermeasures. These classifiers are employed by multiple studies either solely or with other classifiers. Furthermore, we have only mentioned those classifiers that were selected as best classifiers for a study because most of the studies initially tried out multiple classifiers. Considering the reflections from learning type, we classified the classifier structure into four types: base, ensemble, sequential and multi classifiers as shown in Fig 10c and Fig 10d. If a study fails to conclude the best classifier, we classify this study under multi classifier category. Table XV provides a brief definition of these classifier structures and provides their strengths and weaknesses.

In Fig 11, we present the results from the analysis performed to understand the relationship of the classifier with log number of features (linear range 2 to 8000) and dataset size (linear range 130 to  5000000). The log is used because of the high variance in distribution that hinders visualisation. From Fig 11a and Fig 11b, we observed that median of all the classifier structures and common classifiers except Adaboost and OCSVM lie in the log range between 10 to 100 which shows that the number of features chosen to train the model is independent of the classifier structure and classifier. However, the studies using Adaboost classifier used a comparatively large number of features (median $log>100$) while OCSVM used less number of features (median $log<10$). Furthermore, Fig 11c investigates the relationship of commonly used classifiers by the studies and log of dataset size. All the classifiers except KNN, RFT are used with high dimensional datasets (median $log>10000$) to detect data exfiltration.

Hold Out, K-Fold, Leave Out one and Bootstrapping are the most popular (please refer to [49, 26] for details) methods applied by ML approaches to validate the classifier performance. Fig 12 shows an analysis the validation methods used by the reviewed studies based on ML countermeasures, classifier structure and dataset size.

Fig 12a illustrates the distribution of ML-based countermeasures with respect to validation techniques. The K-Fold method is dominant (i.e., 36/92 studies) in all the approaches. K-Fold cross-validation is less bias estimation of accuracy [48], and the entire dataset is used for training and testing. Hold Out method is the second most common (i.e., 32/92 studies) validation methods. However, it is not used in the case of propagation and resource usage-based approaches. Three studies [S40, S51, S77] used both Hold Out and K-Fold methods; six studies [S1, S9, S13, S15, S46, S47] used Hold Out or K-Fold with boosting ensemble. One study [S28] used bootstrapping as a validation technique. Fig 12b shows a relationship between the validation method with the type of classifier. It is evident that base classifiers mostly use the holdout method (20 studies); while K-Fold and bootstrapping is a popular among studies using ensemble and more specifically for DT classifiers. Whereas 14 studies did not report any validation method. We have also investigated the validation technique used in terms of training dataset size, as shown in Fig 12c. Hold Out and combination of Holdout and K-Fold method is used with large training datasets (log median approximately 100k), whereas K-Fold is frequently used for medium datasets (log median approximately 10k). From the above analysis, we conclude that K-Fold and Hold Out standout among other approaches. However, they both have a certain limitation. Hold Out method can be subject to overfitting and is highly dependent on data class distribution as well as the split ratio. Whereas the K-Fold method reduces overfitting as ‘k’ is increased, but due to its iterative nature, it requires high computational time [48, 59]. Hence, the choice of validation method is dependent on constraints such as class distribution, dataset size and split ratio.

In this section, we analyze the performance achieved by the models trained to detect data exfiltration attack vectors in terms of ML countermeasures, attack vectors and classifiers, as shown in Fig 13. We have used accuracy, FPR and F-score to evaluate the performance of the reviewed studies because they collectively give a precise estimation of the performance of the models [26]. Fig 13b-d shows a boxplot depicting the performance reported by the reviewed studies. The x-axis with no boxplot indicates that no study reported the performance measure for that label.