MalDetConv: Automated Behaviour-based Malware Detection Framework Based on Natural Language Processing and Deep Learning Techniques

Extracting behavioural features from malware executable files is a very critical task as malware can damage organizational resources such as corporate networks, confidential information, and other resources when they escape the analysis environment. Hence, obtaining a good and up-to-date behavioural representation of malware is a challenging task [33]. In the case of API calls, having a good representation of API call features from benign and malware programs also remains a challenge as the number of API calls made by malware executable files is relatively long which makes their analysis and processing difficult [28]. In addition, some of the previous techniques have used API calls to detect malicious executable files. Such API call-based malware detection techniques include the ones presented in [28] [34] [35] [36] and [29]. Unfortunately, none of the previous work has attempted to reveal API call features that contributed to the final prediction/classification outcome. Practically, it is ideal to have a machine learning or deep learning model that can detect the presence of malicious files with high detection accuracy. However, the prediction of such a model should not be blindly trusted, but instead, it is important to have confidence about the features or attributes that contributed to the prediction.

In order to improve the performance of existing malware detection techniques, this work proposes MalDetConv, a new automated framework for detecting both known and zero-day malware attacks based on natural language processing (NLP) and deep learning techniques. The motivation for using deep learning is to automatically identify unique and high relevant patterns from raw and long sequences of API calls which distinguishes malware attacks from benign activities, while NLP techniques allow producing numerical encoding of API call sequences and capturing semantic relationships among them. More specifically, MalDetConv uses an encoder based on NLP techniques to construct numerical representations and embedding vectors (dense vectors) of each API call based on their semantic relationships. All generated embedding vectors are then fed to a CNN module to automatically learn and extract high-level features of the API calls. The final features generated by CNN are then passed to a bidirectional gated recurrent unit (BiGRU) feature learning module to capture more dependencies between features of API calls and produce more relevant features. We believe that combining CNN and BiGRU creates a hybrid automatic feature extractor that can effectively capture relevant features that can be used in detecting malicious executable files. Finally, features generated by the BiGRU module are fed to a fully connected neural network (FCNN) module for malware classification. We have also integrated LIME into our framework to make it explainable [37]. LIME is a framework for interpreting machine learning and deep learning black box models proposed by Ribeiro et al. [32]. It allows MaldetConv to produce explainable predictions, which reveal feature importance, i.e., LIME produces features of API calls that contributed to the final prediction of a particular benign or malware executable file. Explainable results produced by LIME can help cybersecurity analysts or security practitioners to better understand MalDetConv’s predictions and to make decisions based on these predictions. The experimental evaluations conducted using different datasets show better performance of the MalDetConv framework over other state-of-the-art techniques based on API call sequences. Specifically, the following are the contributions of this work.

1. 1.

   This work contributes ”MalBehavD-V1”, a new behavioural dataset of API call sequences extracted from benign and malware executable files. We have used the MalBehavD-V1 dataset to evaluate the proposed framework and our dataset is made publicly available for use by the research community. Our dataset contains behaviours of 1285 new malware samples that appeared in the second quarter of 2021 and were not analyzed by the previous works.

2. 2.

   We designed and implemented the MalDetConv framework using an NLP-based encoder for API calls and a hybrid automatic feature extractor based on deep learning techniques.

3. 3.

   MalDetConv uses LIME to provide explainable predictions which are essential for security analysts to identify interesting API call features which contributed to the prediction.

4. 4.

   Detailed experimental evaluations confirm the superiority of the proposed framework over existing when detecting unseen malware attacks with an average of 97.48% in the detection accuracy obtained using different datasets of API calls. Our framework outperforms existing techniques such as the ones proposed in [34] [38] [29].

Structure: The remainder of this paper is structured as follows. Section 2 presents the background and Section 3 discusses the related works. Section 4 presents the proposed method while Section 5 discusses the experimental results. Section 6 presents limitations and future work. The conclusion of this work is provided in Section 7.

Malware production is increasing enormously with millions of threats targeting Windows OS. Developed by Microsoft, Windows is the most widely used and distributed desktop OS with the highest global market share [39]. The distribution of this OS market share can also be viewed in Figure 1. Consequently, its popularity and widespread usage give many opportunities to cybercriminals to create various malware applications (malicious software) against Windows-based systems or devices [40] [41].

The Windows application programming interface (API), also known as Win32 API, is a collection of all API functions that allow windows-based applications/programs to interact with the Microsoft Windows OS (Kernel) and hardware [42] [43] [44]. Apart from some console programs, all windows-based applications must employ Windows APIs to request the operating system to perform certain tasks such as opening and closing a file, displaying a message on the screen, creating, writing content to files, and making changes in the registry. This implies that both system resources and hardware cannot be directly accessed by a program, but instead, programs need to accomplish their tasks via the Win32 API. All available API functions are defined in the dynamic link libraries (DLLs), i.e., in .dll files included in C:\Windows\System32\*.For example, many commonly used libraries include Kernel32.dl, User32.dll, Advapi32.dll, Gdi32.dll, Hal.dll, and Bootvid.dll [45].

Generally, any windows-based program performs its task by calling some API functions. This functionality makes Win32 API one of the important and core components of the Windows OS as well as an entry point for malware programs targeting the windows platform, since the API also allows malware programs to execute their malicious activities. Therefore, monitoring and analyzing Windows API call sequences gives the behavioural characteristics that can be used to represent benign and malware programs [46] [47]. API calls analysis reveals a considerable representation of how a given malware program behaves. Therefore, monitoring the program’s API call sequences is by far one of the most effective ways to observe if a particular executable program file has malicious or normal behaviours [48] [49] [50] [51] [34].

Deep learning (DL) techniques are subsets of machine learning techniques that use artificial neural network architectures to learn and discover interesting patterns and features from data. DL network architectures can handle big datasets with high dimensions and perform automatic extraction of high-level abstract features without requiring human expertise in contrast to ML techniques [52] [53]. DL algorithms are designed to learn from both labeled and unlabeled datasets and produce highly accurate results with low false-positive rates [54]. By using a hierarchical learning process, deep learning algorithms can generate high-level complex patterns from raw input data and learn from them to produce intelligent classification model which performs classification tasks, making deep models valuable and effective for big data manipulation. The multi-layered structure adopted by deep learning algorithms gives them the ability to learn relevant data representations through which low-level features are captured by lower layers and high-level abstract features are extracted by higher layers [55] [56]. The next section introduces CNN and recurrent neural networks, which are some of the popular categories of DL algorithms.

There have been significant efforts in the recent works on malware detection through static and behaviour-based analysis. Static API calls extracted after disassembling benign and malware executables with IDA Pro Disassembler were mined using a hybrid of SVM wrapper and filter methods [10]. Different subsets of features selected by the SVM wrapper were combined with feature subsets generated by the filter method to construct a good representation of API call features and the final feature set was then employed to build different hybrid SVM-based malware detection models. While the model performs well when classifying malware based on API call features, there is no clear approach for selecting filters, which can affect the overall performance, especially for the hybrid methods. The work presented by Naik [7] has implemented a fuzzy-import hashing approach to measure similarity between malware and benign files and their results were further compared with YARA rules. The dataset with four ransomware families namely, WannaCry, Locky, Cerber Ransomware, and CryptoWall was used to evaluate the performance of their proposed approach. YARA rules were generated from strings extracted from these ransomware samples and different Fuzzy-Import Hashing approaches (SDHASH, IMPHASH, SSDEEP, etc.,) were tested. The results reveal that there is a high similarity between ransomware from the same family and a high dissimilarity between ransomware from different families. API calls and API calls’ statistics extracted from binary files through the static malware analysis were used in the detection model proposed in Huda et al.’s work [77]. The step-wise binary logistic regression (SBLR) model was applied to select a subset of specific features and thereafter, the Decision Tree and SVM algorithms were trained and tested to detect malicious activities. While the focus of their work was to reduce the computation complexity, this technique depends on a linear approach which can affect the overall detection accuracy of the proposed model. An instruction sequence-based malware detection framework was proposed in Fan et al.’s work [78] where sequences were extracted from executable samples and malicious sequential patterns were discovered using a sequence mining approach that works in combination with All-Nearest Neighbour (ANN) classifier. Based on the experimental results, this method can identify malicious patterns from given suspicious executable programs with improved detection accuracy.

A signature-based malware detection approach was implemented using deep autoencoders in [24]. An opcode sequence analysis method was used to construct a static-based method that effectively detects malware attacks [9]. Opcode sequences were statically extracted from 32-bit and 64-bit Windows EXE files and the occurrence of each opcode was computed using the term frequency-inverse document frequency (TF–IDF) to obtain a feature set that was used to train KNN, Decision Tree, Adaboost, RF, Bagging and backpropagation neural networks detection models. A total of 20,000 files (malware and benign) was used in the experiment and k-fold cross-validation was used to evaluate the performance of the proposed methods. The results reveal that their malware detection systems can identify and classify different malware attacks with the better performance achieved by the Adaboost model. Logistic regression was used to learn relevant features (to perform domain knowledge operation) from raw bytes to reveal the best byte n-grams features which were further employed to train the Random Forest and Extra Random Trees using the JSAT library [79]. Furthermore, they have also built a long short-term memory (LSTM) and fully connected neural networks detection approach using raw bytes and Keras library [80]. Their performance evaluation shows that neural network models trained on raw bytes features without performing explicit feature generation can outperform the performance of domain knowledge methods that fully depends on constructing explicit features from raw bytes extracted from PE headers [79]. D’Onghia et al. [81] proposed Apícula, a static analysis-based tool that uses the Jaccard index to identify malicious API calls presented in bytes streams such as network traffic, object code files, and memory dumps. The work presented by Kundu et al. [82] has improved the performance of a Light Gradient Boosted Machine (LightGBM) approach for detecting and classifying malicious software using two datasets, the Microsoft EMBER dataset and another private dataset collected from an anti-virus company. The optimization (hyper-parameter tuning) was performed using Microsoft Neural Network Intelligence (NNI) and AutoGluon-Tabular (AutoGluon) automated machine learning frameworks. The AutoGluon was released by Amazon and has been implemented with various ML algorithms including KNN, LightGBM, and Multi-Layer Perceptron (MLP). The results obtained after performing different empirical analyses demonstrate that there is an improvement in tuned models compared to the baseline models. Kale et al.’s [83] used opcode sequences from EXE files to build malware detection approaches based on RF, SVM, KNN, and CNN where feature extraction and representation were performed using different techniques such as Word2Vec, BERT, HMM2Vec, and ELMo. Yeboah et al.[84] proposed a CNN-based approach that detects malicious files using operational code (opcode).

Nevertheless, cybercriminals can easily modify the executable binary file’s code and disguise a small piece of encrypted malicious code, which can be decrypted during its execution/runtime [85]. Such malicious code can even be downloaded remotely and get inserted into the program while running. Accordingly, the work presented by [86] has also revealed that it is possible for encrypted and compressed code to be decrypted and unpacked when loaded into memory during runtime. Additionally, some sophisticated malware uses more than two stages to attack the victim’s systems. For instance, malware designed to target specific platforms such as Windows OS will first check if it is running in the targeted OS. Once detected, a malicious payload can automatically be downloaded from the cybercriminal’s server and get loaded into memory to perform malicious tasks. Malware with the ability to evade antivirus and utilizes it to transfer users’ stolen confidential information from the comprised users without any notice was reported in [85]. Recently advanced malware variants use varieties of sophisticated evasion techniques including obfuscation, dead code insertion, and packing [87] [88] to evade static-based malware detection, making them more vulnerable and unable to detect new sophisticated malware attacks. Fortunately, the dynamic malware analysis approach can handle obfuscated malware and consequently, many of the recent works were focused on dynamics-based approaches/techniques. The work proposed by Vemparala et al. [17] extracted API call sequences through dynamic analysis and employed them to train and test both hidden Markov Models and Profile Hidden Markov Models to observe the efficiency of the dynamic analysis approach while detecting malware. Their study has used a dataset of seven malware types with some malware programs such as Zbot and Harebot that have the capability of stealing confidential information. The results from their experiment show that the HMM-based malware detection approach using behavioural/dynamic sequences of API calls outdoes both static and hybrid-based detection techniques. Nevertheless, the authors have used an outdated dataset, which can prevent the model from identifying new sophisticated malware attacks, considering the rapid scale of different malware variants.

The work in [47] has adopted the DNA sequence alignment approach to design a dynamic analysis-based method for malware detection. Common API call features were dynamically extracted from different categories of malware. Their experimental outcome has revealed that some of the malicious files possess common behaviours/functions despite their categories, which may be different. In addition, their study has also indicated that unknown or new malware can be detected by identifying and matching the presence of certain API calls or function calls as malware programs perform malicious activities using almost similar API calls. However, the limitation of DNA sequence approaches is that they are prone to consuming many resources and require high execution time, making them computationally expensive, considering the high volume of the emerging malware datasets. Seven dynamic features, namely, API calls, mutexes, file manipulations/operations, changes in the registry, network operations/activities, dropped files, and printable string information (PSI) were extracted from 8422 benign and 16489 malware using the Cuckoo sandbox [8]. PSI features were processed using a count-based vector model (count vectorization approach) to generate a feature matrix representing each file and thereafter, truncated singular value decomposition was utilized to reduce the dimension of each generated matrix. Furthermore, they have computed Shannon entropy over PSI and API call features to determine their randomness. Using PSI features, their model has achieved the detection accuracy of 99.54% with the Adaboost ensemble model while the accuracy of 97.46% was yielded with the Random Forest machine learning classifier. However, while their method shows an improvement in the accuracy of the detection model using PSI features, it is worth mentioning that the count vectorization model applied while processing features does not preserve semantic relationship/linguistic similarity between features or word patterns [89] [90]. The count vectorization model is unable to identify the most relevant or less relevant features for the analysis, i.e., only words/features with a high frequency of occurrence in a given corpus are considered as the most statistically relevant words.

Cuckoo sandbox was used to generate behavioural features of benign and malware files during their execution time in the work carried out in [91]. They have used Windows API calls to implement a Random Forest-based malware detection and classification approach that achieved the detection accuracy of 98%. Malicious features were extracted from about 80000 malware files including four malware categories, namely, Trojans, rootkit, adware, and potentially unwanted programs (PUPs) downloaded from VirusTotal and VirusShare malware repository. The detection model was tested on 42000 samples and the results show an improvement in the detection of malware attacks. However, looking at the detection outcome, this approach only performs well when detecting Trojans. For instance, it achieved the true positive rate (TPR) of 0.959 when detecting and classifying Trojans while the TPR of 0.777, 0.858, and 0.791 were achieved while detecting rootkit, adware, and PUPs, respectively. In the work presented by Suaboot et al. [28] a subset of relevant features was extracted from API call sequences using a sub-curve Hidden Markov Model (HMM) feature extractor. Benign and malware samples were executed in the Cuckoo sandbox to monitor their activities while executing in a clean isolated environment residing in Windows 7-32 bits machine. The API call features selected from the behavioural reports of each executable program file (benign and malware) were then used to train and test a behaviour-based malware detection model. Only six malware families (Keyloggers, Zeus, Rammit, Lokibot, Ransom, and Hivecoin) with data exfiltration behaviours were used for the experimental evaluation. Different machine learning algorithms such as Random Forest (RF), J48, and SVM were evaluated, and the RF classifier achieved better prediction with an accuracy average of 96.86% compared to other algorithms. Nevertheless, their method was only limited to evaluating executable program files, which exfiltrates confidential information from the compromised systems. In addition, with 42 benign and 756 malware executable programs which were used in the experimental analysis, there is a significant class imbalance in their dataset which could lead to the model failing to predict and identify samples from minority classes despite its good performance.

A backpropagation neural networks (BPNNs) approach for malware detection was proposed in [92]. The proposed BPNNs–based approach learns from features extracted from benign and malware behavioural reports to classify malware attacks. A dataset of 13600 malware executable files was collected from Kafan Forum and the dataset has ten malware families which were labeled using Virscan. They also have used the HABO online behaviour-based system, which captures the behaviours of each file and generates a behavioural report having all activities performed by the executable file. Behavioural features such as mutexes, modified registry keys, and created processes, to name a few, were used to train and test the proposed BPNNs approach to perform binary classification, which led to the accuracy of 99%. Note the BPNN was built using a sub-behavioural feature generated based on a count-based method which is limited to using an occurrence matrix that does not provide any relationship between the selected sub-behavioural feature set. Malware attacks can be detected using the detection techniques implemented based on the longest common substring and longest common subsequence methods suggested in [93]. Both methods were trained on behavioural API calls, which were captured from 1500 benign and 4256 malware files during dynamic analysis. A Malware detection approach based on API calls monitored by the Cuckoo sandbox was designed in the work proposed by Kyeom et al. [94]. The analysis was carried out using 150 malware belonging to ten malware variants and the detection was achieved by computing similarity between files based on the extracted sequence of API calls features using the proposed sequence alignment approach. Their results also show that similar behaviours of malware families can be found by identifying a common list of invoked API call sequences generated during the execution of executable program. Unfortunately, this method cannot suitable for high-speed malware attacks detection as it fully relies on the pairwise sequence alignment approach, which introduces overheads. The work in [95] has proposed multiple instance-based learning (MIL) approach that exploits the performance of several machine learning classifiers (SVM, KNN, RF, etc.) to detect malicious binary files. The MIL was trained using different dynamic features such as network communication (operations), the structure of file paths, registry, mutexes, keys, and error messages triggered by the operating system. Behaviours of benign and malicious files were monitored in the sandbox and a similarity-based method was used in combination with clustering which allows similar systems resources to be grouped together. MIL has achieved a detection accuracy of 95.6% while detecting unknown malware files.

Several graph-based techniques for malware detection were proposed in the previous studies [96] [97] [98] [99] [100] and [101], to mention a few. All of these approaches have used graphs to represent behaviours of executable files and the detection models learned from the generated graphs. Nevertheless, the complexity of graph matching is one of the major issues with graph-based techniques, i.e., as the graph’s size increases the matching complexity, the detection accuracy of a given detection model decreases [8]. Grouping a common sequence of API calls in a single node is often applied to decrease the matching complexity of graphs. Although this approach does not yield better detection accuracy, it is harder for a cyber attacker to modify the behaviours of a malware detection model based on the graph method [99]. NLP techniques were applied to analyze/mine the sequence of API calls in the malware detection approach implemented in [102]. The API calls were processed using the n-gram method and the weights were assigned to each API call feature using the term frequency-inverse document frequency (TF-IDF) model. The main goal of the TF-IDF is to map n-grams of API calls into numerical features that are used as input to machine learning or deep learning algorithms. Given an input of behavioural report extracted from malware and benign files in a dynamic isolated environment, the TF-ID, processes the report to generate feature vectors by taking into account the relative frequency of available n-grams in the individual behavarioural report compared to the total number of the report in the dataset. The work in [38] has also relied on TF-IDF to compute input features for machine learning algorithms (CART, ETrees KNN, RF, SVM, XGBoost, and Ensemble). Another previous work in [23] has used LSTM and TF-IDF model to build a behaviour-based malware detection using API call sequences extracted with cuckoo sandbox in a dynamic analysis environment.

The work in [33] has used the term frequency-inverse document frequency to process printable strings extracted from malware executable files after dynamic analysis. TF-IDF and Anti-colony optimization (Swarm algorithm) were used to implement a behavioural graph-based malware detection method based on dynamic features of API calls extracted from executable [29]. Unfortunately, like the count vectorization model, the TF-IDF text processing model does not reveal or preserve the semantic relationship/similarity that exists between words. In the case of malware detection, this would be the similarity between API calls or another text-based feature such as file name, dropped messages, network operations such as contacted hostnames, web browsing history, and error message generated while executing the executable program file. Liu and Wang have [103] used Bidirectional LSTM (BLSTM) to Build an API call-based approach that classifies malware attacks with an accuracy of 97.85%. ALL sequences of API calls were extracted from 21,378 and were processed using the word2vec model. In Li et al. [104] a graph convolutional network (GCN) model for malware classification was built using sequences of API calls. Features were extracted using principal component analysis (PCA) and Markov Chain. A fuzzy similarity algorithm was employed by Lajevardi [105] to develop dynamic-based malware detection techniques based on API calls. Maniath et al. [106] proposed an LSTM-based model that identifies ransomware attacks based on Behavioural API calls from Windows EXE files generated through dynamic analysis in the Cuckoo sandbox. Chen et al.’s work [107] proposed different malware detection techniques based on CNN, LSTM, and bidirectional LSTM models. These models were trained on raw API sequences and parameters traced during the execution of malware and benign executable files. An ensemble of ML algorithms for malware classification based on API call sequences was recently implemented in the work presented in [108]. Convolutional neural networks and BiLSTM were used to develop a malware classification framework based on sequences of API calls extracted from executables files [21]. Two detests were used by Dhanya et al. [109] to evaluate the performance of various machine learning-based ensemble models (AdaBoost, Random Forest, Gradient descent boosting, XGBoost, Stacking, and Light GBM) for malware detection. Their study has also evaluated DL algorithms such as GRU, Graph Attention Network, Graph Convolutional Network, and LSTM. The work proposed in [110] has employed a dataset of API invocations, Registry keys, files/directory operations, dropped files, and embedded strings features extracted using the Cuckoo sandbox to implement a particle swarm-based approach that classifies ransomware attacks. Jing et al. [111] proposed Ensila, an ensemble of RNN, LSTM, and GRU for malware detection which was trained and evaluated on dynamic features of API calls.

Some of the previous studies presented above have generated datasets of API calls and have made them public for the research community focusing on malware detection. Examples of such datasets include the ones presented in [112], [113], [47] and [114]. Nevertheless, many existing studies did not share their datasets and as result, a few datasets are publicly available for the research community. The existing datasets are also outdated as they are not regularly updated to include new behavioural characteristics of malware. This situation hinders the development and evaluation of new malware detection models as building these models requires updated datasets having the necessary behavioural characteristics of new malware variants. Therefore, to contribute to the existing challenge related to the availability of benchmark datasets [33], this work generates a new behavioural dataset of API calls based on Windows executable files of malware and benign using the Cuckoo sandbox.

More specifically, we generate MalbehavD-V1, a new dataset that has the behavioural characteristics of current emerging malware such as ransomware, worms, Viruses, Spyware, backdoor, adware, keyloggers, and Trojans which appeared in the second quarter of 2021. The dataset has been processed to remove all inconsistencies/noise, making it ready to be used for evaluating the performance of deep learning models. The dataset is labeled and the hash value for each file has been included to avoid duplication of files while extending the dataset in the future, which makes it easier to include behavioural characteristics of newly discovered malware variants in the dataset or combine the dataset with any of the existing datasets of API calls extracted from Windows PE files through dynamic analysis. More details on the dataset generation are presented in Section 4 and the dataset can be accessed from the GitHub repository in [115].

It is often challenging to find an up-to-date dataset of API calls of Windows executable files. For this reason, we have generated a new dataset of API calls which is used for the experimental evaluations. Different sources of malware executable files such as Malheur [116], Kafan Forum [92], Danny Quist [117], Vxheaven [10], MEDUSA [118], and Malicia [119] were used in the previous studies. Unfortunately, these repositories are not regularly updated to include new malware samples. Hence, we have collected malware executable samples from VirusTotal [120], the most updated and the world’s largest malware samples repository. Nevertheless, considering millions of malware samples available in the repository, processing all malware samples is beyond the scope of this study. Thus, only malware samples submitted in the second quarter of 2021 were collected. We were given access to a Google drive folder having the malicious EXE files which are shared by VirusTotal. Benign samples were collected from CNET site[121]. The VirusTotal online engine was used to scan each benign EXE file to ensure that all benign samples are clean. A total number of 2800 EXE files were collected to be analyzed in an isolated analysis environment using the dynamic analysis approach. However, we experienced issues while executing some files, resulting in a dataset of 2570 files (1285 benign and 1285 malware) that were successfully executed and analyzed to generate our MalbehavD-V1 benchmark dataset. Some benign files were excluded as they were detected as malicious by some of the anti-malware programs in the VirusTotal online engine [122] while some malware files did not run due to compatibility issues. The next sub-Sections (4.2 and 4.3) present the steps followed to generate the dataset.

Monitoring and capturing sequences of API calls from benign and malware executable files through dynamic analysis mostly rely on the use of a sandbox environment as an analysis testbed. This is because automated sandboxes can monitor behaviours of malware executable programs during runtime while preventing them from damaging the host system or production environment. As shown in Figure 5, our isolated environment consists of the main host, host machine, Cuckoo sandbox, virtualization software, and virtual machines (VMs). The main host is a Windows 10 Enterprise edition (64bit) with direct access to the Internet. Considering that some very advanced malware can escape the sandbox analysis environment which could lead to serious damage and a disastrous situation when malware infects the production systems, we have used Oracle VirtualBox, one of the leading virtualization software to isolate the main host and the analysis environment which fully resides in Ubuntu 20.04.3 LTS (Focal Fossa) host machine. More specifically, Oracle VirtualBox 6.1 was installed on Windows 10 main host machine, and then Ubuntu host machine was installed inside the VirtualBox environment.

The Cuckoo sandbox [123] and its dependency software including required analysis modules were installed on the Ubuntu machine. We have installed another VirtualBox that manages all Windows 7 Professional 64-bit SP1 analysis VMs on the Ubuntu machine. A set of software was installed on each virtual machine and some security settings were disabled to make them more vulnerable to malware attacks. Examples of such software include Adobe reader 9.0, Java runtime environment (JRE 7), .NET Framework 4.0, Python 2.7 (32 bit), Pillow 2.5.3 (Python Imaging Library, 32 bit), Microsoft Office, and Cuckoo sandbox guest agent (32 bit). Python allows the Cuckoo guest agent to run while the pillow provides imaging functions such as taking screenshots. The cuckoo agent resides in each Windows VM under the Startup sub-directory. This allows the agent to automatically start whenever the VM boots up. A shared folder was configured in the Windows virtual machine to get all the above files, however, it was disabled after installing each file as some advanced malware looks for the shared folder to discover if they are running in a virtual environment or sandbox. As the Cuckoo guest agent must monitor each file’s behaviours while running and send all captured data back to the host, an Internet connection is needed between the Windows analysis VMs and Ubuntu host machine. Thus, we have set the network adaptor to ”host-only Adaptor” in the VirtualBox to allow the network traffic to only be routed between the Windows virtual machines and Ubuntu host machine without reaching the main host (Windows 10 Enterprise host machine).

The architecture of the analysis environment is mainly based on nested virtualization technology which allows deploying nested virtual machines in the same host. It is also important to note the hardware virtualization on each VirtualBox was set to “enable nested paging” while the paravirtualization interface was set to default. After installing all required software in the Windows 7 VMs, a clean snapshot of each virtual machine was taken and saved. This snapshot is used to restore the virtual machine to its clean state after the analysis of each executable program file. Cuckoo sandbox was configured to generate a behavioural report of each EXE file in JSON format, and each report was further processed to extract relevant features of API calls. A typical structure of JSON report from our dynamic analysis is presented in Figure 7 while details on a complete analysis workflow are given in Figure 6 which describes how each file is monitored by the Cuckoo sandbox at runtime. .

Accordingly, the behavioural reports generated during our dynamic malware analysis reveal that some sophisticated malware can use different API calls that potentially lead to malicious activities. For instance, Table 1 presents some of the API calls used by ae03e1079ae2a3b3ac45e1e360eaa973.virus, which is a ransomware variant. This ransomware ends up locking the comprised device (Windows VM in our analysis) and demands for a ransom to be paid in BitCoins in order to get access back to the system. This variant encrypts the hard drive’s contents and makes them inaccessible. We have also observed that recent malware variants possess multiple behaviours and can perform multiple malicious activities after compromising the target, making detection difficult.

The pre-processing of the behavioural report for each analyzed EXE file is performed at this stage. The pre-processing involves organizing and cleaning the raw features from the behavioural reports to make them suitable for training and testing the classification algorithm. After generating all JSON behavioural reports, they are stored in a local directory in the host machine. A typical JSON report contains various objects storing different data and all behavioural features are located under the “Behaviours” object in each JSON report. However, we are only interested in extracting sequences of API calls of benign and malware executable program files. As JSON stores raw features of API calls, they are not supported by deep learning algorithms. Hence, they must be processed to obtain a numerical representation of each PE file’s API call features that is supported by deep learning models.

The processing involves two main stages. The first stage involves processing all JSON reports to generate a comma separate value (CSV) file. Figure 8 presents various steps that are performed to accomplish the first processing stage. The second stage deals with processing the CSV file to generate a numerical representation of each feature and construct the embedding matrix which is fed to the proposed CNN-BiGRU hybrid automatic feature extractor. When a CSV file containing sequences of API calls that represent benign or malware program files is fed to the proposed framework, the pre-processing module transforms this CSV file into a pandas data frame which is much easier to process. A list of labels that are encoded in binary (1,0), with 1 representing each row of malware and 0 representing each row of benign is loaded and then appended to the data frame to obtain a new data frame with labeled features.

In the next step, the data are shuffled and passed to a splitting function which splits the dataset into training and test sets. All sequences of API calls (in both training and test set) are tokenized using the Tokenizer API from the Keras framework to generate a unique integer encoding of each API call. Thereafter, all encoded sequences are given the same length and padded where necessary (zero padding is used). This is very important for CNN as it requires all inputs to be numeric and all sequences to have the same length. The encoded sequences are fed to the embedding layer which builds a dense vector representation of each API call and then all vectors are combined to generate an embedding matrix which is used for CNNs. More specifically, all steps performed in the second pre-processing stage are presented in Figure 9 and further details are provided in subSection 4.6. Additionally, details on the API call embedding are presented in Section 4.4.

This section first introduces word embedding and then provides details on the Keras embedding layer which is used to construct embedding vectors of API calls in this work.

The next component of the proposed framework is the CNN-BiGRU hybrid automatic feature extractor, and its architecture is presented in Figure4 and 10. More formally, our binary classification problem of benign and malware’s API call sequences with MalDetConv can be addressed as follows.

Given a JSON report containing features of API call sequences representing benign or malware executable programs, the proposed framework performs classification based on all steps described in the previous Sections. First, each JSON file is processed to extract raw sequences of API call features which are followed by encoding each sequence of API calls to obtain an encoded integer representation for each API call. Thereafter, all encoded features are passed to the embedding layer which builds embedding vectors and then concatenates them to generate API calls embedding matrix which is passed to the CNN-BiGRU hybrid automatic feature extractor. The CNN module has filters that convolve to the embedding matrix to select high-level features through convolution operations. The selected features are aggregated together to generate a feature map which is passed to the pooling layer to generate high-level and compact feature representation through max pooling operations. The output from the last pooling layer is passed to a BiGRU layer to capture dependencies between features of API call and constructs more relevant features which are flattened by the flatten layer and then passed to a fully connected neural network module which performs classification. A complete workflow is depicted in Figure 11.

The proposed framework was implemented and tested in a computer running Windows 10 Enterprise edition (64bit) with Intel(R) Core (TM) i7-10700 CPU @ 2.90GHz, 16.0 GB RAM, NVIDIA Quadro P620, and 500 GB for the hard disk drive. The framework was implemented in Python programming language version 3.9.1 using TensorFlow 2.3.0 and Keras 2.7.0 frameworks and other libraries such as Scikit-learn, NumPy, Pandas, Matplotlib, Seaborn, LIME, and Natural Language Toolkit (NLTK) have been also used. All these libraries are freely available for public use and can be accessed from PiPy [144], the Python package management website. The proposed framework was trained and tested using sequences of API call features extracted from Windows EXE files.

Our generated dataset (MalbehavD-v1) was employed to evaluate the performance of the MalDetConv framework. As we wanted the proposed framework to learn and be tested on a variety of different datasets, we have also collected other existing datasets of malicious and normal API calls for our experimental analysis. These datasets include the ones in [47] [114], API calls datasets presented in [112] and [143]. Using all these datasets allows us to assess the performance of the proposed framework while detecting malware attacks.Note that these datasets are freely available for public use. We have used 70% of each data for training while the remaining portion (30%) was used for testing and details on each dataset are presented in Table 2.

Different metrics such as precision (P), recall (R), F1-Score, and accuracy were measured to evaluate the performance of the proposed framework. The computations of these metrics are presented in equations (13), (14), (15), and (16), with TP, TN, FP, and FN indicating True Positives, True Negatives, False Positives, and False Negatives, respectively. Additionally, we have also measured the execution time taken while training and testing MalDetConv.