Zero Day Ransomware Detection with Pulse: Function Classification with Transformer Models and Assembly Language

Due to the rapid increase in the use of digital devices and widespread internet access, the frequency of cyber-attacks targeting information systems is escalating. Malicious software (malware) is the primary weapon used by attackers in executing these cyber-attacks. Various types of malware such as ransomware, trojans, botnets and spyware are continually evolving, incorporating advanced encryption and obfuscation techniques [1]. Further, each type of malware has distinct objectives and functionality [2]. This research focuses on developing Pulse, a framework that uses Transformer models and the Peekaboo Dynamic Binary Instrumentation (DBI) data to detect zero day ransomware [3].

According to Sophos, 59% of 5000 surveyed organizations across 14 countries were hit by ransomware in 2023, where the average ransom demand was USD 4,321,880 [4]. Further, IBM report the average cost of a data breach was USD 4.45 million in 2023, while the average time to identify and contain a breach was 277 days [5]. Ransomware is designed to encrypt victims’ files [6, 7]. Recently, ransomware gangs have added data theft to their tactics, targeting sensitive information including personal details and intellectual property. They threaten to publish or sell this data if the ransom isn’t paid, a method known as double extortion [6, 7]. This approach often results in larger ransoms and higher payments, boosting profits compared to attacks involving encryption alone [8].

Software and malware consist of sets of data, instructions and functions designed for specific tasks, with many distributed in an executable binary format [9]. Anti-malware vendors commonly employ signature-based methods to identify known threats, and heuristic based detection [10, 11, 12]. A signature refers to a sequence of bytes derived from unique data within a binary [12, 13, 14]. When new malware emerges, vendors obtain a sample and define its signature based on strings, URLs, code segments and other patterns in the binary [12, 13, 14]. However, this process is manual and time-consuming, involving both the creation of the signature and its distribution through updates [12]. Moreover, recycled malware that has been altered or obfuscated can evade signature-based detection systems [14]. Creating signatures for new or modified malware is time-consuming due to the analysis required and the client update, which may not occur for months. [15, 12, 13]. Further, with the rapid growth of AI, it is likely that malware will soon use AI techniques, specifically ransomware could adapt encryption methods intelligently and dynamically, making it harder to detect [16]. Consequently, finding automated AI techniques to proactively defend against malware has become increasingly critical. However, the ability of an AI model to correctly classify novel malware is dependent on the quality of the data and features it is trained with. Further the authenticity and veracity of the features is dependent on the analysis tool and the dataset [1, 17, 18].

Evasive malware easily defeats widely used static and dynamic analysis tools due to their various limitations, whereas DBI facilitates the capture of genuine behaviour from sophisticated malware [19, 20, 21, 22, 3]. If behavioural data is extracted from malware while it is concealing its malicious intent and true behaviour in an analysis environment, any AI model that is subsequently trained with that data would not correctly identify the malware when it executed in a non-analysis environment and revealed it malicious intent [19, 23]. Therefore, the data captured by Peekaboo was used in this research [24]. Peekaboo is an automated DBI tool that defeats 97 evasive techniques variously used by malware, then captures the malicious behaviour of the malware by executing each sample and recording every Assembly (ASM) instruction that is executed by the CPU [3]. Peekaboo outperforms static and dynamic analysis tools by forcing the malware to reveal its malicious intent and then extracting it’s authentic behavior [3].

Research employing ASM instructions with AI models has focused on Binary Code Similarity Detection (BCSD) tasks [9, 25]. BCSD refers to the task of identifying similarities between different binary files based on ASM language rather than their source code. This is particularly important in the fields of software analysis where understanding the similarity between binary files can help in analysing software evolution and detecting code plagiarism [9, 25]. Understanding the contextual information embedded in binary files presents significant challenges for several reasons. First, the structure and content of binaries, which are semantically equivalent, can differ depending on factors like obfuscation methods, compilers, optimizations, and architectures. Second, the complex compilation process often strips away semantic details, such as variable names, data structures, types, and class hierarchies. Finally, demonstrating the functional equivalence of any two arbitrary programs is extremely challenging [9]. Similarly, sophisticated and novel malware detection techniques could leverage contextual meanings extracted from binary code, enabling the inference of code semantics from syntactically distinct samples.

Fundamentally, the challenge is deducing previously unseen malicious behaviours based on contextual information and code semantics. That is, can an AI model effectively classify functions it encounters for the first time, as benign or malicious.

Considering the limitations of novel malware detection, the central question this research seeks to answer is how authentic behavioural data can be used with a Transformer model to detect novel malicious functionality.

In this section, we provide background information on Transformer models, the Peekaboo DBI data and Zipfs law.

Research studies in this field that have used ASM instructions with Transformer models have focused on BCSD tasks, such as identifying vulnerabilities and detecting software plagiarism. Further, API call sequences and features derived from static analysis have been used with Transformer models for malware detection. These tasks necessitate extracting contextual meanings from binary code. This section provides a summary of recent research and their focus, where Table 2 summarizes the contributions.

PalmTree [35] used an encoding technique with a small BERT assembly language model and assessed its performance by benchmarking it against other encoding techniques such as Instruction2Vec, word2vec, and Asm2Vec. Intrinsic evaluations covered outlier detection and basic block search. PalmTree consistently outperformed baseline schemes in both tasks, demonstrating its ability to effectively capture semantic differences in opcodes and operands, highlighting its capability to model assembly language effectively. Extrinsic evaluation assesses how well an instruction embedding model serves as input for downstream learning tasks, focusing on specific applications in binary analysis. Three key tasks were chosen for evaluation, BCSD, Function Type Signature Analysis (FTSA), and Value Set Analysis (VSA). Despite significant differences between training and testing datasets, PalmTree consistently demonstrates strong performance in BCSD and outperforms other schemes, indicating its enhanced generalisation across diverse datasets and compilers [35]. FTSA refers to the process of identifying and categorizing the types of functions based on their signatures or prototypes within a software system. A function signature typically includes information such as the function’s name, return type, and parameter types. PalmTree outperformed baselines, including Instruction2Vec and Asm2Vec, highlighting its ability to accurately represent FTSA across diverse datasets generated by different compilers. [35]. In practical terms, VSA helps in understanding and predicting how data is accessed and manipulated within a program, which is crucial for tasks such as optimizing code performance, detecting memory-related bugs, understanding program behavior, and improving software security. PalmTree consistently outperformed the original DeepVSA and other baseline models, particularly excelling in categorizing global and heap memory references [35].

BinShot was presented by [9], a BERT based Siamese architecture for BCSD. IDAPro was used to statically disassemble the binary files, which did not use obfuscation, and convert the ASM into normalised functions using the rules proposed by DeepSemantic [25]. This approach could not be used for malware, which is heavily and widely obfuscated [14, 12, 2]. Further, after the normalization process two different functions may be mapped into the same normalized function and they were excluded from the training data [9]. NSP is excluded from their BERT model as function invocations define the relationship between functions in binary code, not the sequential order [9]. BinShot uses a BERT model to embed the normalised functions and uses a Siamese neural network to compute distances between function pairs based on these embeddings, and trains a binary classifier to predict function pair similarity using a weighted distance vector and binary cross-entropy loss [9]. BinShot outperforms other BCSD models, including Gemini, Asm2Vec, PalmTree and DeepSemantic in predicting the similarity between two functions [9]. However, it should be noted that after the function normalization the binary corpus contained 18,449 tokens.

DeepSemantic is an instruction normalisation process with pre-trained and fine-tuned BERT models for BCSD [25]. An ASM language instruction normalization process is essential for use with neural networks [25]. Prior methods varied widely in how they simplified instructions, with some stripping away too much detail like immediate values, while others broke down instructions into excessively granular tokens, which can lead to difficulties in embedding, understanding semantics and Out Of Vocabulary (OOV) problems [25]. The proposed solution seeks a middle ground by normalizing instructions to accurately reflect their semantics while managing the number of distinct tokens. This approach ensures that important details such as immediate values, that can represent jump targets or call destinations, and register sizes are preserved appropriately. It also maintains the integrity of memory access patterns and pointer expressions [25].

Various deep learning models for highly imbalanced multiclass malware classification based on API calls have been evaluated [36]. The models were evaluated using AUC and F1-scores due to the imbalance in the four datasets that were used. The main findings revealed that a Transformer model with one transformer block achieved slightly better results than the Long Short Term Memory (LSTM) model [36]. Additionally, pre-trained BERT and CANINE models outperformed the one-block Transformer architecture. However, Transformer and LSTM models were noticeably faster in both training and inference compared to BERT and CANINE. Despite this, the proposed bagging-based random transformer forest (RTF) model, an ensemble of BERT or CANINE models, proved to be more practical, considering that training time does not directly impact response time and the differences in inference time were minimal [36].

MalBERT is a model built on BERT that conducts static analysis of Android application source code. It uses pre-processed features to identify and classify malware into various categories [37]. The Manifest.xml file was processed and a text based feature representation was used [37]. The BERT based model achieved a high accuracy, surpassing other baseline pre-trained language models in both binary and cross-category classification tasks [37]. The experimental results showed a peak binary accuracy of 97.61% and a multi-class accuracy of 91.02% [37]. This indicates that using Transformer-based models with text input for feature representation is highly effective. Further, pre-trained Transformer models can outperform traditional Recurrent Neural Network (RNN models), such as LSTM, in cyber security applications, especially when applied to datasets like Androidzoo [37].

EarlyMalDetect utilizes a fine-tuned GPT-2 model that analyzes API calls to predict the subsequent API functions for the early detection of malware attacks [38]. The proposed approach focuses on sequence prediction using Transformer models, bidirectional Gated Recurrent Unit (GRU), and fully connected neural networks. The results show the model is highly effective in detecting and classifying unknown Windows malware [38]. However, the authors note that the dataset was small and the fine-tuned model can generate irrelevant sequences [38].

SeMalBERT is a semantic based model that uses API call sequences and BERT, Convolutional Neural Network (CNN) and LSTM deep learning models [39]. Static analysis was used to extract API function call sequences from the malware binary files. The BERT model then pre-processes these sequences, accounting for semantic order and differences in API function calls across various malware types. The output from the BERT model are subsequently used as input to train the CNN LSTM model for malware detection. The results demonstrate that SeMalBERT is less affected by transformations in the malware achieving an accuracy of 98.81%, that surpasses previous API function call approaches [38].

MALSIGHT is a framework designed for generating summaries of binary malware by analyzing both malicious source code and benign pseudocode [40]. Two types of summaries are constructed, MalS and MalP, using a Transformer model and refining them through manual interaction. The core of the framework, MalT5, is a Transformer-based code model trained on these datasets. During the testing phase, MalT5 iteratively processes pseudocode functions to produce detailed summaries, enhancing understanding of code structure and function interactions [40]. MalT5, with 0.77 billion parameters, achieves performance comparable to larger models like ChatGPT3.5, demonstrating the framework’s efficiency and effectiveness [40].

BERT, RoBERTa, XLNet and GPT-2 base transformer models were fine-tuned with labeled Peekaboo ransomware and benign data, as detailed in Table 1, and evaluated on their ability to classify never before seen functionality as malicious or benign. Fine-tuning adapts the pre-trained BERT model to perform more effectively on specific tasks by adjusting its parameters through additional training on task-specific datasets [27].

A simple feedforward neural network and classification layer were appended to the pre-trained BERT model, and the combined system was trained end-to-end using the labeled Peekaboo data. The classification layer uses a softmax activation function and this provides probabilities for each class, enabling the model to make predictions. Softmax converts the output into probabilities, where each output node represents the probability of the input belonging to a particular class. During prediction the model takes new input data, processes them through the layers, and outputs a probability distribution over the two classes, that is benign or malicious. The class with the highest probability is chosen as the predicted class for the input.

GPT-2 is primarily designed for generative tasks such as text generation, story completion, and question answering [32]. A linear layer was used to transform the output from the GPT-2 model into logits. Logits are unnormalized scores assigned to each class and reflect the model’s confidence in the predictions. They are computed directly from the linear transformation of the embeddings and are not scaled to a probability distribution yet. This step is pivotal in the classification process because it allows the model to learn discriminative patterns from the input data. During prediction, these logits are processed further with an argmax function, to obtain probabilities across the two classes. The class with the highest probability is then chosen as the predicted class by the model.

The ASM instructions captured by Peekaboo for both the malware and benign samples follows Zipf’s law which describes word frequency distribution in natural languages. Consequently, our approach involved extracting the ASM instructions and operands from the Peekaboo reports and variously engineering the data into the equivalent of words and sentences and then training custom tokenizers, as detailed in Tables 3, 4, 5. In this way an ASM instruction corresponds to a word and functions are equivalent to sentences. Every sample in the training dataset was processed into its constituent functions that were labelled accordingly, that is benign or malicious. At this granular level, where we examine ASM instructions, the ransomware samples contain functions that are also found in the benign samples. Consequently, any function present in the malware samples that also appeared in the benign samples was excluded from the malicious dataset.

Various Transformer models were trained with the different feature engineering approaches. The test data consisted of several complete ransomware families and benign samples that were not used in training. Given the main objective of this research was novel ransomware detection, any function present in the test samples that also appeared in the training data was removed from the test samples. Consequently, for the model to make the correct classification it had to infer what ASM instruction combinations and context were malicious and which were benign. The classification layer predicted the number of benign functions and malicious functions for each sample. The output layer performed the final classification for the test samples by utilizing a Support Vector Machine (SVM) hyperplane, which acts as the decision boundary to delineate between benign and malicious samples, as detailed in Figure 2.

To determine the effectiveness of Transformer models with the Peekaboo DBI data two experiments were performed. Experiment A used various Transformer models, the standard tokenizers, and the normalized functions that were not concatenated. Experiment B used various Transformer models, custom tokenizers, and concatenated normalized functions. The train test samples are detailed in Table 6 and the hyperparameters for the various transformer models are shown in Table 7. For more detail please see the source code at [26].

The main objective of this research was to evaluate the Transformer models, using varied feature engineering approaches, on their ability to detect novel ransomware. Consequently, any function present in the test samples that also appeared in the training data was removed from the test samples. As a result, for accurate classification, the model was forced to determine which combinations of ASM instructions and their contexts were malicious and which were benign.

In Experiment A the SVM hyperplane intercepted the x axis within the range of data, this is because there was only one benign sample with 100-1000 functions. The dataset for Experiment B was re-balanced to include benign samples with 100-1000 and 1000+ instructions in the test set which resulted in a more reasonable separation between classes.

After de-duplication and filtering several test samples were left with fewer than 10 functions. Specifically, there were 2 Lockbit samples where after that removal of previously seen functions only 3 functions remained. The only 2 models that correctly classified those samples as malicious were DistilBERT 256 CustomTokenizer and GPT-2 256 CustomTokenizer. These two models also demonstrated the highest performance overall, where only one sample was misclassified. The benign sample 262aa82cfd43744433d44228d12c28b322970afd382587ad82f6bb07b903a501, named googleupdatebroker.exe, was flagged as malicious by both of these models and 5 out of the 6 models in total. Consequently, we investigated it further. This file interacts with numerous registry keys and imports several suspicious APIs, including: TerminateProcess, IsDebuggerPresent, UnhandledExceptionFilter, GetCommandLineW, GetProcAddress. It also employs the SetUnhandledExceptionFilter function, which is commonly used as an anti-debugging technique, and queries the system time. These characteristics, combined with the file’s overall behavior, raise significant concerns about its potential malicious intent. Although it is labeled as benign, the observed features suggest that it might not be a false positive.

A comparison to other approaches is shown in Table 14. Pulse outperforms the others in several key areas. Our method uniquely detects truly new and previously unseen samples and achieves the highest accuracy. There is often little difference between ransomware variants, typically using slight variations on the file enumeration and encryption algorithms. Therefore, if a model has been trained on a variant of a sample present in the test data, it should be relatively easy for the model to correctly classify the test sample. Uniquely, our approach removes any familiar functions from the test samples, forcing the model to infer malicious behavior based solely on context and new ASM instruction combinations.

Various Transformer models were trained with several feature engineering approaches using Peekaboo DBI data. The test data consisted of several ransomware families and benign samples. One of the main objectives of this research was novel ransomware detection, therefore any function present in the test samples that also appeared in the training data was removed from the test samples. Consequently, for the model to make the correct classification it had to infer what ASM instruction combinations and context were malicious and which were benign. The function classification layer predicted the number of benign functions and malicious functions for each sample. The output layer performed the final classification for the test samples by utilizing a Support Vector Machine (SVM) hyperplane, which acts as the decision boundary to delineate between benign and malicious samples. Our experimental results demonstrate the effectiveness and transferability of Pulse, which is robust to detecting previously unseen malicious functions. Pulse outperforms the previous state-of-the-art approaches for novel ransomware detection. Unlike other models that may use familiar functions across train and test samples, our approach removes these, forcing the model to detect malicious behavior based solely on context and new ASM instruction combinations. To the best of our knowledge this is the first research to use ASM language with Transformer models to detect never before seen malicious functions and samples.

In future research we will focus on extending the techniques presented in this research to the other types of malware available in the Peekaboo dataset, including Worms, Spyware, Trojans, Botnets, APT and post exploitation tools, each with distinct objectives and functionality [3]. A critical aspect to the highly accurate models presented in Experiment B is the high quality data captured by Peekaboo. In addition to the semantic information captured in our feature engineering approach, the Peekaboo data also incorporates temporal information and complete network traffic captures and this data could be leveraged in future work. In terms of a real world application of the techniques presented in this research, a limitation is the time it takes Peekaboo to analyse a sample and extract the ASM instructions, in future work we will determine how much information is required by the Transformer models to make an accurate classification.

The fine tuned Transformer models and associated scripts are available in the Peekaboo Transformer Models repository [26].

Matthew Gaber conceived of the presented idea, carried out the experiments, wrote the main manuscript text and prepared the figures. Mohiuddin Ahmed and Helge Janicke helped supervise the project. All authors discussed the results and contributed to the manuscript.