A Transformer-Based Framework for Payload Malware Detection and Classification

As network traffic continues to evolve in modern society, the importance of Deep Packet Inspection (DPI) becomes an essential tool for the analysis of network packets and the security of networks. DPI goes beyond analyzing the network five-tuple, which consists of the source and destination IP address, source and destination port number, and the transport layer protocol, but also examines the data payload of each packet. The data payload of each packet consists of the content or information flowing through the network and processed by the respective end hosts. DPI dives into the actual content of data packets to extract valuable insights to identify threats or anomalies. The importance of DPI lies in the fact that adversaries can modify their MAC addresses or employ third-party devices, such as cloud services, to transmit malicious packets. These alterations make it challenging to identify such packets using only the traditional five-tuple approach. By thoroughly examining the payload of each packet, DPI can help distinguish between malicious or benign payloads, as well as the types of cyber attacks. Malicious payloads may include malware, viruses, or phishing attacks, while benign payloads include legitimate data and information exchanged between the network. Several research papers have contributed significantly to the field of DPI by proposing various approaches and methodologies.

In [1], researchers describe an overview of the three different implementations of DPI. 1) Signature-based Identification: Relies on comparing the signatures (port numbers, string patterns, or bit sequences) of packets with known signatures in order to identify the associated application. Each application is associated with a specific signature that may include port numbers, string patterns, or bit sequences. By matching the signatures, DPI can recognize the data flow and determine the corresponding application. 2) Application layer-based Identification: Particularly useful for applications with distinct control and service features, this method zeroes in on the application gateway via the analysis of control and relative service flows. 3) Behavior-based Identification: Employed when data flows cannot be recognized by any known protocol. Instead, DPI analyzes user behavior or specific terminal characteristics to make judgments. For example, certain behaviors can be used to identify and filter out spam emails. The flow and relative service flow are examined to define the specific protocol associated with the behavior. Machine and deep learning models tend to fall under the category of behavior-based identification since they are designed to identify patterns and behaviors within the data.

Aceto et al. [2] discuss the intersection of deep learning on mobile traffic classification. The authors discuss how deep learning models offer the potential to handle network traffic without relying on port information and can effectively distinguish between traffic generated by different applications. In [3], authors employed various techniques, such as self-taught learning and soft-max regression. The study used the NSL-KDD as a benchmark dataset. Supervised learning techniques were used on over 41 statistical features, where none of the features were strict payload bytes. Doshi et al. [4] leveraged machine learning for identifying Distributed Denial of Service (DDoS) attacks. Four machine learning models and one simple neural network were used to distinguish between normal Internet of Things (IoT) traffic and DDoS attacks. The paper focused on creating a novel dataset to test the algorithms and resulted in a large sample size of unbalanced malicious and benign samples.

In [5], machine learning techniques are leveraged to classify IP traffic on a 4G network. These researchers also generated their own dataset on the 4G network and applied common machine learning algorithms to the packet content of the dataset. The paper does not mention testing on malicious traffic, but rather the IP traffic being transmitted through the network. In [6], the authors applied machine and deep learning techniques to detect DDoS attacks on a sample dataset. None of the included features included raw payload bytes, but instead variables such as total number of packets, bytes, and total duration of packet transaction.

DPI faces several current limitations that impact its effectiveness and performance. DPI is known to be resource-intensive since analyzing packet data requires substantial computational resources, especially since most networks have high volumes of traffic being transmitted through it. Thus, the state-of-the-art DPI algorithms limit the inspection of payloads to the initial bytes of packets. By only inspecting the initial portion of a packet’s data, a system may overlook threats hidden deeper within the payload. Another challenge that DPI is currently facing is the growth of encrypted traffic being transmitted over networks. It is estimated by security researchers at Sophos that nearly 46% of all malware in 2020 was hidden within an encrypted package [7]. This challenge limits the ability to preserve privacy while inspecting payloads and requires parties to share their encryption keys with them to decrypt payloads, perform inspection and encrypt packet contents.

The main contribution of this paper is the introduction of an algorithm for malware detection and classification based on transformers [8]. The proposed architecture capitalizes on the self-attention mechanism of transformers to capture the intricate patterns and dependencies present in the raw bytes of the network packet payloads, rather than relying on statistical-based features from packets. The payload contains the actual content of the network packet and would likely hold discernible patterns or signatures of malicious activity, while other information, such as the packet headers, are only meant for the transmission and management of data over a network and primarily contain addressing and protocol information. The proposed approach achieves not only enhanced accuracy in identifying malicious payloads, but also pushes the boundaries of current methodologies that help distinguish malicious payload types by employing two different classification heads.

In this section, we describe the architecture, training, and evaluation of our model for deep packet inspection.

In our study, we focus on the raw payload bytes within TCP and UDP network packets from the discussed datasets. However, it is crucial to note the limitations when dealing with encrypted traffic. Cryptography–the process of encrypting the data–secures the transmission of data flowing through a network, making the data unreadable to unauthorized users without the proper keys. When the plaintext is encrypted with a key resulting in the ciphertext, the ciphertext gives no information about the plaintext [15]. Let’s assume $m$ is the random variable representing the raw payload message and $E_{k}(.)$ is the encryption algorithm that encrypts the message with the key $k$. Then, knowing the ciphertext $c_{1}$ reveals absolutely no information about the plaintext $m_{1}$: $p(m=m_{1}|E_{k}(m1)=c_{1})=p(m=m_{1})$. In plain words, every time the plaintext $m_{1}$ is decrypted with the same cryptography algorithm $E_{k}(.)$ and the same encryption key $k$, it results in a different ciphertext, thus revealing no information about the plaintext [15]. While several studies have been able to classify the traffic of different applications from the cipher text [16] [17], their success is attributed to the fact that different applications use different random generators for encryptions which appears as a signature in their encrypted payloads but still reveal no information about the plaintext [18]. On the other hand, malware detection algorithms (including the proposed algorithm in this paper) requires access to the information of the plaintext that cannot be revealed by the ciphertext. If any algorithm is able to detect the signature of malware from the ciphertext, it means that the encryption algorithm is not strong enough to hide the information about the plaintext which means $p(m=m_{1}|E_{k}(m1)=c_{1})\neq p(m=m_{1})$. We have conducted several experiments to demonstrate this.

We implemented both Advanced Encryption Standard (AES) encryption [19] and Fernet symmetric encryption [20] on the payloads. AES takes the raw payload data, a 256-bit key, and a 16-byte initialization vector (IV) to produce the encrypted output. Each key and IV combination ensures the uniqueness of the encryption process. Fernet uses AES in Cipher Block Chaining (CBC) mode with a 128-bit key for encryption. After encrypting the payloads, similar procedures discussed in the methodology above were implemented. Each encrypted payload, in hexadecimal format, was converted to a decimal integer sequence and then padded to ensure uniform length for each input.

These inputs were then trained and tested in a similar manner as above with the same architecture. The AES cryptologic algorithm results showed that the model was not able to effectively distinguish between malicious and benign payloads, with a test accuracy of 57.16% and an F1-Score of 44.52%. This demonstrates that encryption algorithms like AES successfully diminish the learnable patterns in the raw bytes, which our model relies on to classify the payloads. However, the Fernet algorithm produced a test accuracy of 91.41% and an F1-Score of 92.09%. This experiment shows that certain encryption algorithms may not be strong enough to hide the plaintext information, while others can.

To evaluate the performance of the proposed method, several metrics are taken into consideration. For this study, we consider accuracy, precision, recall, and F1-Score on the test dataset. It is important to note that the test dataset used in our evaluation process differs from the training dataset. The test dataset consists of randomly selected, unseen samples and ensures that the model is using a distinct dataset to evaluate the performance.

The proposed method is compared against other state-of-the-art models [21] [22]. Our primary intention behind comparing a transformer model with these deep learning algorithms lies in evaluating different sequence processing architectures. While all of these models fundamentally process sequences, their mechanisms are distinct. Convolutional Neural Networks (CNNs) capture localized patterns and hierarchical structures in data. This is valuable when analyzing patterns emerging from chunks of network packets. Long Short-Term Memory networks (LSTMs) utilize recurrent connections to remember patterns over long sequences. However, our transformer-based model is designed to comprehend and capture context over a wider range without the recursive nature of LSTMs. The self-attention mechanism allows the model to weigh significance of different parts of the sequence, providing a global understanding of data.

Our study differs in several aspects. We first eliminated duplicate payload values, which allowed the model to analyze unique payloads, increasing the data quality to better generalize to unseen data. Additionally, our input was not limited to the inital 784 payload bytes as seen in [21], but used the maximum of 1460 bytes. The typical Maximum Transmission Unit (MTU) which can be sent through a packet-based network is 1500 bytes, minus 40 bytes for the header information. The header data was omitted to reduce bias that may be introduced by the networking addresses and protocol details. By utilizing strict payload bytes, we ensured our model had access to all available information within each payload, which is crucial since malicious data may be hidden deeper into the payload of some packets. The limitation of discarding portions of byte data is addressed, showing a more global nature of the complexity of each payload. The evaluation results for the classifiers can be found in Tables II and III. We assessed the performance of these classifiers under two conditions: binary classification and multi-class classification tasks.

This paper explored the application of a transformer-based model for malware detection and classification. The model was evaluated on the UNSW-NB15 and CIC-IOT23 datasets, focusing on the payloads of UDP and TCP packets serving as inputs. Our method produced robust results on the classification of benign versus malicious packets when compared to state-of-the-art methods. Using the payload bytes as input vectors, we were able to classify when different packets are either benign or malicious, as well as which types of attacks were present. While the payload bytes of packets are significantly different from the natural language structure, this study shows that the transformer-based model developed for natural language can be leveraged to capture and learn the intricate sequential patterns of the payload bytes.

This work is partially supported by the UWF Argo Cyber Emerging Scholars (ACES) program funded by the National Science Foundation (NSF) CyberCorps® Scholarship for Service (SFS) award under grant number 1946442. Any opinions, findings, and conclusions or recommendations expressed in this document are those of the authors and do not necessarily reflect the views of the NSF.