A Study on the Importance of Features in Detecting Advanced Persistent Threats Using Machine Learning^††thanks: Submitted to CSCI–RTCW 2024.

APTs are targeted, intelligence-driven, and complex cyber-attacks launched against high-value targets such as critical infrastructures and financial institutions [3]. APTs are multi-step attacks, usually starting with intelligence collection through reconnaissance. Based on the security holes identified during reconnaissance, APT gains access to the system and exploits the system. To ensure continuous access to the targeted system, APTs often inject backdoors into the system so that they can revive the connection when desired. From there, attackers explore the network to extend their access to more sensitive components of the system. Finally, the attacker reaches a malicious objective such as hijacking sensitive data or disrupting critical infrastructures.

Several APT cases are considered in this study. The name of these cases indicates the hacker group responsible for them. These cases are briefly introduced in the following.

To simulate real-world APTs in computer networks, we consider the data introduced in [16]. This data is created based on known APTs with publicly available reports. These data contain 2086 APT samples obtained from 15 different APTs, as listed in Table 1. A total of 9021 non-APT malware are also included in the data. The objective of this case study is to distinguish between APT and non-APT attacks so that malware triage can be performed accordingly. Table 1 also indicates the population of samples associated with each APT mechanism. To minimize the latency, samples are recorded using static features that can be extracted with minimal delay. Initially, 4000 features were extracted using the PEFrame tool. These features consist of optional headers, MS-DOS headers, file headers, obfuscated string statistics, Mutex, Packer, buckets, and imported API.

Experiments are evaluated in terms of accuracy, F1-score, precision, and recall. To ensure the statistical reliability of the results, experiments are cross-validated using 10 folds. The parameters of FS methods are tuned using Grid Search with a search range obtained empirically. Experiments were conducted using an NVIDIA RTX 3080 GPU, an Intel Core i7-12700 CPU, and 32 GB of RAM.

Fig. 2 illustrates the box plot of post-FS detection performance in terms of four different performance metrics. The distribution of recorded performance across different iterations of the cross-validation is shown by solid circles beside each box. In addition, the average and median of this distribution are indicated with a solid square and a red line, respectively. Furthermore, plus signs denote outliers in the plotted distribution. Maximum and minimum values are specified using green lines.

To study the post-FS detection performance, we select 110 features from the original pool of 3000 features for each method. The detection task is carried out using three classifiers, namely decision tree, random forest, and k-nearest neighbors. Throughout the experiments, the recorded performance metrics obtained by these classifiers are averaged to evaluate the performance of each FS algorithm.

Looking into the obtained accuracies in Fig. 2(a), it can be implied that coupling classifiers with FS results in outstanding APT detection performance. Among the selected FS methods, CFMI generally outperforms the rest regardless of the performance metric. RFE, CAE, MI, GDS, SFS, and RliefF are ranked from second to seventh in terms of accuracy. Nevertheless, given the high imbalance ratio in the utilized dataset (see Table 1), accuracy may accurately reflect the detection capability of this approach. Therefore, the F1-score of these methods is reported in Fig. 2(b) to better understand the potency of FS when dealing with highly imbalanced data. While the obtained F1-scores are desirable, the ranking of FS algorithms has changed. Here, CAE outperforms RFE, and ReliefF results in a better F1-score than GDS.

In order to take false positives and false alarms into account, precision and recall associated with APT detection are additionally reported in Fig. 2(c) and (d), respectively. Similar to the previous analysis, the obtained results in Fig. 2(c) and (d) show superior performance for all FS techniques. The comparison of FS methods in terms of precision is very similar to that of F1-score. On the other hand, the ranking of methods when taking recall into consideration, is roughly the same as the comparison performed on the detection accuracy. This implies that the reason for the different ranking of algorithms when considering accuracy and F1-score is the degree of false alarm and false positive each of them leads to.

Fig. 3 depicts the top 73 features selected by the majority of FS algorithms selected in this work. These features fall under different categories that relate to various aspects of binary’s behavior and structure. Studying these categories helps with understanding how they contribute to detecting APTs within the network. Here, we categorize these features into three groups based on the level of priority.

The top nine features selected by all FS methods form the core of the detection strategy. These include winsock.dll, shell32.dll, and advapi32.dll Dynamic Link Library (DLL) imports and structural attributes such as MinorLinkerVersion and AddressOfEntryPoint. Because the features are present in all the selection procedures, it can be inferred that they are important aspects of APT?s behavior. As an example, wininet.dll and shell32.dll are frequently used by malicious programs for network communication and shell interaction. Advapi32.dll is used for performing registry and security operations which makes it a primary target for hackers trying to bypass and manipulate security protocols. AddressOfEntryPoint and MinorLinkerVersion are associated with the compilation and loading process of binaries, where abnormal alterations of the features may indicate the presence of malicious activities. These features suggest that both functional aspects and underlying structure indicators are beneficial in detecting malicious traffic associated with an APT.

The subsequent group of 21 features comprises, among others, additional DLLs and some functions that are substantially involved in the executable behavior. These include msvbvm60.dll, mscoree.dll, and ws2_32.dll. Although these features are rather less important than the top group, they still provide opportunities to ascertain an APT presence. In addition, such features facilitate identifying the areas, resources, or libraries, that the binary code might communicate with. These features are also useful for operations that might be conducted at the system level. For instance, msvbvm60.dll is particular to the visual basic programs, and therefore its presence could indicate the usage of outdated programs that are security risks. mscoree.dll is used for .NET programs, and it is important to track since viruses may use the .NET runtime to facilitate their operations. The presence of ws2_32.dll indicates possible networking operations, which can further be investigated for C2 communication or data exfiltration. Although not as central as the first nine features, this group is critical in narrowing down the broader depiction of what an executable does, in particular, the network-based-executables and managed code execution.

The rest of the features in Fig. 3 are selected by five methods. These features are helpful in improving the detection system. Some of these are modemui.dll, hal.dll, ntdll.dll, as well as type and structure imports and function imports. These features seem to represent behavioral patterns that are more specific to a context or target, and may not be present in all APT attacks, but are still useful flags in certain scenarios. For example, modemui.dll and hal.dll belong to the class of hardware control and system management, vulnerabilities where sophisticated malware that aims to exploit even lower-level parts of a computer system may attack. Usage of ntdll.dll also aids in communication with the kernel, and unorthodox usage of this library may point towards alterations made at the executive level of the system catalog. Such features assist in exposing more of the subtle tactics and approaches that may not be noticeable from the previously discussed groups of features. Table 2 lists features that are selected by four or fewer FS algorithms. These features have lower priority compared to the features included in Fig. 3 as they contribute less in detecting APT samples.