On the Role of Similarity in Detecting Masquerading Files

Similarity has a wide range of uses in computer security. We can determine that a security object is similar to a known good entity or to a known bad entity. Similarity has been applied to a wide range of security objects including applications in spam filters, detecting phishing pages and detecting malware variants.

We use similarity hashes as our approach for similarity as SSDEEP [11] and TLSH [20] are available in public repositories of malware [9] and have been adopted by the STIX and MISP standards [21, 10]. Similarity hashes are typically combined with clustering approaches to form a cluster model of the data [23, 19]. An idealized framework for using similarity and clustering in security works as follows:

1. (a)

   A training set of (partially) labeled data is clustered according to the distance function.

2. (b)

   A reputation is assigned to each cluster based on its membership.

3. (c)

   New security objects can be classified by finding the most similar cluster according to the distance function being employed¹¹1 Issues of scale can be address by tools such as Approximate Nearest Neighbor (ANN) [2, 1].. The naive approach for using such a machine learning solution is to make these predictions by selecting the most common category from the closest cluster.

For most problems in the computer security domain the above process suffers from a fundamental problem. Let us assume that the clustering worked as intended and created a clear separation of the clusters. Bad actors are known to use a wide range of tricks to make malicious content similar to legitimate content [18]. We shall term such a malicious sample as being a masquerading sample. Some small number of clusters will contain members created by bad actors which are masquerading as legitimate samples.

The masquerading samples potentially cause serious problems for machine learning solutions. This is made worse by the fact that security datasets are typically only partially labeled [5], so we expect many clusters to have members which are missing labels and some masquerading samples to be un-labeled. The problems arise when a masquerading sample is included in an otherwise legitimate cluster. Some of the problems include:

1. 1.

   Bad actors can circumvent machine learning solutions by using masquerading samples. The naive approach for prediction will assign a legitimate category for a masquerading sample.

2. 2.

   A cluster with masquerading members may be labeled as malicious, resulting in false positive predictions.

3. 3.

   If a cluster with undetected masquerading members is used to form the baseline of legitimate activity, then the baseline will include malicious behaviours.

Problem 1 is the most serious issue. We note that it only applies if we use the naive approach for prediction. An extreme solution would be to configure the machine learning to avoid making predictions for clusters which contain both masquerading members and legitimate members. This extreme solution is unsatisfactory as it means that the model will fail to make predictions for important clusters²²2 We will see examples of files masquerading as Microsoft and Google in the Section 3..

Traditional security employs digital signatures for applications such as signing executable programs [3] and authenticating email [4] to detect malicious attempts at masquerading. In this paper, we explore the issue of the interplay of machine learning with similarity-based methods and digital signatures. While there is a wealth of academic literature on both sides of this interplay, there is virtually no discussion on how these technologies should be applied together to produce results.

To explore this issue, we focus on executable files and concentrate on the interplay of code signing with machine learning. The contributions of this paper are:

• •

  We give a taxonomy for masquerading files.

• •

  We show how a similarity-based system can be used to find real world masquerading files in a public database of malware. While these real-world masquerading samples are in the minority, they confirm the problems raised here.

• •

  We offer methods for including digital signatures in ML and similarity-based solutions.

There is a range of methods available to bad actors for creating masquerading files. The Mitre Attack Framework [17] includes 7 types of masquerading under T1036 [18] of which 6 are relevant to executable files.

• •

  Syntactic Masquerading: These attacks focus on tricking end-users that the file is legitimate. They include renaming files to legitimate filenames (Mitre T1036.003), obfuscating filenames with spaces (T1036.006), and obfuscating filenames with double file extensions (T1036.007). These methods may also evade security rules and methods which rely on surface level features (such as filenames) or when digital signatures are not available (for example with unsigned files).

• •

  Content Masquerading: This involves inserting malicious content into an existing file (often signed). When malicious content is inserted into a signed file, Mitre terms this ”Invalid Code Signature” (T1036.001). The Neshta malware family takes existing executable files and inserts malicious content [13]. If the file was signed, then it will leave the original X509 certificate in the executable and the new file will no longer have a valid signature.

• •

  Certificate Attacks: This is the situation where an attacker compromises the digital signature [14] or the part of the OS that checks the digital signature [6].

• •

  Supply Chain Attacks: This is the situation where an attacker compromises the source of software [15, 16, 8].

• •

  Adversarial ML: This is an area of Machine Learning where a sample is created to trick a ML system into making an incorrect classification. This approach has proven effective at tricking image recognition systems [22, 12] and has proven effective for deceiving a ML security solution on executable files [7].

For the remainder of the paper, we focus on content masquerading, certificate attacks and supply chain attacks. We will not study Syntactic Masquerading files as they are often self-evident when inspected and would typically be detected by antispam and antivirus products. At this stage, we have seen no evidence that adversarial ML has been adopted by cybercriminals for the creation of malware.

The first step in performing an analysis of masquerading files is to collect a dataset of them.

One important class of masquerading files is files where the binary content of the file is similar to legitimate software and the file has been classified as malicious. We can use the Malware Bazaar [9] dataset which as of August 2023 has over 700,000 malware samples. It is ideal for research purposes such as this as (i) it is a malware dataset of reasonable size, (ii) all submissions are TLP:WHITE, (iii) it has been indexed by hashes that enable similarity search (TLSH and SSDEEP), and (iv) all files are available for downloading by other researchers.

We applied the following process to get a candidate list of masquerading files:

1. 1.

   Build a cluster model of a large set of customer files. This cluster model is assumed to contain a significant majority of legitimate files. We built this model using the HAC-T algorithm with the TLSH distance function [19].

2. 2.

   Process the Malware Bazaar dataset of malware using the TLSH distance function and identify those files that were within a distance of 30 of a customer-based cluster (most likely legitimate). The distance 30 was used as the threshold to target a false positive rate of $0.002\%$ (see Table 2 of [20]) which would reflect an estimate for the probability of files being incorrectly assigned to a cluster.

The process resulted in 703 of the 700,000 malware samples being flagged as candidate masquerading files.

Table 1 shows the number of files with each signature state. We then categorized the state of the certificate according to a combination of the files signing state, metadata provided by Malware Bazaar and some crowdsourced rules (including ”cert_blocklist” from
https://github.com/reversinglabs/reversinglabs-yara-rules and ”knownbad_certs” from
https://github.com/ditekshen/detection).

We now go through the various cases and give clear cut example where we point back to information on Malware Bazaar. We highlight how analysis with similarity is helping understand the nature of masquerading files. The examples below were created by hand to determine that the files are masquerading. We note that each case here could be automated in a straightforward way to generate alerts or detections for the case in question. Any alerts or detections generated can give reasoning as to the nature of the anomaly discovered.

We believe that the determination that a file is a masquerading file would be very difficult without the additional cluster information. We note that security analyses of files very rarely specify which legitimate file a malicious file is masquerading as.

Another approach for finding masquerading files is to find clusters with inconsistent signature information. We searched through the model for clusters where:

• •

  We had a majority that were signed and verified.

• •

  We had a minority that were unsigned.

This search process found the following example:
Unsigned File SHA256 0a1cfbf61797a565b649d443dcf6102f0e179b68e2582a788a97acf05a4ecd72 TLSH T18E75AE05F951D07AC1162070E41DF3396B345E59CB214ADFE7D87E9A3EB02D12A3A2AF Filename chrome.exe Signer Google LLC (but does not pass signature verification) Notes This file has a X509 certificate claiming it is signed by ”Google LLC” but does not pass signature verification. This file is a version of Chrome which was corrupted by the Neshta malware family [13]. Cluster and Signed Member Details TLSH T10075AE01F850D0B6D5122071F41DF339AA355E198B658EDBE3987E9A3FB02D25A3A39F SHA256 606394cc56a3f5b37dff0540795823be3a78f58dace822855f5557ce1ebb4ffe Filename chrome.exe Signer Google LLC Notes This cluster has 45 members all of which were chrome.exe signed by ”Google LLC”. Distance from File to Closest Cluster: 29

This property of clusters mostly containing legitimate signed files with a small number of unsigned versions is an indicator that the unsigned files may be infected with Neshta.

Another cluster property to investigate is clusters where:

• •

  The members are signed and verified.

• •

  The members differ in file reputation.

This search process found the following example:
Bad Reputation Files SHA256 ce77d116a074dab7a22a0fd4f2c1ab475f16eec42e1ded3c0b0aa8211fe858d6 TLSH T1EA25C60177EC8A09E1FF2B75AAB441280B73F95A9A76D75E294C109E0FB3B008E51777 SHA256 019085a76ba7126fff22770d71bd901c325fc68ac55aa743327984e89f4b0134 TLSH T14F25C60177EC8A09E1FF2B75AAB441280B73F95A9A76D75E194C109E0FB3B008E517B7 SHA256 32519b85c0b422e4656de6e6c41878e95fd95026267daab4215ee59c107d6c77 TLSH T1F525C50173EC8A49F5FF2B74AAB441680B73B8569A7AD74D154C619E0FB3B008E11BB7 Filename SolarWinds.Orion.Core.BusinessLayer.dll Signer Solarwinds Worldwide, LLC Cluster and Good Reputation Member TLSH T10025D54177FC4A09F6FE2B74AAB441190B73B91AAA7AD74E154C209E0FB3B40CE617B7 SHA256 671d9bd3ef6b0ecb0507dda84ed5800238844b4deec5f387ff503835909a8cee Filename SolarWinds.Orion.Core.BusinessLayer.dll Signer Solarwinds Worldwide, LLC Distances from Files to Closest Cluster: 25, 23, 24

The malicious files are a part of the Solarwinds supply chain attack [16]. At the time of the attack using similarity on file content would not be useful to detect this attack. We do note that having the cluster of legitimate files could help establish a baseline for the behaviour of the files within the cluster which may be useful for identifying anomalous behavior during incidence response.

We have seen that a sample of candidate list of 703 files from Malware Bazaar indeed includes masquerading files with a range of sophistication levels. This would indicate that we expect at least 1 in 1000 malware is a masquerading file³³3 This is a lower bound as we should assume that we did not find all the masquerading files..

If a ML or similarity-based solution is being used for security on files, then it is important that the solution can effectively deal with masquerading files. The use of digital signatures should be used as a part of ML solutions. In addition, rulesets which test for stolen certificates, revoked certificates and certificates with a history of signing malware should also be employed. Digital signatures can be either applied as a post-processing step (as done in the various experiments in this paper) or could be added as additional input features.

Special care should be used during the training and testing of ML systems. ML systems should be tested with masquerading files to test whether they are capable of distinguishing between the legitimate versions and masquerading versions. One approach to this may be to ensure that if legitimate software forms a part of the training set, then it is important to include masquerading versions in the malicious part of the training set.

In conclusion,

• •

  Machine Learning and/or similarity should play an important role in security operations and security products for the identification of masquerading files.

• •

  ML solutions in security need to consider how to deal with masquerading samples. Specifically, ML solutions for files need to be able to deal with masquerading files; one approach for this is to tightly couple the ML with a digital signature solution.

• •

  Digital signatures are not a panacea. Many files are not signed, and further research needs to establish how ML solutions can effectively operate in the presence of masquerading samples for unsigned files. We encourage all software producers to adopt the use of digital signatures.