A Survey on Adversarial Attacks
for Malware Analysis

In this paper, we structure our survey in a hierarchical manner as shown in Figure 3. Section 6 is placed before section 5 in the Figure 3 just to manage space but actual order in the paper is in incremental order of section number. We begin our survey, as discussed in Section 1, by introducing the field of adversarial machine learning along with motivation for the need to study adversarial attacks in the malware analysis domain. Note that Table II provides acronyms that are used frequently in the survey. Section 2 discusses different machine and deep learning algorithms that are used in state-of-art adversarial research. Understanding key concepts of machine learning prerequisites provides the readers the appropriate background to grasp adversarial generation techniques discussed later in the survey. Section 3 explains the structures of Windows, Android and PDF files. The structure of files plays a key role in apprehending adversarial attacks as perturbation depends on flow and robustness of files’ structure. Section 4 provides an introduction to malware detection approaches against which adversarial attacks are designed. Section 5 models the adversarial threat from different dimensions. This section briefly elaborates on attack surface, attacker’s knowledge, attacker’s capabilities and adversarial goals. Section 6 discusses various adversarial algorithms that are considered as standard techniques for perturbation generation across different domains. Section 6 taxonomizes existing real adversarial attacks based on the execution domains (Windows, PDF, Android, Hardware, Linux) and algorithms maneuvered to carry out attack. This section discusses real attacks carried out against malware detection approaches in detail and provides comparisons among related works. Section 8 highlights challenges of current adversarial generation approaches and sheds the light on open research areas and future directions for adversarial generation in malware analysis. Finally, Section 9 concludes our survey.

To discover the relevant state-of-art works and publications in adversarial attacks on malware analysis, we relied on different digital libraries for computer science scholarly articles. Our major sources are IEEE Xplore¹³¹³13https://www.ieee.org/, ACM digital library¹⁴¹⁴14https://dl.acm.org/, DBLP¹⁵¹⁵15https://dblp.uni-trier.de/, Semantic Scholar¹⁶¹⁶16https://www.semanticscholar.org/ and arXiv¹⁷¹⁷17https://arxiv.org/. Apart from these digital libraries, we also searched directly through Google¹⁸¹⁸18https://www.google.com/ and Google Scholar¹⁹¹⁹19https://scholar.google.com/ to get impactful papers in domain that were somehow missed in other libraries. Among numerous keywords used to fetch the papers from public libraries, ”Adversarial Malware” and ”Adversarial attacks in malware” gave us the most number of relevant papers. After listing all the published works in adversarial generation between year 2013 to 2021, we filtered out papers with good impact, relevance and prepared the final list to conduct our detailed survey.

Deep Learning (DL) is a sub-field of machine learning that uses supervised and unsupervised techniques to learn multiple level of representations and features in hierarchical architectures. The ability of conventional machine learning was very limited while processing a raw data. Deep learning has been able to make significant breakthroughs for challenges faced by ML practitioners by showcasing its ability to find patterns in very high dimensional data. Deep learning has enabled researchers to reach unparalleled success in fields of image recognition [137, 138, 139], speech recognition [140, 141, 142], neuro-science integration [143], malware detection [144, 145] and most of the ML powered research areas. Since the start, structuring conventional machine learning algorithm required careful feature engineering and high level domain expertise to extract meaningful features from raw data. The effectiveness of machine learning is largely dependent on the representation ability of feature vectors. Representation learning is an approach for ML which allows models to be fed with raw data and automatically learns the representation required to make decisions.

Deep-learning methods are representation learning approach with multiple level of representation, obtained by non-linear transformations from lower to higher abstraction level. By combining such simple, non-linear transformation, machine finally learns complex function. Taking representation to higher level in each step signifies amplifying aspects of input which are important for discrimination while suppressing irrelevant features. It can be observed in Figure 6 that all the feature extraction overhead of traditional learning is replaced by neural nets in deep learning. A standard neural network architecture are made up of connected neurons which are the processors and each neuron outputs a sequence of real-valued activation. Environmental input obtained by sensors activate the input neurons while deeper neurons get activated through weighted connections from previously active neurons. Deep learning operations are usually composed of weighted combination of a group of hidden units having a non-linear activation function, based on a model structure. The architecture of neural network resembles to the perception process of human brain, where a specific sets of unit get activated if it has a role in influencing the output of neural network model. Mathematically, the deep neural network architecture are usually differentiable, so that the optimal weights of the model are learned by minimizing a loss function using variants of stochastic gradient descent through back propagation. For the example mentioned in Figure 6, we can consider an image classification example where image input comes in the form of an array of pixel value [146]. During the first layer of representation, deep learning models learns the presence or absence of edges at particular orientation. Second layer tries to detect some arrangements in detected edges discarding the minute variations in the position of edges. These arrangement of edges are combined into larger combinations, corresponding to the sections of familiar objects and subsequent layer capable of giving the detection results.

Convolutional Neural Network (CNN) is one of the most popular deep learning architecture inspired by natural visual perception of the living beings[147]. It takes its name from mathematical linear operation between the matrixes called convolution. One of the first multi layer artificial neural network (ANN), LeNet-5 [148, 149], as shown in Figure 7, is considered to have established the modern framework of CNN architecture. CNN has received ground breaking success in recent years in the field of image processing [150] which has been replicated to many other fields. One of the biggest aspects behind the success of CNN is its ability of reducing the parameters in ANN.

CNN is mainly composed of 3 layers as shown in Figure 7: convolutional layer, pooling layer and fully connected layer. Convolutional layer aims to learn feature representation from the input raw data. Feature maps are computed using convolutional kernels, with each neuron of a feature map connected to a region of neighbouring neurons in the previous layer. New feature map is received by convolving around the inputs with a learned kernel and applying element-wise nonlinear activation function on the convolved results. During feature map generation, kernels are shared by all the spatial locations of the input. The role of pooling layer is to reduce the number of connections among convolutional layers which in turn helps in reducing the complexity of computation. Pooling layer hovers over each activation map and scales the dimensionality using appropriate functions like max, average and so on. Stride and filter size of pooling defines the scaling. Fully connected layers in CNN have same role as that of standard ANN, producing class scores from the activation. Other common CNN architectures include AlexNet [150], VGG 16 [151], Inception ResNet [152], ResNeXt [153], DenseNet [154].

Reinforcement learning (RL) can be viewed as a learning problem and a sub-field of machine learning [155]. Basically, its about using past experience to enhance the future manipulation of a dynamic system and learning by maximizing some numerical value which helps to meet long term goals. A supervised learning model learns from data and its labels whereas a RL model completely relies on its experience. In RL, a model is trained to make sequence of decisions through the action of agent in a game-like environment. Diagrammatic representation of reinforcement learning is shown in Figure 8 where it is shown as the combination of four elements: an agent capable of learning, the current environment state, an action space from which an agent can choose an action and the reward value that an agent is provided in response to each action. Program is deployed to go through a trial and error process to reach the solution of a problem. Agent acting on an environment gets either a reward or penalties for an action it performs and the goal of learning is to maximize the total reward. The programmer sets up the action space, environment and reward policy required for learning and the model figures out the way of performing tasks and maximizing the reward. An agent learning starts with random trials and errors leading to highly sophisticated tactics and superhuman decision making. In a formal definition, a system governed by machine learning algorithm observes a state $s_{t}$ from its environment at time step $t$. The agent performs action $a_{t}$ in state $s_{t}$ to make transition to a new state $s_{t+1}$. The state is basically the information about environment which is sufficient for an agent to take best possible actions. The best sequence of actions are defined by the rewards provided by the environment while executing the actions. After completion of each action and transition of environment to new state, environment provides a scalar reward $r_{t+1}$ to the agent in form of feedback. Rewards could be positive to increase the strength and frequency of the action or negative to stop the occurrence of action. The goal of an agent is to learn a policy $\pi$ that maximizes the reward. Reinforcement learning faces the challenge of requiring extensive experience before reaching optimal policy.

Exploration and exploitation through all possible directions in high dimensional state spaces leads the learning process to an overwhelming number of states and negatively impact the performance. This had limited the previous success of reinforcement learning [156, 157, 158] to lower-dimensional problems. We have discussed the rise of deep learning in last decade by providing low dimensional representation in previous sections. In its way to solve the curse of dimensionality, deep learning also enabled reinforcement learning to scale to very high-dimensional states problem, which were previously considered impractical. Mnih et al. [136] work to play Atari game using deep reinforcement learning and beating human level experts, easily elevated the application of reinforcement learning in combination with deep learning. An actor-critic model with experience replay was used to reach such performance on the Atari game. In deep reinforcement learning framework, agent acting on end-to-end way, takes raw pixels as an input and outputs the associated rewards for each actions. The learned reward function is the basis for deep Q-learning which keeps refining over the experience. Deep reinforcement learning has already been very successful in fields such as robotics [159, 160, 161, 162] and game playing [163, 164, 165] where learning from experience is very effective, replacing hand-engineered low-dimensional states.

Neural networks has already been established as a very powerful tool to perform in many supervised and unsupervised machine learning problems. Their ability to learn from underlying raw features which are not individually decipherable has been unparalleled. Despite their significant power to learn from hierarchical representations, they rely on assumption of independence among the training and test sets [166]. Despite of neural net’s ability to function perfectly with independent test cases, their assumption of independence fails while data points are correlated in time or space. Recurrent Neural Network (RNN) being a connectionist model, are able to pass information across the sequence steps and processes single sequential data at a time. We can relate this to understanding meaning of a word in text by understanding the previous contexts. RNN is a adaptation of the standard feed-forward neural network allowing it to model sequential data. The basic schema of RNN is shown in Figure 9 where hidden unit takes input of current unit as well as contextual units to provide output. Different from the feed-forward neural networks, the decision for current input depends on activation from previous time steps[167]. The activation values from previous state are stored inside the hidden layers of a network which provides all the temporal contextual information in place of fixed contextual Windows used for feed forward neural networks (FFNN). Hence, dynamically changing contextual window helps RNN better suited for sequence modeling tasks.

The gradients of the RNN are very easily computed using back-propagation through time [168], and gradient descent is a suitable option to train RNN. However, dynamics of RNN makes effectiveness of gradient highly unstable, resulting to exponential gradient decays or gradient blows up. To resolve this issue, enhanced RNN architecture, Long Short-Term Memory (LSTM) is designed [169]. The architecture of LSTM are made up of special units called memory blocks inside the hidden layer of RNN. Memory cells are made up of memory blocks storing the temporary state of network and gates controlling the information flow. A forget gate prevents LSTM models from processing continuous input streams by resetting the cell states. Today RNN are being extended towards deep RNNs, bidirectional RNNs and recursive neural nets. Among many application areas, language modeling [170, 171, 172], text generation  [173, 174, 175], speech recognition [176, 177, 178], text summarization [179, 180, 181] are the major areas transformed by the use of RNN models.

Generative Adversarial Network (GANs) are the generative modeling approach using deep learning methods. Goodfellow et al. [182] proposed GAN as a technique for unsupervised and semi-supervised learning. In a GAN model, two pairs of networks namely: Generator and Discriminator are trained in combination to reach the goal as presented in Figure 10. Creswell et al. [183] define the generator as an art forger and the discriminator as an art expert. The forger create forgeries, with the aim of making realistic images whereas discriminator tries to distinguish between forgeries and real image. The generator’s goal is to mimic a model distribution and the discriminator separates the model distribution from the target [184]. The concept here is to consecutively train the generator and the discriminator in turn, with goal of reducing difference between the model distribution and the target distribution. During the training of GANs, discriminator learns its parameters in such a way that its classification accuracy is maximized and generator learns its parameters is such a way that it maximally forges the discriminator. The generator and the discriminator must be differentiable, while not necessarily being invertible.

GAN’s ability to train a flexible generator functions, without absolutely computing likelihood has made GAN successful in image generation [185, 186] and image super resolution [187, 188]. The flexibility of the GAN models has allowed them to be extended to structured prediction [189, 190], training energy based models [191, 192], generating adversarial examples for malware [193, 194], and robust malware detection [45, 195]. GAN models suffers from issues of oscillation during training process [196], depriving them from converging to a fixed point. Approaches that has been taken to stabilize the learning process still rely on heuristics which are very sensitive to modifications [197]. Recent research work [198, 199] is being carried out to address the stability issues of GANs.

Windows PE file format is an executable file format based on the Common Object File Format (COFF) specification. The PE file is composed of linear streams of data. The structure of Windows PE file as shown in Figure 11 is derived and confirmed from [200, 201, 202]. The header section consists of MS-DOS MZ header, MS-DOS stub program, PE file signature, the COFF file header and an optional header. File headers are followed by body sections, before closing the file with debug information. First 64 bytes of PE file are occupied by MS-DOS header. This header is required to maintain the compatibility with files created on Windows version 3.1 or earlier. In absence of MZ header, the operating system will fail to load the incompatible file [201]. The Magic number used in the header determines if the file is of compatible type. Stub-program is run by MS-DOS after loading the executable and is responsible for giving output messages which include errors and warnings.

PE file header is searched by indexing the e_lfanew field to get the offset of file which is the actual memory-mapped address. This section of the PE file is one of the target areas to perform modification by using these locations as macros in order to create adversarial examples. The macro returns the offset of file signature location without any dependency on the type of executable file. At offset 0x3c, 4-byte signature is placed which helps to identify the file as a PE image. The next 224 bytes is taken by optional header. Even though it may be absent in few types of file, it is not an optional segment for PE files. It contains information like initial stack size, program entry point location, preferred base address, operating system version, section alignment information and few other [201]. Section headers are of 40 bytes without any padding in between. The number of entries in the section portion is given by the NumberofSections field in the file header [203]. Section header contains fields like Name, PhysicalAddress or VirtualSize, VirtualAddress, SizeOfRawData, PointerToRawData and few more pointers with characteristics.

Data is located in data directories inside data section. Information from both the section header as well as optional header are required to retrieve data directories. The .text section contains all the executable code sections along with the entry point. An uninitialized data for the applications are stored in the .bss section which includes all declared static variables and .rdata section represents all the read only data like constants, strings and debug directory information. The .rsrc section contains resorce information for the module and export data for an application are present in .edata section. Section data are the major area where perturbation takes place to make a file adversarial. Debug information is placed on .debug section but the actual debug directories resides in the .rdata section.

Android APK file has been recently victimized as a tool for adversarial attacks  [204, 205, 206, 207]. APK file is basically a ZIP files containing different entries as shown in Figure 12. Different sections of APK files are described below:

• •

  Androidmanifest.xml: AndroidManifest.xml contains the information to describe the application. It contains the information like application’s package name, components of application, permissions required and compatibility features [208]. Due to presence of large amount of information, AndroidManifest.xml is one of the majorly exploited section in APK file for adversarial attack.

• •

  classes.dex: As Android applications are written in Java, source code will be with extension .java. These source code are optimized and packed into this classes.dex file.

• •

  resources.arsc: This file is an archive of compiled resources. Resources include the design part of apps like layout, strings and images. This file form the optimized package of these resources.

• •

  res: Resources of app which is not compiled to store in resources.arsc stays in res folder. The XML files present inside this folder are compiled to binary XML to boost performance [209]. Each sub-folder inside res store different types of resources.

• •

  Meta-INF: This section is only present in signed APKs and has all the files in APK along with their signatures. Signature verification is done by comparing the signature with the uncompressed file in archive [210].

In this section we will look into the internal structure of PDF file format. PDF is a portable document with wide range of features, capable of representing documents which includes text, images, multimedia and many others. The basic structure of a PDF file is shown in Figure 13 and are discussed below:

• •

  PDF header: PDF header is the first line of PDF which specifies the version of a PDF file format.

• •

  PDF Body: The body of a PDF file consists of objects present in the document. The objects include image, data, fonts, annotations, text streams, etc. [211]. Interactive features like animation and graphics can also be embedded in the document. This section provides the possibility of injecting contents and files within it, which makes it the most favourable avenue for adversarial attackers.

• •

  Cross-reference table: The cross-reference table stores the links of all the objects or elements in a file. Table helps on navigating to other pages and contents of a document. Cross-reference table automatically gets updated on updating the PDF file.

• •

  The Trailer: The trailer denoted end of PDF file and contain a links to cross-reference table. The last line of trailer contains the end-of-file marker, %%EOF.

Signature is a short sequence of bytes unique to each malware and helps in identifying malware from rest of the files. Since this approach works by maintaining malware signature database, there are very low false positives rate. Signature based detection has been very effective and fast for malware detection but it is not able to capture the ‘unseen’ malware. Figure 15 shows the malware detection process using signature. As shown in figure, signature database is predefined list of all the possible malware signatures and is solely responsible for entire malware detection process. The anti-malware engine if detects the malicious objects, malware signature is updated in signature database for future detection. A good malware detector’s signature database has a huge number of signature that can detect malware [213]. Signature based malware detection are good at speed of detection, efficiency to run and broad accessibility [214]. However, the inability to detect zero-day malware whose signature is not available in database of anti-malware engine led to question the reliability of signature based approaches. Digital signature patterns can be extracted easily by attacker and implemented to confuse the signature of malware. Current malware comes with polymorphic and metamorphic properties [215, 216] which can easily change their behavior enough to change the signature of file. With complete dependence over known malware, signature based detection can neither detect zero day attacks nor the variations of existing attacks. In addition to it, signature database grows exponentially with malware family growing at a rapid pace [217].

To overcome the limitations posed by signature based approach, zero-day malware detection techniques are focused to capture the unseen malware. In modern zero-day detection approaches, suspicious objects are identified based on behavior or potential behavior of the file [22, 21]. An object’s potential behavior is first analyzed for suspicious activities before deploying in a real-time production environment. Those behavior which are anomalous to benign file actions, indicates the presence of malware. Most of the zero-day detection approaches are built around machine learning systems, with state-of-art works using modern deep learning architectures. The captured behavior of the file under inspection is generalized using machine learning models, which is later used to detect unseen malware family. Here, we discuss three different types of zero-day malware detection in this section below:

• •

  Static Approach: Static malware detection is the closest approach to signature based system as detection is carried out without running the file. Execution of unknown file may not be always possible in system due to security risks and this is where static detection comes into play. Anti-malware system captures static attributes like hashes, header information, file type, file size, presence of API calls, n-grams etc. from binary code of executable using reverse engineering tools. Once the features are extracted, they are pre-processed to keep only non-redundant and important features. Among numerous available features, n-Grams [218, 219] for byte sequence analysis and Opcode [220] used to analyze the frequency of ’Operation Code’ appearance are the most widely used ones. As shown in Figure 16, the extracted features are fed to different machine learning algorithms ranging from classical to deep learning architectures to train the detection model. The trained model is then used to carry out detection of malware from static features. However, static detection alone is not sufficient to detect more sophisticated attacks [221, 222, 223] as the static features can not reflect the exact behavior of malware on run time, which limits its applications [224] in real world.

• •

  Dynamic and Online Approach: Dynamic approaches [225, 226, 227] are constructed by executing a suspicious file inside the isolated virtual environments like a sandbox [228] and detecting malware based on a run-time behavior of a program. The use of closed environment is to prevent malware from escaping and attacking the system where analysis is being conducted. Malware on execution can change the registry key maliciously and obtain the privileged mode of operating system [229]. During the execution of malware, properties of operating system changes which is logged by agent in controlled environments. Dynamic analysis enables system to capture dynamic indicators like application programming interface (API) calls, registry keys, domain names, file locations and other system metrices. These features are pre-processed and fed to machine learning model to train malware detector in the flow as shown in Figure 17. Dynamic analysis are considered more powerful than static due to ability to capture more number of system features. Code obfuscation approaches and polymorphic malware are considered ineffective against dynamic malware detection [230] reflecting its resilience from such sophisticated malware.

  Dynamic approach, though overcoming some limitations of the static detection, have its own challenges. Every suspicious file needs to be executed in an isolated environment for specific time frame which results in expense of significant time and resources [231]. The malware file does not guarantee to exhibit a same behavior both in a sandbox and live environment [232]. Modern smart malware comes with the ability of detecting the presence of sandbox and stay dormant till they reach live systems. In most of the current applications, both static and dynamic analysis are combined to detect the presence of malware in the file [233, 234, 235]. To combat the issue of polymorphic and metamorphic malware which are evasive to control malicious functionality only during some particular events, online malware detection approaches are performed [236, 237, 238, 20, 19]. Online malware analysis continuously monitors the system for the presence of maliciousness in any file [239]. Continuous monitoring helps to capture the malware at any time in live environment. However, online detection also demands for continuous monitoring overhead to the system. As most of the adversarial attacks performed so far in the literature are on static malware detection approaches, this survey will primarily focus on evasion attacks carried out against static malware detection.

The adversary’s knowledge is the amount of information about a model under attack that the attacker has, or is assumed to have, to carry out adversarial attacks against the model. An adversarial attack can be classified into two groups based on the attacker’s knowledge:

• •

  White box attack: In a white box approach, an attacker has full knowledge about the underlying model. Such knowledge might include, but not limited to, the name of the algorithm, training data, tuned hyper-parameter, gradient information, among others. It is relatively easy to carry out attacks in white box model due to large amount of available knowledge. Current state-of-art works on white box environment have achieved near perfect adversarial attacks [100].

• •

  Black box attack: In a black box approach, an attacker only have access to inputs and outputs of the model. There is no information provided about the internal structure of the model. Generally in black box attack, surrogate model is created by making guess on internal structure of target model using input and output [240, 81]. In addition, in a gray box attack [241], a type of black box attacks, the attacker knows the output performance of the model in the form of accuracy, confusion matrix or some other performance metrics.

There is large variation on the amount of adversarial knowledge starting from complete access to actual source codes to receiving only output of models. In general, it is assumed that black box adversarial attacks are difficult to orchestrate compared to white box, primarily due to the information available regarding the underlying target model. However, black box attacks reflect more real world use-cases where, in practical sense, an attacker will not likely have any knowledge of models or other parameters.

Attack surface includes different vulnerable points by which an attacker attacks the target model. Machine learning algorithms pass through a pipeline of different stages before deployment. The flow of data through this data pipeline introduces vulnerabilities in each stage [242]. Starting from collection of data, transformation and processing to output generation, an attackers have different attack entry points. Attack surface comprise all those points in machine learning models (malware defender models in our case), where adversaries can carry out their attacks. Based on different approaches to carry out attacks, attack surface has been classified into following broad categories [243]:

• •

  Poisoning Attack: This attack is carried out by contaminating training data during the training process of models [244, 245, 246]. Training data is poisoned with faulty data, making machine learning models learn on wrong dataset. As a result of poisoned training data, the entire training process is compromised.

• •

  Evasion Attack: This attack is performed by trying to evade a trained system through adjusting malicious input samples at test time [97, 247, 100]. Evasion attacks do not require any access to training data but requires some level of access to the target model.

• •

  Exploratory Attack: This attack is carried out against a model with blackbox access [240, 248]. Attackers try to maximize their knowledge without direct access to the underlying algorithm and attempt to reflect the similar input data patterns.

Adversarial capabilities denote the abilities of adversaries and are dependent on their knowledge of the target model. Some adversaries have access to training data, some have access to gradient information of the model, while others do not have any access to the model at all. The capabilities of attacker vary depending on the information and phase (i.e. training or testing phase) of the model they are attacking. The most straightforward attack approach is attacker having access to full or partial training data. For adversarial attacks carried out on malware files, adversarial capabilities can be classified into following categories:

• •

  Data Injection: Data injection is the ability of attackers to inject a new data. There are multiple types of data injection that might take place. One type of injection can be done on training data before training process. Another type of data injection is carried out by inserting a perturbations which forms a new section or replaces original section within an existing file. Injected data can corrupt the original model or cause the data injected file to evade detection.

• •

  Data Modification: Data modification can also be performed both for training data and evading file. If attacker has access to training data, data can be modified to cause model learn on modified data. Attacker can also modify input data to cause perturbation and leading to evasion.

• •

  Logic Corruption: Logic corruption is the most dangerous ability to be possessed by attacker and also the most improbable. Whenever an attacker has complete access over a model, they can modify the learning parameters and other hyper-parameters related to model. Logic corruption can go undetected which makes it hard to design any remedies.

An attacker tries to fool the target model, causing it to produce misclassifications. Details of algorithms used to successfully attack and achieve the adversaries goals are discussed in section 6. Typically, the adversarial goals of attacker’s are categorized as follows:

• •

  Untargeted Misclassification: An attacker tries to change the output of model to a value different than original prediction. For a malware classification problem, if a ML model is predicting a malware file as family A, the goal is to force the model to misclassify it as a family other than A.

• •

  Targeted Misclassification: An attacker tries to change the output of the model to a target value. For example, if a ML model is predicting a malware file as family A, the goal is to force the model to misclassify it as a family B.

• •

  Confidence Reduction: An attacker’s goal is to reduce the confidence of a ML model’s prediction. It is not necessary to change the prediction value but a reduction of confidence is enough to meet the goal.

To summarize, Figure 18 gives an overview of the adversarial attack difficulty with respect to the attacker’s knowledge, capabilities and goals. While moving in the direction of increasing attack complexity from confidence reduction to targeted misclassification, attack difficulty also increases for the attacker. However, whitebox attacks with higher attacker’s capability has least attack difficulty.

Szegedy et al. [71] proposed one of the first gradient based approaches for adversarial example generation in the imaging domain using the box constrained Limited-Memory Broyden-Fletcher-Goldfarb-Shanno optimization technique. The authors studied counter-intuitive properties of deep neural networks which allow small perturbations in the images to fool deep learning models for misclassification. Adversarial examples trained for particular neural network are also able to evade other neural networks trained on completely different hyper-parameters. These results are attributed to non-intuitive characteristics and intrinsic blind spots of deep learning models learned by back propagation, with structure connected to data distribution in a non-obvious way. Traditionally, for small enough radius $\epsilon$$>$0 around the given training sample $x$, $x+r$ satisfying $||r||<\epsilon$ will be classified correctly by a model with very high probability. However, many underlying kernels are found not holding to this kind of smoothness. Simple optimization procedure is able to find adversarial sample using imperceptibly small perturbations, leading to incorrect classifications by classifier. While adding noise to an original image, the goal is to minimize perturbation $r$ added to the original image under $L_{2}$ distance. A classifier mapping pixel value vectors to a discrete label set is denoted as $f:R^{m}\xrightarrow{}\{1...k\}$ and the loss function associated is given by $loss_{f}:R_{m}*\{1...k\}\xrightarrow{}R^{+}$. For a given image $x\in R^{m}$ with a target label $l\in\{1...k\}$, box-constrained optimization problem is defined as :

where x is the original image, r is the added perturbation, f is the loss function of the classifier and l is the label of incorrect prediction by the classifier. Perturbed $x+r$ is arbitrarily chosen using distance minimizer. The computation of distance $D(x,l)$ is done by approximation using box-constrained L-BFGS. After this early proposal of L-BFGS for adversarial examples generation, plenty of research were triggered to dive into flaws of deep learning.

Considering gradient-based optimization technique as a workhorse of modern AI, Goodfellow et al. [74] proposed an efficient approach for generation of adversarial perturbation in image domain. In contrast to earlier works which explained adversarial phenomena to non-linearity and overfitting, the authors argued the linear nature of neural networks leading to their vulnerability. Linear behaviour in high dimensional space are found sufficient to cause adversarial samples. Linearity is the result of trade off while designing models that are easy to train. LSTMs [169], ReLUs and maxout networks [250] are all found to be intentionally designed to behave linearly for ease of optimization. To define the approach formally, let’s consider $\theta$ as a parameter of model, $x$ as input to the model, $y$ as target associated with $x$ and $J(\theta,x,y)$ be the cost function for training neural network. On linearizing the cost function around the current parameter values $\theta$, perturbation can be obtained by

where required gradient can be computed using backpropagation and the approach is called as Fast Gradient Sign Method.

Conversion of features from problem space to feature space affects the precision. Commonly images are represented by 8 bits per pixel and all other information below 1/255 of continuous range are discarded. With limited precision, classifier may not be able to respond to all perturbations whose size is smaller precision of feature. Classifier having well-separated decision boundary for for classes are expected to assign same class for original sample $x$ and perturbed sample $x^{{}^{\prime}}$ until $||\eta||_{\infty}<\epsilon$ where $\epsilon$ is small enough to be discarded. Taking the dot product and weight vector w and an adversarial example $x^{{}^{\prime}}$:

The adversarial perturbation increases the activation by $w^{T}\eta$. The amount of perturbation can be controlled by keeping max norm constraint on $\eta$ and assigning $\eta=sign(w)$. Taking $w$ with $n$ dimensions and having average magnitude weight vector $m$, the activation grows by $\epsilon mn$. Even though $||\eta||_{\infty}$ does not grow with increasing dimensionality of the problem but for high-dimensional problems, the activation change caused by the adversarial perturbation can grow linearly. In presence of sufficient dimensions, even simple linear models are seen to have adversarial examples. Adversarial examples are found to occur in contiguous regions of 1-D subspace defined by the fast gradient sign method, where traditional belief was in fine pockets. This allows adversarial examples to be abundant and generalizable across different machine learning models. FGSM being one of the most efficient techniques for adversarial with fast generation of samples, is among the most used technique in this field.

Different from the one step perturbation approach where single large step in direction of increasing loss of classifier, Iterative Gradient Sign Method takes iterative small steps while adjusting the direction after each step [72]. Basic iterative method extends FGSM approach by applying it multiple times with small step size and clipping the pixel values after each iteration to ensure the perturbation within $\epsilon$ neighbourhood of original image.

where $X_{N+1}^{adv}$ is the perturbed image at $N^{th}$ iteration and $Clip_{X,\epsilon}\{X^{{}^{\prime}}\}$ function performs pixel wise clipping on image $X^{{}^{\prime}}$ in order to keep perturbation inside $L_{\infty}\epsilon$-neighbourhood of source image X. Kurakin et al. [72] extended basic iterative method to iteratively least likely class method to produce adversarial for targeted misclassification. Desired class for this version of iterative approach is chosen based on the prediction of the trained network, given as:

To make adversarial classified as $y_{LL},\;logp(y_{LL}|X)$ is maximized taking iterative steps in direction given by $sign{-\nabla_{x}logp(y_{LL}|X)}$. Now the adversarial generation cost function can be viewed as:

This iterative algorithm helps to add finer perturbations without damaging the original sample even with higher $\epsilon$.

Most of the adversarial generation techniques are based on observing output variations to generate input perturbations, while Papernot et al. [251] crafted adversarial samples by constructing a mapping of input perturbations with output variations. The approach is based on limiting the $l_{0}$-norm of the perturbation which deals with minimal number of pixel modification. The proposed adversarial generation algorithm against feed forward DNN modifies small portion of input features by applying heuristic search approaches. Adversarial sample $X^{*}$ is constructed by adding perturbation $\delta_{X}$ to benign sample $X$ through following optimization problem:

where $X^{*}=X+\delta_{X}$ is the adversarial sample and $Y^{*}$ is the desired adversarial output. Forward derivative is used to evaluate the changes on output due to corresponding modifications in input and these changes are presented on matrix form called as Jacobian of the function. Replacing gradient descent techniques with forward derivative allows attacker to generalize attack for both supervised and unsupervised architecture for broad families of adversaries. Forward derivative of Jacobian matrix of function F is learnt by neural network during training process. For a function with single dimensional output, Jacobian matrix is given as:

Forward derivative helps to distinguish the region which are unlikely to generate adversarial sample and focus on features with high forward derivative for efficient search and smaller distortions. JSMA is a black-box attack with only assumption of DNN architecture using differentiable activation function. Algorithms take a benign sample $X$, a target output $Y^{*}$, a feedforward DNN F, a maximum distortion parameter $\gamma$, feature variation parameter $\theta$ and undergoes following steps to give adversarial sample $X^{*}$ such that $F(X^{*})=Y^{*}$.

• •

  Compute forward derivative $\nabla F(X^{*})$

• •

  Construct a Saliency map S based on the derivative

• •

  Modify an input feature $i_{max}$ by $\theta$

The forward derivative calculate gradients similar to those computed for back-propagation, taking derivative of network directly in place of its cost function and differentiating with respect to input features in place of the network parameters. Consequently, gradients are propagated forward which helps in determining input components leading to significant changes in network outputs. Authors extended application of saliency maps [252] to construct adversarial saliency maps which gives features having significant impact on output and thus is a very versatile tools to generate wide range of adversarial examples. Once saliency map gives the input feature that needs to be perturbed, benign samples are perturbed using distortions limited by parameter $\gamma$. The limiting parameter $\gamma$ depends on human perception of adversarial sample. The experiment is carried out on LeNet architecture using MNIST dataset. Adversarial crafting is done by increasing or decreasing the pixel intensities of images. Before wrapping up JSMA, we discuss briefly about Saliency Vector. Saliency Vector contains the features in input blocks of data and their significance for machine learning model. Importance of input feature given by saliency vector can be thought of as a function of network’s sensitivity to changes in the input feature [253]. The regions of element in original files corresponds to the position of elements in the vector and value of that element gives the measure of importance of that feature region. Zhou et al. [254] proposed Class Activation Mapping to produce visual interpretation for CNN-based model. Authors used global average pooling to indicate discriminative image regions used by CNN to make the decision. Due to difficulty of modifying and retraining the original model to obtain CAM, Selvaraju et al. [255] proposed Grad-CAM method. Gradient-weighted CAM uses the gradient information flowing into the final convolutional layer to produce a localization map highlighting the important regions of image. Saliency vector allows to observe database bias and improve the models based on training data.

Carlini & Wagner [256] proposed an adversarial generation approach to overcome the defensive distillation. Defensive distillation has been recent discovery to harden any feed-forward neural network against adversarial examples by performing only a single retraining. Proposed approach is able to perform three types of attacks: $L_{0}$ attack, $L_{2}$ attack and $L_{\infty}$ attack to evade defensively distilled and undistilled networks. These attacks are based on different distance metrics which are:

• •

  $L_{0}$ distance, measuring the number of pixels modified in an image

• •

  $L_{2}$ distance, measuring the standard Euclidean distance between original sample and perturbed sample

• •

  $L_{\infty}$ distance, measuring the highest change among any of the perturbed coordinates

The optimization problem for adversarial generation of input image $x$ is given as:

where input $x$ is fixed and goal is to reach $\delta$ that minimizes $D(x,x+\delta)$. $D$ could be any of distance metric among $L_{0},\;L_{2}\;or\;L_{\infty}$. Different approaches are taken to limit the modification and generate valid perturbations:

• •

  Projected gradient descent is allowed to perform only one standard gradient descent, clipping all other coordinates

• •

  Clipping gradient descent does not clips input perturbation on each iteration, but clips into objective function to be minimized

From their experiments [256], it is observed that $L_{2}$ attack has low distortion while $L_{\infty}$ and $L_{0}$ is not fully differentiable as well as bad suited for gradient descent.

Dezfooli et al. [257] proposed an untargeted white-box adversarial generation technique called as DeepFool. DeepFool works by minimizing the euclidean distance between perturbed sample and original samples. Attack begins by generating linear decision boundary to separate the given classes and accompanied by addition of perturbation perpendicular to the decision boundary that separates classes as demonstrated in Figure 19. Attacker projects the perturbation into a separating line called hyper-plane and tries to push it beyond for mis-classification. In high dimensional space, decision boundaries are usually non-linear, so the perturbation are added iteratively by performing multiple attacks till evasion. Attack for such multiclass finds the closest hyperplane and projects input towards that hyperplane and then proceeds to other. The minimal perturbation required to misclassifiy classifier is the orthogonal projection of $x_{0}$ onto $f$ and is given by closed loop formula in Equation 10.

where $r$ is perturbation, $f$ is classifier function. $w$ is gradient and $x_{0}$ the original sample.

All of the previously discussed adversarial generation algorithms are dependent on gradient of detector module which limits the adversarial attack space within white-box attack. Chen et al. [258] proposed a black-box adversarial generation approach by estimating the gradients of targeted DNN with only access to input and output of a target. Zeroth order methods are gradient-free optimization approach requiring only the zeroth order oracle for optimization process. The objective function is analyzed at every two close points $f(x+hv)$ and $f(x-hv)$ with a very small h, so that a gradient along the direction of vector $v$ can be estimated. Gradient estimation is followed by an optimization algorithms like gradient descent. While attacking black-box DNN with large input size, use of a single minute step of gradient descent can be very inefficient as large number of gradient needs to be estimated. To resolve this optimization issue, coordinate descent method is used by optimizing each coordinate iteratively.

Zeroth Order Optimization attack is inspired by formulation of the C&W attack. The loss function of C&W attack is modified in such a way that it is only dependent to output of DNN and desired class label. A new hinge like loss function based on the output $F$ of DNN is defined in Equation 11.

The proposed adversarial generation technique does not require gradient to be estimated accurately. To accelerate the zeroth order methods, attack-space dimension reduction approach is carried out which reduces the number of gradient to be estimated. Attack-space dimension reduction might lead to insufficient search space to find adversarial. So, hierarchical attack scheme is used where dimension is gradually increased during process of optimization. Using coordinate descent and importance sampling, attacker can also update pixels on a selective basis.

Another gradient free adversarial generation approach is proposed by Su et al. [259] by generating one pixel adversarial perturbations based on differential evolution (DE). Differential evolution is a population based optimization algorithm which has ability to find higher quality solutions than gradient based approaches [260]. Since gradient information is not required for DE, the need of differentiable objective functions is also omitted. One pixel attack perturbs single pixel using only probability labels. Single pixel modification allows attackers to hide the adversarial modifications making it imperceptible. To carry out the attack, each image is represented as a vector where each scalar element represents one pixel. With $f$ as the target function, $x=(x_{1},...,x_{n})$ representing n-dimensional inputs, $t$ being the original class, $e(x)=(e_{1},...,e_{n})$ denoting the perturbation to be added to the input with maximum modification limited to $L$ , the optimized solution is given by Equation 12.

where d is a small number. This approach deals with determining two values: dimension to be perturbed and the required corresponding magnitude of modification for each dimension. Unlike other attack strategies, OPA focus modifications on only one pixel without limiting the strength of modification.

Microsoft Windows is a dominant operating system on PCs with more than 70% market share and 1.5 billion users worldwide [261]. Gartner research [262] predicts 30% of cyberattacks by 2022 will be carried out in form of adversarial. Abundant availability has placed Windows malware at the core of adversarial threats. Even the continuously evolving machine learning based malware detectors are not able to withstand adversarial attacks. In this section, we will cover different adversarial attacks carried out on Windows malware detectors.

Machine learning based models being data hungry, feature engineering is a critical task to feed important features as input. However, the advent of deep neural network has allowed models to learn features themselves from complex raw data. Deep neural networks have shown impressive performance in malware detection by providing whole binary file as a input without any hand crafted feature engineering effort. Different malware detectors have been previously discussed but we want to include Raff et al. work [263] in this section (referred as MalConv) which has been a industry standard in the field by making detection considering whole executable. Most of the attacks discussed in the survey are carried out against MalConv.

Detection using raw bytes comes with the sequence problem with millions of time steps and batch normalization hindering the learning process. Raff et al. tried to replicate the neural net’s success in learning features from raw inputs, successfully performed in image [75], speech [177] and text [264] domain. With bytes in malware having multiple modalities of information and contents exhibiting multiple spatial correlation, MalConv becomes the first ever architecture with ability to process a raw byte sequences of more than two million steps. MalConv addresses high amount of positional variation present in executable files. It’s architecture as shown in Figure 20, combines the convolutional activation with a global max-pooling before going to fully connected layers allowing the model to produce its activation regardless of the locations of the detected features. MalConv as one of the only robust static detector, has been considered a baseline for most of the static adversarial attacks.

Android has over 2.8 billion active users and owns 75% market share in mobile phone industry [313]. The wide usage of Android platform has attracted security threats in numerous forms and adversarial evasion attack is one of them. Table IX provides brief comparison among different adversarial attacks crafted against Android files. Grosse et al. [314] [304] generated adversarial examples for state-of-art Android malware detection trained on DREBIN dataset [309] with more than 63% accuracy. Authors migrated the method proposed by Papernot et al. [251] to handle binary features of Android malware while preserving the malicious functionality. Binary features are derived by statically evaluating code based on system call and usage of specific hardware. These derived features are also known as binary indicator vectors. The final goal of adversarial attack is to find perturbation/noise $\delta$ such that the prediction results of $y^{{}^{\prime}}$ of $F(X+\delta)$ is different from the original result, i.e, $y^{{}^{\prime}}\neq y$. $y^{{}^{\prime}}$ gives target class while crafting adversarial. Authors adopted Jacobian matrix of neural network $F$ for adversarial generation. To get adversarial, the gradient of function $F$ with respect to $X$ is calculated to get the direction of perturbation such that output of classification function will change. Perturbation $\delta$ with highest positive gradient in direction of target is selected and is kept small enough to prevent negative change due to intermediary alterations of gradient. Functionality is preserved in this approach by changing features resulting in addition of only single line of code. Research also confine the modifications to manifest features related to AndroidManifest.xml file contained within Android application. With permissions, intents and activities being the most frequently modified features, authors successfully evaded DREBIN classifier[309] preserving the semantics of malware.

To overcome the white box attack issues, Rosenberg et al. [205] implemented GADGET framework to convert malware binary to an adversarial binary without access to malware source code. Proposed end-to-end black-box method is extended to bypass the multi-feature based malware classifiers relying on the transferability in RNN variants. For target RNN detector, malicious API call sequence is the adversarial example to be generated. Authors perform mimicry attacks where malicious code mimics the system calls of benign code to evade [315]. Adversaries train surrogate model having same decision boundaries as that of detector and then execute white-box attack on surrogate model. To build the surrogate model, black-box detector is queried with synthetic input values chosen by a Jacobian based heuristics in the prioritizing directions where model output varies. API calls which are nearest to the direction given by Jacobian are inserted to generate adversarial sequence. The label assigned to the input value by a black-box model gives the sign of the Jacobian matrix dimension. Jacobian matrix of the surrogate model is used for evaluation and after each iteration, synthetic example is added to each existing sample. However, finite set of legitimate API embeddings may not be enough for adversarial insertion, causing insertion of most impactful API call in direction indicated by Jacobian. Adversarial examples that are able to fool surrogate model has high likelihood of fooling original model as well [193]. Adversarial generation showed same success against the substitute and blackbox model with short API sequences, making adversarial generation faster. Framework also uses Cuckoo Sandbox to verify the malicious functionality of generated adversarial malware. GADGET framework wraps malware binary with proxy code and increases the risk even higher providing malware-as-a-service.

Adversarial attack on malware domain has not considered to manipulate the feature vector to see impact of mutation due to strict functionality preserving requirements of malware. Malware Recomposition Variation (MRV) based approach proposed by Yang et al. [305] performed an analysis of malware file semantically and construct a new variant of malware. Mutation strategies synthesized by conducting semantic-feature mutation analysis and phylogenetic analysis are used to perform automatic program transplantation [316]. Changing the traditional belief over mutation, authors followed feature patterns of existing malware to preserve the functionality. The proposed framework performs inter-component, inter-app, and inter-method transplantation. More comprehensive attack is performed on both the manifest file as well as dex code. Use of RTLD features allows substitute model to approximate targeted detector and also helps in separation of essential features and contextual features.

• •

  Resource features: These features provide the security-sensitive resources that are impacted by the malware. Resource features are extracted by forming call graphs and pinpointing call-graph nodes of those security-sensitive methods.

• •

  Temporal features: These features provide the environmental context at the trigger point of the malicious property and the context is inferred from the attributes at entry points.

• •

  Locale features: These features describe the location of programs where malicious activities are observed and can be either of the Android components of concurrency constructs.

• •

  Dependency features: These features are provided by constructing an inter-procedure control-flow graph, indicating the control dependencies when malicious activities are invoked.

The goal of malware evolution attack is to imitate and automate the evolution of malware using phylogenetic evolutionary tree [317]. A phylogenetic evolutionary tree shows the inferred relations between different samples based on the similarities and differences of feature representation. A pairwise distance between samples is fed to phylogenetic tree generation algorithm, Unweighted Pair Group Method Average (UPGMA), to generate phylogenetic tree which reflects the structure present in a similarity matrix. Another approach proposed by authors, confusion attack, tries to complement malware evolution attack against robust malware detectors [318]. The feature values modified by confusion attacks can be shared by both the malware sample and benign apps. This approach is more complete due to flexibility of mutation by different number of means but introduces challenges on keeping functionality intact due to higher volume of modification [207].

Several adversarial generation approaches have been conducted by making minor perturbations on existing attacks. Liu et al. [306] proposed a Testing framework for Learning-based Android Malware Detection systems (TLAMD). Framework uses genetic algorithm to perform black-box attack against Android malware detection system. Android files are modified by adding the request permission code to AndroidManifest.xml file which was originally proposed by Grosse et al. [304]. The restriction was imposed on the types and magnitude of permissions that can be added to AndroidManifest file. A random population is generated giving the characteristics of permission to add and followed by calculating the disturbance size for the sample malware. Using the evaluated perturbation size, adversarial is generated and tested against the detection model. Based on the result of detection either new disturbance size is calculated using genetic algorithms or perturbation is successfully added on Android application. Another important aspect of genetic algorithm is to model the fitness function which is this framework has been defined in Equation 18.

where $w_{1}$ and $w_{2}$ are the two weights, $\delta$ is the added disturbance, $num(\delta)$ is number of permission features added and $F(X+\delta)$ gives the probability of malicious sample being detected as malware. Fitness function searches for optimal solution to perform mutation leading to a new fit individual able to evade detection. Random forest approach is used to filter out insignificant features during feature extraction. Disturbance generated by genetic algorithm are able to bypass malware detectors trained on neural networks, logistic regression, decision trees and random forest.

Shahpasand et al. [206] implemented GAN to generate adversarial by keeping threshold on the distortion values of generated samples. The generated optimum perturbation $\delta$ is added to existing malware to produce adversarial. Like every other GAN architecture, generator can learn the distribution of benign samples, generating perturbations which are able to bypass learning based detector. The discriminator implicitly enhances the perturbation by escalating the loss of generator while the adversarial samples are identifiable with benign files. Loss function for GAN is similar to that of Goodfellow et al.’s [182] work given in Equation 19.

where $D(x)$ gives probability of sample $x$ coming from benign software distribution, $L_{GAN}$ helps maximizing resemblance of adversarial malware with benign sample while limiting distortion less than c. Loss function of adversarial malware $(L_{adv.Mal})$ is given as:

where benign class is targeted for an adversarial sample $x+G(z)$ against classifier $f$ trained on $l_{f}$ loss function. Finally, model loss is defined as:

In above equation, first loss forces the GAN to generate adversarial similar to benign samples while the second loss inclines adversarial samples to have higher miss-classification rate.

The goal of adversarial generation has been to bypass malware detector without losing functionality. However, due to growth in adversarial malware in recent time, defenders are employing firewalls to stop adversarial sample. Li et al. [307] extended the work of MalGAN [282] to make it robust against detection system equipped with firewall. Despite its high evasion rate against malware detectors, MalGAN is found to be less effective against detection systems using firewall. Bi-objective GAN with two discriminator having different objectives are used. One of the discriminator helps distinguish between malware and benign whereas another discriminator helps to find out whether the samples are adversarial or normal ones. Due to this feature, adversarial generated by generator can successfully bypass through the firewall as well as the malware detection. Authors used permissions, actions and application programming interface calls as a features to generate adversarial. In every round of training, gradient descent is used to update the parameters of discriminators and generator, represented by $\theta_{d_{1}}$, $\theta_{d_{2}}$ and $\theta_{g}$ respectively. $G$, $D_{1}$ and $D_{2}$ are considered functions implemented by generator, discriminator 1 and discriminator 2. First discriminator, used to separate benign and malicious class is trained to update $\theta_{d_{1}}$, by minimizing its loss

where $N_{B}$ and $N_{M}$ denote the distributions of benign and malicious samples. Second discriminator used to separate between normal and adversarial samples is trained by updating $\theta_{d_{2}}$ by minimizing its loss

where $N$ is the distribution of normal samples and M is the distribution of generated samples detected as malicious. Generator is updated twice each round. After updating $\theta_{d_{1}}$, $\theta_{g}$ is updated by minimizing

where $M$ is distribution of malicious samples fed to generator and $P$ is the distribution of noises fed to generator. After $\theta_{d_{2}}$ is updated, $\theta_{g}$ is updated by minimizing

Pierazzi et al. [207] formalized the adversarial ML evasion attacks in the problem space and proposed a problem space attack on Android malware. The proposed approach formalizes the set of restriction in transformations available, semantics preserved, robustness of preprocessing approaches, and veracity. Research work is focused on evasion attacks at test time by modifying the objects in real input space corresponding to feature vector. With a goal of overcoming inverse feature-mapping problem from previous researches, author presents the idea of side-effect features. Side effect feature defines and proves the necessary as well as sufficient conditions behind the problem space attacks. An attack on a feature space is projected towards a feasibility region satisfying the problem space constraints to obtain the side effect features. To formally demonstrate the side-effect features, an object $x\in X$ is initialized within a feasible region. A gradient-based attack takes an object $x$ in feature space to $x+\delta^{*}$, with $\delta^{*}$ being the perturbation. The addition of perturbation misclassifies malware files as benign with high confidence. However, the new point with perturbed feature on feature vector may not be inside the feasibility region. Side effect features in perturbation help to map $x+\delta^{*}$ to the region of feasible problem-space.

Since side effect features contribute towards preserving validity of malware due to impact of original gradient based perturbation, side effect features alone can have both positive or negative influence on the classification score. Authors use automated software transplantation [316] to extract byte-codes from benign donor applications to inject into a malicious host, also known as organ harvesting. Insertion into a host is carried out between statements in non-system package to preserve the functionality and Cyclomatic Complexity is used to take a heuristic approach maintaining existing homogeneity and preventing violation of plausibility. Prior research works relied heavily on addition of permissions to the Android Manifest which is considered dangerous in Android documentation [324]. Authors bind the modifications to inject single permission to the host app. Gradient based strategy using greedy algorithm proposed in this approach overcomes previous limitations of preserving semantics and pre-processing robustness.

To overcome the challenges of limited access to target classifiers while circumventing black-box Android malware detectors, Bostani et al. [308] proposed a novel iterative and incremental manipulation strategy. The attack is carried out in two-step: preparation and manipulation. In preparation phase, automated software transplantation is employed to prepare action sets from Android apps. The n-gram-based similarity method is used to identify benign apps that closely matches to malware files. Insertion of extracted gadgets of closely matching benign files force malware sample towards the blind spots of the classifier. In the manipulation stage, perturbation on malware samples are applied incrementally, choosing from the collected action set. The search method randomly chooses suitable transformation and applies them to malware samples. This approach shows a high success rate in query efficient approach but increases the size of adversarial perturbation which in turn increases the risk of perturbation being easily detected.

Along with widespread applications and adoption, PDF documents have been one of the most exploited avenues for adversarial malware attacks. Initially, JavaScript based and structural properties detection were prominent for recognising malware in PDF. But freedom to distribute chunks of Javascript code and assemble together at run-time and high degree expressiveness in JavaScript language led to failure of Javascript based detection. Despite significant growth in PDF malware detection from JavaScript using deep learning techniques, the challenges posed by adversarial examples still exist. Early evasion attempts on PDF documents were crafted by Smutz et al. [325] and Šrndić et al. [326] using heuristic approaches. The authors proposed approach to build more robust PDF malware detection techniques, showcased the adversarial ability to mislead linear classification algorithm successfully.

Flexible logical structure of PDF has allowed to craft adversarial by carefully analyzing its structure. Maiorca et al. [319] demonstrated evasion technique called reverse mimicry attack against popular state-of-art malware detectors  [327, 328, 325]. Traditionally, malicious PDF files are believed to be structurally different from benign PDF files. Taking advantage of this structural difference, most of malware detectors were able to discriminate PDF files with very high accuracy. However, malware files which can imitate the benign file structure or vice-versa can easily fool the detector. Reverse mimicry attacks can make benign files malicious with minimal changes in their structure. Malicious payloads poison the samples, initially classified as benign. Three kinds of malicious payloads introduced to benign files take the sample across the decision boundary of malware detector. First one is EXE payload with malicious embedding, which is introduced using Social Engineering Toolkit²⁴²⁴24https://www.secmaniac.com/ as a new version after its trailer. On the addition of a new root object, the new trailer will point to a new object. In this payload, authors embedded malicious PDF files inside another benign PDF files using embedded function of PeePDF[329] tool. The embedded PDF file automatically opens without user interaction, allowing malicious PDF to be embed inside a benign one without any restriction on embedding file. PDF file injection enabled an attacker to have fine-grained control of structural features in the carrier file. A final kind of payload insertion is carried out by encapsulating a malicious JavaScript code without reference to other object. It helps to minimize variation in the benign-file structure by adding only one object to the tree. Such attacks based on structure are even capable of evading detectors using non-structural features. Table X provides overview of adversarial attacks carried out on PDF files.

Optimization based evasion attack against PDF malware detection was introduced by Biggio et al. [320, 321]. The attack was carried out using a gradient based optimization procedure inspired by Golland’s discriminative directions technique [330] to evade linear as well as non-linear classifiers. The proposed work was able to carry out complete knowledge and constrained knowledge attacks on non-linear models like Support Vector Machine(SVM) and neural networks. Work relied on easiness to insert new objects than to remove an embedded object to prevent from corrupting the PDF’s file structure. This approach used a gradient descent procedure with special consideration to avoid getting stuck on local optima. To increase the probability of successful evasion, an attacker needs to reach attack points that are legitimate, and to reach this, the additional penalizer term is introduced using a density estimator. The extra component helps imitate features of known legitimate samples, reshaping the objective function by biasing the gradient descent towards the negative class concentration region. This optimization-based approach dates before the realization of adversarial examples against deep learning architectures [331].

Srndic et al. [322] further enhanced optimization based attack against deployed system PDFrate [325] using mimicry attack, and Gradient Descent and Kennel Density Estimation (GD-KDE) attack. The attack takes advantage of discrepancy between functioning of PDF readers and PDFrate in terms of interpretation of semantic gaps as explained in [332]. The dummy contents to insert should be ignored by PDF readers but affect the feature computation in PDFrate. PDFrate evaluates sets of regular expressions from raw bytes, reading from beginning of PDF files while PDF readers parse PDF files using PDF format authorized by ISO 32000-1. PDF reader looks at the end of PDF for cross-reference table and goes to locate the object directly. Among 135 features of PDF rate, MIMICUS²⁵²⁵25https://github.com/srndic/mimicus, 35 features are modified while incrementing values for 33 features with preserved functionality. Trailer section of PDF files were moved arbitrarily far away from cross-reference table, generating an empty space for file injection without affecting functionality of PDF document. A string pattern that is separated by whitespace is injected into the gap between CRT and the trailer of targeted PDF files as demonstrated in Figure 28. To match with specific PDFrate regular expression, patterns are also crafted. Two attack algorithms are implemented as explained:

• •

  Mimicry attack independent of underlying classifier, mimics a benign file by changing the modifiable features of malicious file. To increase effectiveness of the approach, each malware is trained to mimic 30 different benign files.

• •

  Gradient Descent and Kernel Density Estimation (GD-KDE) attack defeats classifier with a known and differentiable decision function [320]. GD-KDE algorithm follows the gradients of the classifier’s decision function and the estimated density function of benign samples.

PDF detection techniques are mostly reliant on PDF parser to extract features for classification [333, 334, 335]. These parser are unable to extract all JavaScript of PDF file. Carmony et al. [323] created a reference JavaScript extractor that measured the difference between the parser and Adobe Reader by tapping Adobe reader on locations given by binary analysis. Manual analysis refines the few candidate tap points provided by dynamic binary analysis. JavaScript extraction tap points are a function from which Adobe Reader extracts and executes JavaScript code from PDF documents. The memory accessed by Adobe Readers when reading PDF files using automatically executable JavaScript is analyzed to determine the raw JavaScript extraction tapping points. Proposed PDF parser confusion attack apply obfuscation on malicious PDF sample by analyzing the weaknesses of extractors in approach as shown in Figure 29. Reference extractor enables several new obfuscation in comparison to existing extractors and combination of these obfuscation were able to bypass all JavaScript extractor based detector. Metadata based detection systems require parser confusion attacks to be combined with reverse mimicry attack as core content of sample is not changed by confusion attacks.

In order to preserve maliciousness, most of research works take conservative approach by only inserting new contents and refraining from modification or removal of existing contents. Xu et al. [247] proposed a black-box generic method to evade the classifier as shown in Figure 30. As in figure, first the population is initialized by performing random modifications on malicious file. Then, each member of populations are passed through target classifier to measure maliciousness and through oracle to confirm the functionality. If no any samples are able to evade target classifier with functionality intact, subset of initialized population are chosen for next generation based on fitness score which indicates the progress towards evasive sample. Now, the population generation is repeated and this process is continued till the evasive sample is found or threshold iterations is met. Efficiency of search is enhanced by collecting traces of used mutation operation and reuse the effective operations. These effective traces are used for population initialization to generate variants for other malware. The author uses genetic programming (GP) to bring off stochastic modifications in an iterative manner till evasion. The oracle output and results of prediction from the target classifier need to be fed to the fitness function. Non malicious samples determined by oracle are assigned with low fitness score while malicious samples are provided with high score. Tools used as an oracle for maliciousness verification of adversarial samples, is an open computational challenge along with the selection of appropriate fitness function. The generic method is capable of automatically finding evasive variants irrespective of detection algorithms.

Hardware malware detectors use low-level information of features collected from hardware performance monitoring units available in CPUs. Hardware malware detectors are prone to reverse engineering [339], allowing mimicry attack [340] to reverse-engineered models. Adversarial against such detectors are carried out by generating perturbations in form of low-level hardware features, following the architecture shown in Figure 31. These adversarial generation approaches differ only on type of features used in comparison to previously discussed works. Table XI provides brief comparison of adversarial attacks against hardware malware detectors. Khasawneh et al. [336, 341] demonstrated evasion of Hardware Malware Detectors(HMD) after being reverse engineered, using low overhead evasion strategies. Data collected by running malware and cleanware programs on a virtual machine operating on Windows 7 are used to train a reverse-engineered model. Execution of malware requires disabling of Windows security services and firewall. Data required for training are dynamic traces while executing the program and are collected by using Pin instrumentation tool [342] by executing 500 system call or 15 millions of committed instructions. These dynamic traces are profiling of a run time behaviour of the programs. This dataset is comprised of three types of feature vectors:

• •

  Instructions feature giving the frequency of instructions.

• •

  Memory address patterns giving distribution of memory references.

• •

  Architectural events giving the occurrence of architectural events.

Reverse engineering allows to methodically create model similar to HMD, given the attacker has ability to query the target detector. Target detectors are considered to be based on logistic regression and neural networks to generalize most of the classification algorithms. Reverse engineering is carried out in all three types of feature vectors using both algorithms. Attackers have no information about size of instruction window, the specific features used by detector or the classification algorithm, detectors are trained on but have similar detectors to test their hypothesis. Authors[336] constructed a Dynamic Control Flow Graph (DCFG) of the malware to insert instructions into the executing malware dynamically. Injection of instruction feature increases the weight of the corresponding feature while memory feature injection alters the histogram of memory reference frequencies. Instructions are inserted using two approaches, block level and function level. Khasawneh et al. picked the instructions with negative weights to move the malware away from the decision boundary. Heuristic approach was taken to identify the candidate instructions for insertion. Weighted injection strategy where probability of selecting particular instruction is proportional to negative weight allowed to bypass HMD with around 10% dynamic overhead.

Dinakarrao et al. [337] also proposed an adversarial attack on low-level micro-architectural events captured through Hardware Performance Counters (HPC). Victim’s defense system (HMD) being black-box needs to be reverse engineered to mimic the behaviour. Number of HPC patterns required to bypass HMD is unknown which leads to need of adversarial sample predictor. The HPC patterns perturbing mechanism are implemented using a lower-complexity gradient approach, Fast Gradient Sign Method (FGSM).The adversarial perturbations needed to misclassify HPC trace is calculated using gradient based cost function of neural network. With $\theta$ being hyperparameters of neural network, $x$ being input HPC trace to the model and $y$ as output, cost function $L(\theta,x,y)$ is defined as:

where $\epsilon$ is a scaling constant ranging between 0.0 to 1.0 and used to limit the perturbation to a very small value. The LLC load misses and branch misses are the most significant micro-architectural events of malicious application [343]. Linear models are built to find dependency of array sizes(n) and elements flushed(k) in route to determine LLC load misses generated. The adversarial perturbation generator is executed separately with a sample application to misclassify. The significance of running in a separated thread is to stop adversarial HPC generators from interfering with the application’s source code.

Micro-architectural events are useful when the malware program executes on a hardware device causing some changes in the performance of hardware. In addition to malware program execution, threat of injecting a malicious circuit during fabrication of hardware devices has grown [344]. Malicious circuits, also known as hardware Trojans, can be inserted into circuits producing logically equivalent results. Hardware circuits like processors include contribution of number of vendors before reaching to final product, and hence, carries high risk of adversarial circuit insertion anywhere between designing stage to manufacturing stage. Modifications in manufacturing stage are more tedious in comparison to design stage as few changes in hardware description language (HDL) are enough to embed hardware Trojans to the circuit. A trigger circuit allows the payload circuit to trigger malicious behavior such as information leakage and degrading performance after satisfying the trigger condition. A gate level netlist (the lists of nets and circuit elements) is used for malware detection by analyzing its structural features [345]. Several hardware Trojan detection have been recently adopted towards machine learning approaches including neural networks [346, 347, 348, 349].

Nozawa et al. [338] proposed an architecture to develop the adversarial hardware Trojan using Trojan-net Concealment Degree (TCD) and Modification Evaluating Value (MEV). Feature mapping issues like in all other adversarial attacks in Windows, Android and PDF are prevalent in hardware Trojan as well. Hardware circuits are represented in graphs structure and modifications in feature space does not guarantee the transfer back of modification to graph structure. Two stages in designing period are known for adversarial attack. First one being RTL (Register-transfer level) description design step and second one after logic synthesis. Authors take the assumption of Trojan detector using neural network architecture and the availability of raw output values from the detector to train adversarial model. Goal of adversarial is to maximize the loss function which is given as cross entropy(H) in Equation 27:

where K is the number of units in output layer, $p_{i}(x(e))$ is the function to return label of $x(e)$ and $q_{i}(x(e))$ is function that returns the prediction result from classifier.

Summing up the loss function for each net and calculating the average gives Trojan-net concealment degree (TCD):