Reinforcement Learning for Feedback-Enabled Cyber Resilience

A cyber-resilient mechanism (CRM) can be decomposed into four stages: Preparation, Protection, Response, and Recovery. We also call it a P2R2 CRM. The first stage of the CRM is preparation. At this stage, we assess the security risk of the cyber system and design appropriate security policies, such as the deployment of honeypots or deception mechanisms [6, 7], training of the employees [8], proper configurations of detection systems [9], and the preparation of the backups and contingencies for recovery plans [10]. Preparation is often done offline and ahead of the real-time operations. Good preparation can help facilitate effective prevention and fast response to unanticipated scenarios in later stages.

The second stage is prevention. At this stage, we implement the designed security policies and protect the cyber systems. Some attacks can be easily detected and thwarted in real-time because of the meticulous preparation and design of the security policies. For example, consolidating moving targeting defense [11] into the communication protocols would make it harder for the attacker to map out the traffic patterns and consequently thwart the denial of service attack. However, despite this effort, there is still a probability for an attacker to become successful, especially for a highly resourceful and stealthy one.

The third stage of response is critical to defending against attacks when we fail to thwart them at the prevention stage. At this stage, we acquire the information of the footprint of the attacker and reconfigure the cyber system to minimize the further risk of the attack on the cyber system [12, 13]. The response to attacks is delay-sensitive. The information acquisition and the reaction to the observables need to be fast and done before the attacker moves onto the next stage of its attack. A fast response mechanism would rely on the acquired information and the designs at the preparation stage. There would be a cost in usability and performance when we respond to the attacks. The optimal response mechanism is a result of studying such tradeoffs.

The fourth stage of the CRM is recovery. The goal of the recovery stage is to reduce the spill-over impact of an attack and restore the cyber system performance as much as possible. The response to attacks in real-time is often at the sacrifice of the performance of the cyber system. There is a need to maintain system’s operation and gradually restore its functionality to normal while reacting to the attacks.

Fig. 1 illustrates the four stages of a typical CRM over a timeline. Before the operation, the cyber system designs a protection mechanism by considering the anticipated and known attacks and the risks of its own system. The cyber system starts to operate at $t_{0}$. At time $t_{1}$, an attacker launches attack $a_{1}$. As the cyber system prepares for it, this attack is successfully thwarted, and the system is protected until the attacker launches a new attack $a_{2}$ at time $t_{2}$. The cyber system did not prepare for this attack. The attack accomplishes its goal (e.g., lateral movement or penetration of the system) and, as a result, the cyber risk increases or the cybersecurity performance drops. The cyber system needs to respond to the successful attack and the ensuing attack $a_{3}$ at time $t_{3}$. After fast learning from attacker’s footprint, the cyber system makes strategic decisions to reconfigure and adapt to the adversarial environment and gradually improves the security posture of the system. The adapted system becomes more robust to attacks. The ensuing attack $a_{4}$ at time $t_{4}$ can be thwarted by the adapted protection mechanism. The cyber system restores to its best-effort post-attack performance at $t_{4}$.

From Fig. 1, one can measure the cyber resilience across the four stages by the time it takes from the first attack to the recovery, i.e., $T=t_{4}-t_{2}$. This time interval indicates how fast the response is [14, 15]. The worst-case performance degradation between the interval $t_{2}$ and $t_{4}$ is $M$. The gap between the restored performance and the initial or planned performance, i.e., $D$, shows the effectiveness of the resilience. The natural goal of the CRM is to minimize $T,M,$ and $D$. It is also evident that these metrics not only depend on the cyber system itself but also the type of attacks that the adversary launches. An optimal design would require understanding what threats are preventable and what can be mitigated by resilience.

One critical component of the P2R2 CRM for cyber resilience is the response mechanism, which creates a dynamic process that involves information acquisition, online decision-making, and security reconfiguration. These components are strongly interdependent. One way to characterize this interdependence is through a feedback loop, as illustrated in Fig. 2.

The feedback structure provides a system and control approach to cyber resilience [16]. The cyber system can be viewed as a plant. The CRM collects information from the cyber system through sensing and observation. The acquired information is used to reason about the current situation and decide how the system should react. The CRM acts on the recommended decisions by reconfiguring the security parameters and updating the cyber systems. The reasoning for the optimal cyber response can be viewed as a controller. The security reconfiguration can be interpreted as the actuation of the cyber system. The control perspective provides a rich set of tools and concepts from control theory to design an optimal, robust, and adaptive CRM to improve cyber resilience.

One pivotal control framework to achieve an optimal, robust, and adaptive CRM is reinforcement learning (RL). Similar to classic control approaches, RL leverages the idea of feedback architecture to guarantee certain aspects of performance such as optimality and stability. However, RL approaches are distinguished from classic control approaches from several facets.

First, RL allows the CRM policies to adapt to the online observations without knowing the underlying dynamics. The CRM can only observe the state $s$ and the cost $c$ from the cyber system at each step and then choose action $a$ to optimally improve cyber resilience. RL is particularly useful for cyber systems as their precise models are often unavailable, and the influence of the attacker makes it even harder to map out an accurate cyber system model. There are also many unknowns in the attack model and cyber systems that require online learning and adaptation. RL provides an appropriate framework to bridge the gap between control theory and cyber resilience.

Second, RL allows the CRM to have a more generic space that captures the activities in the cyber system. Classic control oftentimes deals with physical systems such as mechanical systems [17], electrical circuits [18], chemical systems [19] and biological systems [20]. The state space of such systems is mostly continuous and finite dimension represented by $\mathbb{R}^{n}$ with $n$ being a finite integer. Cyber systems are computer & communication systems integrated with physical systems whose state space is discrete or hybrid (discrete and continuous) in most cases. RL is versatile at dealing with different types of state space. This versatility enables CRM to capture the complex status of cyber systems. Besides, RL equipped with function approximation techniques copes well with large-scale cyber systems that contain a large number of elements or have high-dimension state spaces [21, 22].

Third, RL requires different stages than the classic control approaches before being implemented in real systems. Specifically, the implementation of RL techniques undergoes three phases: the training phase, the testing phase, and the execution phase. In the training phase, the agent interacts with the environment or the simulator to obtain data and improves his/her policy using the obtained data. In the testing phase, we evaluate the performance of the improved policy. If the policy reaches optimality or a satisfactory level of sub-optimality, the learned policy (or the learned CRM in our case) will be put into use, and the implementation enters the execution phase. Classic control approaches go through four phases: the modeling phase, the design phase, the testing phase, and the execution phase. In the modeling phase, modelers with domain knowledge build a mathematical model for the underlying system. In classic control, the training phase is replaced by a design phase where experts with domain expertise design a policy/controller based on the model constructed for the cyber system. The designed policy is then tested and evaluated before being executed in real systems. The training phases consume an enormous amount of data to improve the learned CRM constantly. Much effort need to be spent obtaining the data through either simulation or interacting with the environment. Data-inefficiency needs to solved by combining modern RL techniques with domain expertise from cyber security experts.

The uncertainties in the cyber system are caused by not only the complexity in its configuration but also the incomplete information of the attack models. An attack model specifies the vulnerabilities an attacker would exploit on the attack surface of the cyber system. The design of the CRM pivots on the type of vulnerabilities that the defender aims provide resilience for if he cannot succeed in defending against attackers who exploit them. We categorize the vulnerabilities into three major types. The first one is the class of posture-related vulnerabilities that arise from the resource disadvantage of a defender. The defender is not able to prepare for all vulnerabilities on the attack surface and has to rely on CRM to overcome this disadvantage. The second one is the class of information-related vulnerabilities that result from the information asymmetry between an attacker and a defender. The attacker has more information about the defender than the defender has about the attacker. The CRM is useful to tilt the information asymmetry and allow the defender to create uncertainties for the attacker while gaining reconnaissance of the attacker. The third type of vulnerability is human-induced. Humans are the weakest link in cyber protection. Social engineering, human errors, and insider threats are common attack vectors that an attacker can use to gain access to cyber systems before further penetrating the systems and reaching the targeted assets. The CRM can monitor human performance and guide humans to reduce the risks of their behaviors.

We use three applications to elaborate the design of CRMs for the three major types of vulnerabilities. The first application designs moving target defense (MTD) for posture-related vulnerabilities that result from the defender’s natural disadvantages. The MTD creates reconnaissance difficulties for attackers by introducing uncertainties into the cyber systems. The authors in [11] explore the capability of RL in changing the system configurations to create the most uncertainty for the attackers while minimizing the change cost, which leads to a trade-off between security and usability. Both attackers and defenders apply RL to learn the risk and update their policies.

The second application addresses the information-related vulnerabilities that arise from the defender’s information disadvantage. Deception technologies, including honeypots and honeyfiles, have been used to reduce information asymmetry between attackers and defenders. The authors in [23] use RL to design the deception technologies to engage the attacker and obtain threat information. The goal is to extract the most threat intelligence from attacks while minimizing the risks of detection and evasion. The study leads to the security insight that compared to an immediate ejection after detection, it can be more beneficial to engage attackers in the honeypots to obtain threat intelligence.

The third application designs CRM to mitigate human-related vulnerabilities. Adversaries have exploited human inattention to launch social engineering and phishing attacks toward employees and users. Compared to the reactive attentional attacks that exploit the existing human attention patterns, proactive attentional attacks can strategically change the attention pattern of a human operator or a network administrator. The authors in [24, 25] have identified a new type of attacks called Informational Denial-of-Service (IDoS) attacks that generate a large volume of feint attacks to overload human operators and hide actual attacks among feints. They create an RL-based CRM to learn the optimal alert and attention management strategy.

As we use RL to improve the resilience of the cyber system, an intelligent attacker can also seek to compromise or mislead the RL process so that the cyber system ends with a worse or vulnerable security configuration. In this case, RL enlarges the attack surface and creates opportunities for the attacker. Hence it is essential to safeguard the RL while using it to secure the cyber system. Section 5 presents several attack models that target at RL algorithms and discusses their impact and ways to secure them. RL agents learn a ‘good’ policy through interacting with the environment and exchanging key information such as the rewards, the state observations, and the control commands. The three exchanged signals, which can be exploited by the attacker, become the vulnerabilities of most RL-enable systems. To understand RL in an adversarial environment, the first is to understand the adversarial behaviors of the attackers. To do so, we need to craft the attack models, which specify the attacker’s objective, the attacks available to the attacker, and the knowledge the attacker has. The second is to understand how different types of attacks affect the learning results of RL algorithms, which requires the assessment of the performance degradation of the RL-enabled systems. With the understanding of attack models and their impact on the learning results, we can develop countermeasures to mitigate the effect of the attacks and to protect RL-enabled systems from attacks.

We review the current studies on security of RL and focus on attacks on the three signals, i.e., attacks on the rewards, attacks on the sensors, and attacks on the actuators. We present a sequence of attack models that can achieve a significant impact on RL-enabled systems with only a tiny effort on attacking. Given that this branch of research is still in its infancy, we identify several research gaps that researchers from the learning community and the control community can help narrow. One missing component is that most works focus on deceptive attacks that falsify the values of the feedback information. In contrast, few works study the DoS/jamming attacks on RL algorithms. Another missing component is the study of attacks on actuators and the defensive mechanisms against these attacks.

Throughout the paper, we use calligraphic letter $\mathcal{X}$ to define a set and boldface letter $\mathbf{X}$ to denote a vector. Notation $\mathbb{E}$ represent expectation and $\mathbb{R}_{+}$ represents the set of positive real numbers. The notations $\mathbb{Z}_{\geq 0}$ and $\mathbb{Z}_{>0}$ represent the set of non-negative and positive integers, respectively. The indicator function $\mathbf{1}_{\{x=y\}}$ equals one if $x=y$, and zero if $x\neq y$. If two sets $B\subset A$, $A\backslash B$ denotes the set $\left\{x\ \middle|x\in A,x\notin B\right\}$. If $\mathcal{A}$ is a finite set, then we let $\Delta\mathcal{A}$ represent the set of probability distributions over $\mathcal{A}$, i.e., $\Delta\mathcal{A}:=\{p:A\leftarrow\mathbb{R}_{+}|\sum_{a\in\mathcal{A}}p(a)=1\}$. The main notation of this paper is introduced in Section 2.1. For sections that are dense with parameters, we include separate notation tables to help readers navigate.

The remainder of the paper is organized as follows. Section 2 first introduces a general schema of RL methods to help readers navigate in the rich universe of RL algorithms. After that, Section 2 gives an overview of literature that studies how RL is leveraged to fight against different types of attacks on cyber systems. While Section 2 focuses on the different attacks, Section 3 centers on three major classes of vulnerabilities: the posture-related vulnerabilities, the information-related vulnerabilities, and the human-related vulnerabilities. In Section 3, we discuss how each vulnerability can be addressed by RL. Following the discussion of 3, Section 4 delves into three application domains, each representing one class of vulnerabilities, and presents design methodologies for cyber resilience. Compared to previous sections that are about RL for cyber security, Section 5 is about security of RL, which introduces potential security threats faced by RL itself. In Section 5, we discuss the vulnerabilities of RL algorithms and provides an overarching review on this emerging research topic. We conclude the paper in Section 6 and discuss several directions for future research.

The RL agent learns a near-optimal, if not optimal, to protect, prevent, or recover from attacks by constantly exploring and exploiting the environment in a feedback manner, as is shown in Fig. 2. In Fig. 2, $s$ represents the security state of the cyber system, and the cost $c$ is the monetary cost or performance degradation of the system induced by attacks and operations. The RL agent, or the defender, aims to find an action $a$ that optimally modifies the system configuration or other parameters, leading to changes in the security state and the cost. The feedback loop forms an iterative process of agent-environment interactions.

RL is underpinned by a mathematical framework called Markov Decision Process (MDP). An MDP is denoted by a $5$-tuple $\langle\mathcal{S},\mathcal{A},c,\mathcal{P},\beta\rangle$, where $\mathcal{S}$ is the state space containing all possible states of the cyber system. The action space $\mathcal{A}$ denotes the actions available for the defender to protect the cyber system, to recover from the damage caused by attacks, or to mitigate the effect of attacks. The cost $c$ depends on the current and/or the next security states, and the current action, which is usually denoted by a function that maps $\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}_{+}$. The transition kernel $\mathcal{P}$ defines the rule of how the system state evolves based on the actions taken by the defender. The discount factor $\beta$ is a weighting factor that assigns more weight to current costs than future costs. The goal of an MDP problem is to find a policy $\pi_{t}:\mathcal{S}\rightarrow\mathcal{A}$ to minimize a certain form of the accumulative costs $\sum_{t=0}^{\infty}\beta^{t}c_{t}$ over time.

MDPs are generic modeling tools that can model the dynamic and feedback nature of various types of cyber systems. Conventional MDP approaches require the knowledge of the transition probability of the cyber system and the explicit definition of the reward function to find an appropriate strategy. However, in reality, obtaining such information is prohibitive for two reasons. The first is due to the complexity of the cyber system and its dynamics. The second is because the attackers are usually on the dark side whose behavior is unknown to the defender. RL solves the MDP problem without the knowledge of the transition kernel $\mathcal{P}$ and the cost function $c$. Instead of utilizing a prior known transition kernel and cost function, RL agent learns the optimal policy from sequences of states $\{s_{t}\}_{t\in\mathbb{Z}}$, costs $\{c_{t}\}_{t\in\mathbb{Z}}$, and actions $\{a_{t}\}_{t\in\mathbb{Z}}$, which are obtained by interacting with the environment over time.

Having been actively studied for decades, RL has a rich universe of algorithms that help the agent find a satisfactory policy. Covering every single algorithm is beyond the scope of this paper. Here, we highlight a general schema of RL models shown in Fig. 3 and present a concise taxonomy of RL algorithms.

The rapid development of Information and Communication Technologies (ICTs) has fueled significant growth in Internet of Things (IoT) usage and Cyber-Physical Systems (CPS) deployment. CPS involves embedded computers and networks used to monitor and control the physical processes, with feedback loops where physical processes affect computations and vice versa. The Internet of Things (IoT) refers to a network comprised of physical objects capable of gathering and sharing electronic information, with physical objects receiving orders from the network and the network receiving information from the physical objects. Both IoT and CPS involve computation units, networking units, and physical entities that form a feedback cycle with human or machine decisions in the loop. Due to the wide adoption of IoT and CPS in many domains such as manufacturing, power, agriculture, battlefield, homes, etc., security threats faced by IoT and CPS systems need to be well addressed.

First, in a CPS, traditional physical controllers have been replaced by micro-computers and processors with embedded operating systems [51]. Micro-computers provide advantages to the CPS, such as flexible configuration with an accessible human-machine interface and digital communication abilities that enable remote control and access. However, the embedded software opens up doors for software attacks such as code injection attacks, cross-site scripting attacks when using a web server for system configuration, and malware attacks. Second, CPS is networked, meaning that the essential components in the physical systems are often connected to corporate networks and the Internet. Security issues may be raised because complete isolation is difficult to achieve. Connectivity can occur in unexpected ways. For example, Stuxnet, a malicious computer worm accused of stymieing Iran’s nuclear-fuel enrichment program, was introduced to the target environment via an infected USB flash drive [52]. The security challenges of CPS will become more severe as the scale and scope of the Internet and other communication techniques grow.

Similarly, IoT is also facing security challenges amid the increasing adoption of IoT technology. IoT employs advanced computing and communication technology such as Zigbee, Bluetooth, radio-frequency identifications (RFIDs), and cloud computing. The employment of these technologies provide opportunities for the attacker to launch cyber attacks, such as DoS attack, eavesdropping, data injection attacks, etc. Beyond that, unlike CPS, IoT usually encompasses small physical units with lower power that are mobile and geographically distributed. Hence, IoT is more susceptible to physical attacks. Due to the dynamic and feedback nature of CPS and IoT, RL has been applied to address the security and privacy challenges faced by these systems. In this section, we provide a brief review of the applications of RL for the security of CPS and IoT. The review follows a taxonomy based on the attack types and different RL algorithms.

The technologies to mitigate posture-related vulnerabilities have been extensively studied in the literature. Moving Target Defense (MTD) is one of the modern technologies to neutralize attacker’s position advantage by creating reconnaissance difficulties and uncertainties for attackers. There is a surge of recent literature on using RL to choose an adaptive configuration strategy to maximize the impact of MTD with particular focuses on the dynamic environment [69], reduced resource consumption [70], usability [11], partially observable environment [71, 72], and multiagent scenarios that contains both the characteristics of the system and the adversary’s observed activities [73, 72].

Other techniques have been proposed to protect disadvantageous cyber systems from falling victim to resourceful and determined attacks such as Distributed Denial-of-Service (DDoS) or powerful jamming attacks. For example, [58] mitigates DDoS Flooding in Software-Defined Networks; [74] introduces a multi-objective reward function to guide an RL agent to learn the most suitable action in mitigating application-layer DDoS attacks; [75] proposes a feature adaption RL approach based on the space-time flow regularities in IoV for DDoS mitigation. Among the works of RL for DDoS attack, there has been recent attention on large-scale solution [21, 22] via cooperative RL, low-rate attacks in the edge environment [76], and sparse constraint in cloud computing [77]. Besides DDoS attacks, RL has a wide application in mobile edge caching [78], mobile offloading for cloud-based malware detection [79], and host-based and network-based intrusion detection [80, 81].

Deception is a ubiquitous phenomenon and has been used as a proactive defense method. Honeypots and the related cyber deception technologies, e.g., honeyfiles, honeynet, honeybot, etc., have been widely used to mitigate stealthy and deceptive attacks. In particular, RL has been widely used to develop self-adaptive honeypots in normal conditions [82, 83, 84] and resource-constrained environment [85]. As attackers become more sophisticated, they manage to fingerprint and evade honeypots [86]. This reduces the size of a captured dataset and the chance to extract threat intelligence from it. Thus, there is an urgent need for stealthy honeypots [87] that can counter fingerprinting. In [88], the authors use RL to conceal the honeypot functionality. Besides honeypot fingerprinting, RL is also used to extract the most threat intelligence [89, 90, 23].

Besides honeypots, there are emerging technologies to detect and respond against deceptive attacks. In [91, 92, 93], the authors model the interactions between stealthy attackers and the defenders as a dynamic Bayesian game of discrete or continuous type where multi-stage response strategies are designed based on belief updates. The convergence of continuous-type Bayesian game under a sequence of finer discretization schemes is discussed in [94]. In [95], the authors design the information revealed to the users and attackers to elicit behaviors in favor of the defender. In [96] and [97], RL is used to optimally deploy the deception resources and emulate adversaries, respectively. Many works have attempted to address various spoofing attacks using RL. In [61], the authors propose RL-based spoofing detection schemes where the interactions between a legitimate receiver and spoofers are formulated as a zero-sum authentication game. In [98], the authors model the behavior of exploring face-spoofing-related information from image sub-patches by leveraging deep RL. Developing counter-deceptive and defensive deceptive technologies is in its infancy. RL is a promising tool to make the design quantitative, automatic, and adaptive.

We classify human vulnerabilities into acquired and innate vulnerabilities, depending on whether they can be mitigated through short-term security training and awareness programs. Social engineering [99] is a common attack vector that targets acquired human vulnerabilities such as fear to express anger, lack of assertiveness to say no, and the desire to please others. Threat actors use psychological manipulation techniques to mislead people to break normal security procedures or divulge confidential information. The authors in [100] have used RL to detect cyberbullying automatically, and [101, 102] have proposed a feedback learning framework, e.g., RL, to fight against social engineering attacks. As a representative form of social engineering, phishing attacks use email or malicious websites to serve malware or steal credentials by masquerading as a legitimate entity. Non-technical anti-phishing solutions include security training and education programs, while technical solutions include blacklisting, whitelisting, and feature-based detection. To handling zero-day phishing attacks, (deep) RL has been used both to detect phishing emails [103], phishing websites [104], spear phishing [105], and social bots [106] in Online Social Networks (OSN). Attackers can also exploit human innate vulnerabilities induced by bounded attention and rationality. The authors in [24, 25] have identified a new type of attacks called Informational Denial-of-Service (IDoS) attacks that can exacerbate the human operators’ cognition overload by generating a large number of feints and hiding real attacks among them.

Most of the existing works take humans as an independent component in the cyber system and aim to compensate indirectly for the human vulnerability through additional mechanisms. An alternative way is to directly affect the human component and consider an integrated human-cyber system. Due to the unpredictability and modeling challenges of human behaviors, RL and feedback control serves as the tool to determine how to affect human incentives and perceptions effectively and efficiently. In [107], the penalty and reward are changed adaptively through a feedback system to improve compliance of human employees and mitigate insider threats. In [108], RL is used to develop the optimal visual aids to enhance users’ attention and help them identify phishing attacks. In [25], the authors use RL to determine resilient and adaptive strategies for alert and attention management. Developing security-assistive technologies that directly affect humans is in its infancy, and it is a promising direction to explore further.

Many legacy Information Technology (IT) systems operate in a relatively static configuration. As many configuration parameters, e.g., IP addresses, remain the same over long periods, advanced attackers can reconnoiter and learn the configuration to launch targeted attacks. MTD is a countermeasure that dynamically configures the system settings and shifts the attack surface to increase uncertainties and learning costs for the attackers.

Many networked IT systems nowadays consist of hierarchical layers and adopt the Defense-in-Depth (DiD) approach where a series of defensive mechanisms reside on each of these layers. Let $\mathcal{N}:=\{1,2,\cdots,N\}$ denote the set of $N$ layers in a system and $\mathcal{V}_{l}:=\{v_{l,1},v_{l,2},\cdots,v_{l,n_{l}}\}$ be the set of $n_{l}$ system vulnerabilities that an attacker can exploit to compromise the system at layer $l\in\mathcal{N}$. Assuming that perfect security is unattainable and attacks can penetrate the layered defense of different depths, MTD can be applied for all layers to interact and slow down the attacker’s rate of penetration. At each layer $l\in\mathcal{N}$, the defender can choose to change the system configuration from a finite set of $m_{l}$ feasible configurations $\mathcal{C}_{l}:=\{c_{l,1},c_{l,2},\cdots,c_{l,m_{l}}\}$. Different configurations result in different subsets of vulnerabilities among $\mathcal{V}_{l}$, which are characterized by the configuration-vulnerability map $\pi_{l}:\mathcal{C}_{l}\rightarrow 2^{\mathcal{V}_{l}}$. Under the $h$-th configuration $c_{l,h}\in\mathcal{C}_{l}$, the outcome of the configuration-vulnerability mapping, i.e., $\pi_{l}(c_{l,h})$, represents the attack surface at layer $l\in\mathcal{N}$.

After the attacker has reached the layer $l\in\mathcal{N}$, the attacker can launch an attack $a_{l,k}$ from a finite set $\mathcal{A}_{l}:=\{a_{l,1},a_{l,2},\cdots,a_{l,n_{l}}\}$. Let $\gamma_{l}:\mathcal{V}_{l}\rightarrow\mathcal{A}_{l}$ be the vulnerability-attack map that associates vulnerability $v_{l,j}\in\mathcal{V}_{l}$ with attack $a_{l,k}\in\mathcal{A}_{l}$. Without loss of generality, there is a one-to-one correspondence between $\mathcal{A}_{l}$ and $\mathcal{V}_{l}$, then the corresponding inverse map is denoted as $\gamma_{l}^{-1}:\mathcal{A}_{l}\rightarrow\mathcal{V}_{l}$. Attack action $a_{l,k}\in\mathcal{A}_{l}$ incurs a bounded cost $D_{hk}\in\mathbb{R}_{+}$ when the current attack surface $\pi_{l}(c_{l,h})$ under configuration $c_{l,h}\in\mathcal{C}_{l}$ contains the vulnerability $v_{l,j}=\gamma_{l}^{-1}(a_{l,k})$. Otherwise, the attacker fails to exploit the existing vulnerabilities under the configuration and incurs zero damage. Thus, the damage caused by the attacker at layer $l\in\mathcal{N}$, denoted by $r_{l}:\mathcal{A}_{l}\times\mathcal{C}_{l}\rightarrow\mathbb{R}_{+}$, takes the following form:

The attacker’s goal is to penetrate and compromise the system, while the defender aims to minimize the damage or risk. Since vulnerabilities are inevitable in modern IT systems, the defender adopts MTD to randomize between configurations and make it difficult for the attacker to learn and exploit the vulnerabilities at each layer. Fig. 4 provides a paradigmatic example to illustrate the benefit of MTD. At the first layer highlighted by the blue box, there are three vulnerabilities. Configuration $c_{1,1}$ in Fig. 4(a) has an attack surface $\pi_{1}(c_{1,1})=\{v_{1,1},v_{1,2}\}$ while configuration $c_{1,2}$ in Fig. 4(b) has an attack surface $\pi_{1}(c_{1,2})=\{v_{1,2},v_{1,3}\}$. For each attack surface, the existing and non-existing vulnerabilities are denoted by the solid and dashed arcs, respectively. Then, if the attacker takes action $a_{1,1}\in\mathcal{A}_{1}$ that exploits vulnerability $v_{1,1}$ but the defender changes the configuration from $c_{1,1}$ to $c_{1,2}$, then the attack is thwarted at the first layer.

The authors in [11] model the conflict between a multi-stage attacker and a defender adopting MTD as a zero-sum game. At each layer $l\in\mathcal{N}$, the defender’s randomized strategy over the configuration set $\mathcal{C}_{l}$ is denoted as $\mathbf{f}_{l}\coloneqq\{f_{l,1},f_{l,2},\cdots,f_{l,m_{l}}\}\in\Delta\mathcal{C}_{l}$ where $\Delta\mathcal{C}_{l}$ is the distribution of set $\mathcal{C}_{l}$. The attacker can also choose a randomized strategy $\mathbf{g}_{l}:=\{g_{l,1},g_{l,2},\cdots,g_{l,n_{l}}\}\in\Delta\mathcal{A}_{l}$ over the set of feasible attacks at layer $l\in\mathcal{N}$ to increase his chance of a successful vulnerability compromise. In particular, $f_{l,h}\in[0,1]$ and $g_{l,k}\in[0,1]$ represent the probabilities of the defender taking configuration $c_{l,h}$ and the attacker taking action $a_{l,k}$, respectively, at layer $l\in\mathcal{N}$. Under the mixed-strategy pair $(\mathbf{f}_{l}\in\Delta\mathcal{C}_{l},\mathbf{g}_{l}\in\Delta\mathcal{A}_{l})$, the defender’s expected cost $\mathbbm{r}_{l}$ is given by

The defender’s optimal randomized strategy $\mathbf{f}_{l}^{*}\in\Delta\mathcal{C}_{l}$ against the worst-case attacks can be obtained from the mixed strategy saddle-point equilibrium (SPE) $(\mathbf{f}_{l}^{*}\in\Delta\mathcal{C}_{l},\mathbf{g}_{l}^{*}\in\Delta\mathcal{A}_{l})$ of the zero-sum game where the game value $\mathbbm{r}(\mathbf{f}_{l}^{*},\mathbf{g}_{l}^{*})$ is unique, i.e.,

Traditional cybersecurity techniques such as the firewall and intrusion detection systems rely on low-level Indicators of Compromise (IoCs), e.g., hash values, IP addresses, and domain names, to detect attacks. However, advanced attacks can revise these low-level IoCs and evade detection. Thus, there is an urgent need to disclose high-level IoCs, also referred to as threat intelligence, such as attack tools and Tactics, Techniques, and Procedures (TTP) of the attacker. As a promising active cyber defense mechanism, honeypots can gather essential threat intelligence by luring attackers to conduct adversarial behaviors in a controlled and monitored environment.

A honeynet is a network of honeypots, which emulates the real production system but has no production activities nor authorized services. Thus, an interaction with a honeynet, e.g., unauthorized inbound connections to any honeypot, directly reveals malicious activities. From an attacker’s viewpoint, the production network and the honeypot network share the same structure as shown in Fig. 6.

An attacker from the production system or external network can be attracted to the honeynet. Then, the attacker can move inside the honeynet through either physical (if the connecting nodes are real facilities such as computers) or logical (if the nodes represent integrated systems) links. The attacker’s transition is restricted by the network topology. Once the attack launches an attack at a honeypot node, the defender gains an investigation reward by analyzing the attack behaviors and extracting high-level IoCs. Since the defender can obtain more threat intelligence when the attack sustains for a longer time, the investigation reward increases with the engaging time. The defender aims to dynamically configure the honeynet through proper engagement actions to obtain more threat intelligence while minimize the following three types of risks.

• T1:

  Attackers identify the honeynet and thus either terminate on their own or behave misleadingly in honeypots.

• T2:

  Attackers circumvent the honeywall and use the honeypots as a pivot to penetrate other production systems.

• T3:

  The cost to engage attackers in the honeypots outweighs the resulted investigation reward.

To strike a balance between maximizing the investigation reward resulted and minimizing the risks, the authors in [23] model the attacker’s stochastic transition in the honeynet as an infinite-horizon Semi-Markov Decision Process (SMDP) to quantify the long-term reward and risks. The SMDP consists of the tuple $\{t\in[0,\infty),\mathcal{S},\mathcal{A}({s_{j}}),tr(s_{l}|s_{j},a_{j}),\allowbreak z(\cdot|s_{j},a_{j},s_{l}),\allowbreak r^{\gamma}(s_{j},a_{j},s_{l}),\gamma\in[0,\infty)\}$. We illustrate each element of the tuple through a $13$-state example in Fig. 7.

Each node in Fig. 7 represents a state $s_{i}\in\mathcal{S},i\in\{1,2,\cdots,13\}$. At time $t\in[0,\infty)$, the attacker is either at one of the honeypot nodes denoted by state $s_{i}\in\mathcal{S},i\in\{1,2,\cdots,11\}$, at the normal zone $s_{12}$, or at a virtual absorbing state $s_{13}$ once attackers are ejected or terminate on their own. At each state $s_{i}\in\mathcal{S}$, the defender can choose an action $a_{i}\in\mathcal{A}(s_{i})$. The action set $\mathcal{A}(s_{i})$ is finite and depends on the state $s_{i}\in\mathcal{S}$. For example, at honeypot nodes, the defender can conduct action $a_{E}$ to eject the attacker, action $a_{P}$ to purely record the attacker’s activities, low-interactive action $a_{L}$, or high-interactive action $a_{H}$, i.e., $\mathcal{A}(s_{i}):=\{a_{E},a_{P},a_{L},a_{H}\},i\in\{1,\cdots,11\}$. At normal zone, the defender can choose either action $a_{E}$ to eject the attacker immediately, or action $a_{A}$ to attract the attacker to the honeynet by generating more deceptive inbound and outbound traffics in the honeynet, i.e., $\mathcal{A}(s_{12}):=\{a_{E},a_{A}\}$. No actions needed in the virtual absorbing state, i.e., $\mathcal{A}(s_{13}):=\emptyset$.

Based on the attacker’s current state $s_{j}\in\mathcal{S}$ and the defender’s action $a_{j}\in\mathcal{A}(s_{j})$, the attacker transits to state $s_{l}\in\mathcal{S}$ with probability $tr(s_{l}|s_{j},a_{j})$ and the sojourn time at state $s_{j}$ is a continuous random variable with probability density $z(\cdot|s_{j},a_{j},s_{l})$. Once the attacker arrives at a new honeypot $n_{i}$, the defender dynamically applies an interaction action at honeypot $n_{i}$ from $\mathcal{A}(s_{i})$ and keeps interacting with the attacker until the attacker’s next transition. Since the defender makes decision at the time of transition, the above continuous-time model over $t\in[0,\infty)$ can be transformed into a discrete decision model at decision epoch $k\in\{0,1,\cdots,\infty\}$. The time of the attacker’s $k^{th}$ transition is denoted by a random variable $T^{k}$, the landing state is denoted as $s^{k}\in\mathcal{S}$, and the adopted action after arriving at $s^{k}$ is denoted as $a^{k}\in\mathcal{A}(s^{k})$.

At decision epoch $k\in\{0,1,\cdots,\infty\}$, the defender’s investigation reward $r$ consists an immediate cost $r_{1}$ of applying engagement action $a^{k}\in\mathcal{A}(s^{k})$ at state $s^{k}\in\mathcal{S}$ and a reward rate $r_{2}$, i.e.,

The reward rate $r_{2}$ at time $\tau\in[T^{k},T^{k+1}]$ represents the benefit of threat information acquisition minus the cost rate of persistently generating deceptive traffic. Considering a discounted factor of $\gamma\in[0,\infty)$ to penalize the decreasing value of the investigation as time elapses, the defender aims to maximize the long-term expected utility starting from state $s^{0}$, i.e., $u(s^{0},\pi)=\mathbb{E}[\sum_{k=0}^{\infty}\int_{T^{k}}^{T^{k+1}}e^{-\gamma(\tau+T^{k})}(r(S^{k},A^{k},S^{k+1},T^{k},T^{k+1},\tau))d\tau],$ where the engagement strategy $\pi\in\Pi$ maps state $s^{k}\in\mathcal{S}$ to action $a^{k}\in\mathcal{A}(s^{k})$. Based on dynamic programming, the value function $v(s^{0})=\sup_{\pi\in\Pi}u(s^{0},\pi)$ can be represented as

Assuming a constant reward rate $r_{2}(s^{k},a^{k},T^{k},T^{k+1},\tau)=\bar{r}_{2}(s^{k},a^{k})$, (16) can be transformed into the equivalent MDP form as follows, i.e.,

where ${z^{\gamma}}(s^{0},a^{0},s^{1}):=\int_{0}^{\infty}e^{-\gamma\tau}z(\tau|s^{0},a^{0},s^{1})d\tau\in[0,1]$ is the Laplace transform of the sojourn probability density $z(\tau|s^{0},a^{0},s^{1})$ and the equivalent reward $r^{\gamma}(s^{0},a^{0},s^{1})\allowbreak:=r_{1}(s^{0},a^{0},s^{1})+\frac{\bar{r}_{2}(s^{0},a^{0})}{\gamma}(1-z^{\gamma}(s^{0},a^{0},s^{1}))\in[-m_{c},m_{c}]$ is assumed to be bounded by a constant $m_{c}$.

As an analogy to the DoS attacks that generate a large number of superfluous requests to exhaust the limited computing resources in communication systems, IDoS attacks generate a large number of feints to exhaust the operators’ limited cognition resources (e.g., attention, reasoning, and decision-making) in human-in-the-loop systems. IDoS attack is a broad class of attacks and poses the following significant security challenges. First, the strategical feint generation exerts additional cognition load to the human operators and exacerbates alert fatigue in the age of infobesity. Second, IDoS directly target human operators in the Security Operating Center (SOC), which is regarded as the cyber immune system. Third, it requires operators of a high expertise level to identify feints and respond to alert timely in complicated systems. Therefore, there is a need to understand this kind of attentional attack, quantify its consequences and risks, and develop new mitigation methods.