Multiple Domain Cyberspace Attack and Defense Game Based on Reward Randomization Reinforcement Learning

With the deepening of the understanding of the concept of cyberspace, especially the concept of cyberspace, more and more researchers realize that cyberspace is affected by multiple domain action. Cyberspace can be thought of as merging in multiple domains such as the physical domain, the digital domain, the network domain and the social domain, underlining the multiple domain feature of cyberspace. In this paper, because the social domain mainly involves the social relationship between the defender (administrator) and the attacker, this paper does not discuss the social domain feature of multiple domain cyberspace, we define the multiple domain cyberspace includes physical domain, digital domain and network domain. The physical domain describes device information in the space, like room, building, computer terminal, server terminal. In physical domain, attacker and defender have enter room, out room, control computer terminal and other physical domain actions. The network domain describes interface, path, and action device information related to network transmission, that is impressions of the general cyberspace for most of people. At last, the digital domain describes digital information in cyberspace, like username, password, etc.

In addition, we have some network security protection rules in cyberspace, including physical domain security protection rules, network domain security protection rules and digital domain security protection rules. Physical domain security protection rules mainly describe how to prevent unauthorized access. Physical domain, which prevents an attacker from entering a specific space. Network domain security protection rules mainly describe the methods to prevent unauthorized access in network domain. Generally, access control lists, static routes, and VLAN partitions can be used to achieve network isolation. In this paper, the network domain security rules we set up are described as allowing data to pass through the source port address, the port on the destination address, and the service on the destination address. Digital domain security protection rules mainly describe the information to prevent unauthorized access. The main method is to encrypt the stored or transmitted information, that is, to decrypt the file requires a secret key or password.

The problem of cyberspace attack and defense is a complex one, but it can be described as a game problem from the perspective of policy. In the model, both sides of the attack and defense are defined as the attacker and the defender, respectively representing the attacker and defender in multiple domain cyberspace. We divide the time slice of continuous into time slices, each time one and can only contain a cyberspace state, but the state is likely to be the same in different time slices. In each time slice, attacker and defender can choose an action to change the present state of cyberspace, the goal is to make the defender’s total reward is maximum. When both the attacker and the defender select an action, the cyberspace state changes to the next state. The attacker and the defender check the current cyberspace state again, and then select an action based on their own policies to obtain rewards from the environment. The next cyberspace state is under the joint action of both attacker and defender. The purpose of proposed method in this paper is to make the defender get long-term, high and stable rewards in the multiple domain cyberspace attack and defense game.

Under completely rational conditions, both attacker and defender can estimate Nash equilibrium, which is generally considered to be the optimal strategy of both sides. But in practice, because of attacker and defender won’t execute Nash equilibrium strategy from the start, but in the reinforcement learning method of their attack or defense strategy was improved, which means that reach the Nash equilibrium is not the result. Because of the different learning mechanism and strategy, could reach Nash equilibrium to deviate from the equilibrium.

The game theory-based model constructed precisely solves the problem game in this aspect, that is, defenders choose from the list of actions or strategies allowed to be selected at the same time or successively, one or more times under certain rules under certain environmental conditions, and then implement them and obtain corresponding results. From the point of view of game theory, we can think of attack and defense process as a game between attackers and defenders. Each side of the game chooses its own actions based on its own observation of the cyberspace environment and the prediction of the other side’s actions. It can be considered that in the process of this game, the information of both sides is incomplete. As the game process progresses, both attacker and defender can obtain more information about the other side.

In the above game model, both the attacker and the defender are rational participants. For both sides, they have certain training data and know some attack policy or defense policy of the other side.

The game model is composed of each state of cyberspace attack and defense, the state transition between the actions and states chosen by attacker, defender in this state.

For each cyberspace state, we define the state as follows:

Constrained by bounded rationality, the benefits of attacker and defender from the cyberspace environment are set as their private information, unknown to the outside world, and the cyberspace state is set as the visible information of both sides, and the existing cyberspace state information can be observed by both sides.

Definition of time slice:

In general, each agent will observer cyberspace state, and then perform an action. We set the attacker and defender perform action at the same time, namely under the same condition, both attacker and defender choose an action and make cyberspace state transition to the next state. It does not exist under the same condition, attacker first action selection, following defender selects an action or the defender first selects an action and the attacker selects an action later.

Definition of cyberspace state transition model:

Probabilistic model is adopted to represent the randomness of cyberspace state transition. Markov property means that the current cyberspace state is only related to the previous cyberspace state, so our transition probability is, where, $S_{t}$ is the cyberspace state, and $(a_{t},d_{t})$ is the attack and defense actions of the attacker and the defender.

On the above basis, the relevant game model is constructed.

Definition 1: In cyberspace attack and defense, a standard form of game can be defined as: where $N=(attacker,defender)$ is the attacker and defender of the two agents participating in the game; $S=(s_{1},s_{2},...,s_{n})$ is the set of game states, which refers to the state of the cyberspace in the attack and defense game. $A=(A_{1},A_{2},...,A_{n})$ is the defense action set, where $A_{k}={a_{1},a_{2}...A_{m}}$ is the action set of the defender in the network state $S_{k}$. $R(s_{i},a_{i},s_{j})$ is the rewards the defender gets from the environment after executing the action $a_{i}$ in state $s_{i}$. $Q_{d}(s_{i},a_{i})$ is the value function of the defender’s state action, indicating that the defender reaches a certain state $s_{i}$ after adopting a certain policy in the defense, and the expected rewards after executing the action $a_{i}$ in the state $s_{i}$ after continuing to execute the action according to the policy.

Definition 2: The cyberspace attack and defense of the Nash equilibrium is refers to in the game process, after the cyberspace game model both sides of the policies choices of several times, to get both sides unwilling or does not change his policy alone or policy combination, in this kind of attacker and defender both sides are reluctant to change their policy is called a Nash equilibrium policy situation.

The Nash equilibrium in cyberspace game model can be expressed as follows: Nash equilibrium is the optimal attack and defense policy of both attacker and defender in the game state $s_{i}$, and the attacker and defender can maximize their rewards only by adopting Nash equilibrium.

Theorem 1: Nash equilibrium exists. Given a game model, if the state set $S$ and attack and defense action set $A$ is a finite set in the game, then there must be a Nash equilibrium.

Definition 3: Defense effectiveness: For the attack action $a_{i}$, the defender also adopts defense action $d_{i}$. For the effectiveness of the defense action $d_{i}$, we use $\alpha(a_{i},d_{i})$ to represent it, that is, when the attack of attacker is failure, $\alpha(a_{i},d_{i})=1$; if the attack of attacker succeeds, the defense is judged to be failed, $\alpha(a_{i},d_{i})=0$. In general, $0<\alpha(a_{i},d_{i})<1$.

Definition 4: Defense cost ($DC$): For the defender, a series of defense actions needs related costs, such as the restart service, adding a firewall policy, etc. Also, defense actions may be limit other user’s normal use, so we define the defense cost is sum of operating cost and negative defense costs.

Definition 5: Attack Reward ($AR$), represents the reward obtained by the attacker, is defined as

Where, $DC(a)$ represents the system loss cost caused by the attack action $a$, $DC(d)$ is the defense cost required by the defense action $d$, and $AC(a)$ is the cost required by the attacker to launch the attack.

Definition 6: Defense Reward ($DR$), represents the reward the defender gets from the environment for taking an action.

Where, $AC(a)$ represents the attack cost, the cost required by the attack action $a$, $DC(a)$ represents the cyberspace loss cost, and $DC(d)$ is the defense cost required by the defense action $d$.

Here, we set $AR+DR=0$, and we define multiple domain cyberspace game as a zero-sum game. In this model, the goals and returns of both sides are opposite. Although the rewards may be different, the benefits of both sides are complementary, so it is reasonable to adopt zero-sum game.

Attack and defense game model based on reinforcement learning is an intelligence technology, represented by deep reinforcement learning, which have been trained with an agent that can perceive and recognize the environment. With rules and learned experiential knowledge, defender pass through the attacker’s actions, to realize the choice of optimal action. Game model policy is the mapping of state to action in game model, and it’s the policy function $\pi$ in deep reinforcement learning.

On the basis of describing the process of game based on MDP, this section constructs the game model based on deep reinforcement learning. Through the interaction between the agent based on reinforcement learning and the simulation cyberspace environment, the defender game policy under game task is trained and generated. The intelligent game defense model based on reinforcement learning method is shown in Fig. 1.

Compared with the general single agent reinforcement learning, the single agent becomes the multiple agent. In this paper, we set one attacker and one defender here. On this basis, the defense algorithm is given, and we use DDPG as a defense algorithm for defender.

The algorithm first initializes the attack and defense game model and related parameters, defender agent will observer cyberspace states and perform defense actions from the algorithm. The next step is that the defender detects the cyberspace’s state, and the next step is to make defense decision, select related defense actions according to the current policy. Then it’s time to update the algorithm’s parameters based on the reward, with the goal of maximizing the defender’s reward. Every time the agent chooses an action, it will adjust its learned policy according to the action feedback and the new cyberspace state, and the cycle repeats. After constant trial and error and continuous interaction with the environment, the agent finally learns the optimal policy of game in the cyberspace environment.

In order to improve the learning efficiency of reinforcement learning and reduce the dependence of reinforcement learning on data, we introduce the reward randomization method to set the reward of defender agent, because in reinforcement learning training, agent’s reward function is difficult to set. Because in reinforcement learning, the reward function $R$ highly influences the polices that reinforcement learning may learn. Further, even if a policy is difficult to learn in a particular $R$, it may be easier to $R_{0}$ other functions in some $R$. Therefore, if we can define a suitable space $R$ on different state and draw samples from $R$, we may discover the ideal new policy to run reinforcement learning on some sample reward function $R_{0}$ and evaluate the effect of the obtained policy on the original game.

In this paper, we extend reward randomization to general multiple agent game. Considering the current game model is defined as follows: $S$ is the observable space state, $A$ is the action space of the defender, and the defender agent generates the policy $\pi(\theta_{a})$, which is the parameter of the policy. In this game model, both the attacker and the defender hope to get the maximum reward. The defense game model process is shown in Fig. 2.

In general, for Nash equilibrium, the desired outcome is to maximize the rewards for both sides. But when it comes to cyberspace attack and defense, we only expect the defender to maximize his reward. Therefore, this problem can be equivalent to: Under the above conditions, we need to find a policy that maximizes defender’s rewards. Because the two sides of the attack and defense is a zero sum game, that is the attacker gains the least.

The existing problem is that although the current decentralized policy method is a popular learning method for complex Markov games, the strategies it eventually trains are often suboptimal, that is, there will be a policy. We still can’t find this with our current methods.

Therefore, reward randomization is introduced into the attack and defense game model is based on the following reasons: if the relevant reward function in the game makes it difficult for reinforcement learning to discover optimal policies, it may be easier to achieve this goal if the reward function is interfered with. Therefore, we can define a reward function space $R$, so that we can train from the reward function sampling and calculate the associated optimal policy, instead of simply learning a fixed reward. Formally, we define a subspace of the reward function to guide the training of reinforcement learning in the game. By using the reward randomization method, we take $n$ samples and get corresponding policies in each training. Get the corresponding policies by calculating the original game. Finally, we can further improve the performance of the model by fine-tuning the policy. This training process is called reward randomization reinforcement learning. We define the linear mapping reward randomization as follows:

which $random$ is a randomly generated data in experiments and $0\leq random\leq 1$, $random(R_{n})$ means taking a sample from reward space $R_{n}$.

In general, feature-based reward functions are often found in relevant reinforcement learning reference. For example, in games with relevant distance, reward is usually set as the negative distance from the target position to the agent’s position plus the most successful reward. In the general reinforcement learning game, it can be a single reward for each agent, or the reward of the whole group, or it can be set up to give more emphasis to the agent.

Finally, we fine-tune the reward. Generally, fine-tuning has two benefits: Firstly, the policy found in the perturbation game may not maintain equilibrium in the original game, and fine-tuning can ensure the final results convergence; Secondly, in experiments, fine-tuning can further help to get rid of suboptimal policy.

We used Python to simulate a cyberspace environment in this experiment. There are a total of five places in this experimental environment. A physical space, such as a district, a college, is represented by the outermost layer. Room 1 is the terminal’s location, Room 2 is the security equipment’s location, Room 3 is the terminal’s location, and Room 4 is the service’s location. As shown in Fig. 3, there are five types of equipment: computer (T1 and T2, respectively stored in Rooms 1 and 3), firewall (FW1 and FW2, respectively stored in Rooms 3 and 4), security equipment (D1, stored in Room 2), router (R, stored in Room 4) and switch (SW, stored in Room 4), server (S1, S2, stored in Room 4) and their equipment connection relationships. S2 is where the security information is kept. If an attacker is able to gain access to S2 in order to obtain the security file, the attack has been successful.

Experiments are carried out in the above-mentioned experimental context. The following baseline approaches are used to demonstrate the effectiveness of our proposed method:

1. 1.

   Deep Q Network (DQN): DQN is a typical reinforcement learning that employs a neural network to predict Q-value and constantly updates the neural network to learn the best action path. In DQN, there are two neural networks. The target network, which is relatively fixed, is used to obtain the value of Q-target, while the evaluate-network is utilized to obtain the value of Q-evaluate.

2. 2.

   DDPG. This method is described in the section ”Related Works.”

Attackers are guided by knowledge-based rules, and their attack goal is to successfully attack the server. The following are the specific attack steps:

First, attacker enters space Room 2 and obtains the firewall FW1’s management service password, FW1 password;

Second, attacker uses device T1 or D1 to gain access to FW1’s management service; add the following access control list: allow T1 or D1 to gain access to T2’s management service, T2 manager;

Third, attacker uses T2 manager to obtain the password FW2 password for the firewall FW2 and the password S1 web password for the service S1 web;

Fourth, attacker opens the T2 S1 port, log in to the FW2 manager firewall, and adds the following access control list: allow T1 or D1 access to the S1 web and S2 web services;

Fifth, attacker visits the service S1 web using T1 or D1 and obtain the password S2 web password of S2 web;

Finally, the attacker can utilize T1 or D1 to get access to the S2 web service and obtain the security file using the S2 web password. At this point, the attacker is finished his attack.

For each method, we train 100 episodes. In each episode, 100 attackers were generated at random for training purposes in order to assess the method’s effectiveness. After 100 episodes, the DQN, DDPG, and our suggested reward randomization DDPG method were tested for performance. Meanwhile, we’ve selected the Assault Success Rate (ASR) as the criterion for evaluation. We define following:

The total number of attackers in this experiment is 100 in an episode.

Each time slices the attacker performs an action in the experiment, the attacker performs an action and the defender perform an actions and receives reward. Rewards for all agent actions were added as a training reward during the model training phase. During each training session, the model will be trained 100 times. The following are the additional parameters in the experiment: The attacker’s ratio in the experimental cyberspace environment is UP=0.4, the attacker’s ratio of launching an attack is AP=0.3, and each user performs no more than 60 actions. If attacker cannot complete the attack in 60 time slices, he will be log down this cyberspace environment. Also, if attacker attack successfully, he will get $AR=10$, defender will get $DR=-10$; if not, he will get $AR=-10$, defender will get $DR=10$. Also, if attacker have no harvest in 60 time slices and he will not captured by defender, he will get $AR=-5$, and defender will get $DR=0$.

Firstly, the game model was realized in the above experimental cyberspace environment. At the end of each episode of method training, we recorded the $ASR$. The experimental results are shown in the figure Fig. 4.

Secondly, in order to evaluate the performance of RRDDPG method, we recorded $DR$ in different methods, and the experimental results are shown in Fig. 4.

From the experimental results, with the increase of training times, the $ASR$ is constantly decreasing, while the defender’s $DR$ is constantly increasing, which indicates that the multiple domain cyberspace attack and defense game based on reinforcement learning can effectively improve the defense ability of defenders. At the same time, the method we proposed makes the attacker have the lowest $ASR$ and the highest $DR$, which shows the advanced performance of the our proposed reward randomization DDPG method.

Meanwhile, in the multiple domain cyberspace attack and defense game, from the experimental results, the following conclusions are:

1. 1.

   The multiple domain cyberspace attack and defense game system designed in this paper can provide a research direction for the next step of cyberspace attack and defense game intellectualization;

2. 2.

   A reward randomization DDPG method is proposed, it can improve the intelligent defense level of defenders in multiple domain cyberspace attack and defense games;

3. 3.

   In this model, we assess the defense and attack rewards and include them into the model such that the defensive option becomes more effective and more in line with real-world applications.