CybORG: An Autonomous Cyber Operations Research Gym

Autonomous Cyber Operations (ACO) is concerned with hardening and defending computer systems and networks through autonomous decision-making and action. We are motivated by the need to protect systems that may be isolated or operate in contested environments where skilled human operators may not be present nor have remote access, and to unlock the potential for rapid and wide scale cyber operations.

This setting has characteristic properties. ACO is adversarial and evolving: behaviour is appropriate when it best achieves objectives in the face of evolving adversary behaviour (tactics, techniques and procedures, or TTPs). Thus the setting incorporates blue team and red teams and terrain (i.e., hosts and networks). Addressing these together requires that ACO research should be concerned with both red and blue team decision making models that may co-evolve.

Further, ACO is dynamic, the terrain may change in uncontrolled or unpredictable ways or as the deliberate results of red and blue activity. Importantly it is also varied; whilst it might be interesting to solve for decision making models that are optimal for a given single scenario or configuration, computer security problems are varying and diverse. Consider by way of metaphor that we wish to solve Chess960 (random starting positions each game) rather than standard Chess. By generalising over diverse and dynamic scenarios, decision-making models might become resilient to variations in the environment, rendering them more adaptable and reliable.

Finally, ACO environments have high complexity and dimensionality, with very many interdependent and related components and large state (information relevant to decision-making and goals) and action (choices) spaces. Given that decision-making may be considered a mapping from state to action, this is generally a harder problem for larger state and action spaces.

These characteristics (adversarial, evolving, dynamic, varied, high complexity and dimensionality) suggest a learning approach for developing decision-making models that are trained and evaluated across a wide range of adversary behaviours and scenarios.

One possible approach is supervised learning, where a dataset provides examples of optimal decisions at each state of a game which may be used for training, validating and evaluating models. However, due to the aforementioned characteristics, collecting relevant datasets and adapting to evolving adversaries could prove problematic for supervised learning.

An alternative approach is reinforcement learning, in which an agent ¹¹1Throughout this paper we use the term agent as in [22], to paraphrase: a decision-making agent that seeks to achieve a goal in its environment. learns by interacting with an environment in a trial-and-error manner guided by rewards for achieving desirable results. In this way, the algorithm learns optimal solutions by trying to maximize its reward  [22]. Reinforcement learning can be very effective in high complexity, high-dimensional environments. Following this direction requires an appropriate gym or environment for training.

However, a typical problem in reinforcement learning is the requirement for a high number of samples and in many applications the resource cost and time taken to train in real world environments is prohibitive or impractical  [21, 1]. Often algorithms can take considerable time and resources just to begin converging on a merely adequate solution, with an optimal one still being some distance away  [13].

The adversarial and varied setting of ACO excacerbates the problems RL has with complexity. Training co-evolving red and blue team models and training across dynamic and varied scenarios compound these challenges.

A common strategy for overcoming this problem is to use simulations of a problem with lower fidelity and lower cost as a training ground and to then transfer the models to a higher fidelity environment [21, 13]. However, while simulations are often useful, they come with their own set of challenges, not the least of which is that it can be difficult to build a sufficiently accurate model to have any hope of real world application  [1]. Thus, the need for real world or high-fidelity, high accuracy environments remains.

Another concern when applying machine learning is overfitting. A model may produce solutions that work well for one specific problem and environment, but perform poorly when applied to similar problems or environments  [9, 1, 24, 6]. There are numerous strategies to combat this problem including sampling multiple environments  [24, 6], and varying evaluation conditions  [24], but these depend on the ability to modify and vary an environment. This further motivates the need for an ACO gym where the environments can be varied and dynamic.

This work is focused on a suitable environment for intensive reinforcement learning towards ACO. We are inspired by such toolkits such as OpenAI Gym  [4] (hence our adoption of gym) which provide a basis for learning experiments and a set of benchmarks with which to measure progress and evaluate competing approaches. We here propose a number of requirements appropriate for a gym for ACO research.

Firstly, we favour both simulation and emulation modes for playing through ACO scenarios. Simulation presents the most efficient means to achieve millions of episodes for reinforcement learning necessary for agents to train, whilst emulation presents the most realistic environment within which to experience and learn such that the derived models might behave appropriately in the real world. For intensive reinforcement learning, it may be appropriate to train a model in simulation (most efficient), and later optimise the model in emulation (most realistic), which confers an added benefit that that the simulation need not be highly accurate. Therefore we require the ability to play the same scenario in both simulation and emulation modes, with the same APIs presented to decision-making models and learning systems.

Given that we envisage intensive reinforcement learning over millions of scenarios, it is advantageous to leverage cloud computing resources for elastic, parallel and large scale experiments. This requires an ACO gym that may operate with and on commonly used cloud infrastructure. Given the potential need to play millions of games to train decision-making models, the toolkit should be as efficient and low cost as possible.

Informed by the OpenAI Gym  [4] and Atari  [2] environments, it is desirable for a library of scenarios or games to be available that provides a series of increasingly challenging and diverse settings and tasks as challenges and benchmarks. Necessarily for ACO, it is an important feature that each be controllable to incorporate variability and dynamics. We also require the scenarios to be adversarial, with red and blue teams competing to achieve their objectives and disrupt their opposition.

We envisage that such a gym would be appropriate for learning ACO decision-making models, but may also find other uses for operator training, human-machine teaming (performing with or against an ACO system), or discovering or evaluating tactics and capabilities. Therefore it is important that the gym and its API and interfaces are easy to use and understand.

The following section presents relevant work in cyber ranges, simulation and emulation environments that have been developed for purposes including research and training operators. The design and work-in-progress implementation of CybORG are described in Sections 3 and 4, with Section 5 providing an early evaluation considering the results of initial learning experiments. We conclude this paper with our conclusions and plans for future work.

CybORG’s simulated environment is designed to be fast and lightweight, able to run thousands of interactions per second on a single host. This is achieved using a finite state machine design that can model networks, hosts and actions informed loosely by the real world behaviour of these components. The simulator can model the stochastic nature of some actions, but is not required to accurately capture or model real world behaviour since the emulation is available for refinement of decision-making models, maximising simulation efficiency and reducing development effort. By keeping the simulation design lightweight, it is relatively cheap to scale.

To provide an accurate environment in which to perform reinforcement learning, CybORG’s design incorporates a high-fidelity emulation environment. By employing Virtual Machines (VMs) deployed and organized into realistic networks, with actual cyber-related software tools in real-time, it can provide real inputs, outputs and behaviour. The emulator design allows for rapid deployment, configuration and management of numerous permutations of scenarios with varying complexity and supports cloud deployments for elastic scalability, limited only by budget.

CybORG’s design supports reinforcement learning of a single decision-making model in both environments: a low-fidelity, efficient simulation and a high-fidelity, realistic emulation. Switching between the two environments is smoothly enabled by a common API and user interface for controlling experiments and accessing game state and action space.

Deploying a particular scenario involves specifying whether it will be simulated or emulated (a flag) and providing a scenario description file (YAML). The scenario file includes details for configuring the environment including hosts, networking and subnet information, and the set of actions available to red and blue agents. The scenario files are deliberately simple, requiring a minimum of information and employing many default behaviours to reduce the burden on users when creating files.

The supported host configurations and actions are specified in separate ’Images’ and ’Actions’ YAML files, respectively. Parameters for hosts include "Name", "OS", "Services", "Credentials" and an ID for an image of a deployable image in AWS (for the emulation).

Action parameters describe the set of possible red and blue team actions, constituting the action space. By combining different network architectures with selections from the available images and actions it becomes possible for users to generate and permute a wide variety of different scenarios.

A network diagram of an example scenario that is included with CybORG and that can be deployed in both the Simulation and Emulation components of CybORG is shown in Figure  3.

The emulation currently uses Amazon Web Services (AWS) with virtual machines to create a high fidelity cyber security environment, though it is intended that CybORG support a wide range of cloud/IaaS options.

The simulation environment of CybORG provides a lightweight and efficient model for ACO research. It is implemented as a script employing a FSM that uses classes for hosts and subnets to simply model a computing environment with security tools.