On the Use of Reinforcement Learning for Attacking and Defending Load Frequency Control

LFC maintains power balance and grid frequency consisting of primary, secondary, and tertiary control levels [27]. The primary level employs droop-governor control to regulate frequency while the secondary level uses automatic generation control (AGC) to regulate the net interchange of power. Tertiary control provides additional frequency support mechanisms by restoring power reserve. Failure to regulate frequency can cause frequency protection devices (including ANSI 81U/O/R) to isolate power system equipment to protect them from damage sustained due to operation at abnormal frequency resulting in unwanted system reduction.

The implementation of LFC over a wide-area network with open communication protocols and minimal human supervision increases its cyberattack surface. Interference of LFC operation is possible by exploiting a variety of vulnerabilities in insecure legacy electric grid networks, open communication protocols and operating systems, as well as introducing malware through infected emails or USBs, performing supply-chain attacks, or capitalizing on disgruntled insiders [28, 29, 30]. These cyberattacks often aim to compromise critical measurement signals to ultimately destabilize the grid.

This work focuses on cyberattacks aimed at the unwanted triggering of frequency protection relays in power grids, which can initiate sudden power imbalance leading to grid instability, cascading failure, and blackout. Previous studies highlight the significance of investigating these attacks, showing that attackers can modify loads [26, 31, 32, 33] or tie-line power [26], inject disturbances to automatic generation control [17], or corrupt microgrid synchronization [34] to trigger protection devices and/or destabilize the grid. While these authors applied strong assumptions and/or knowledge of the system dynamics in the development of cyberattacks, in this paper we demonstrate that RL can synthesize attacks with zero to little prior knowledge.

In RL, an agent is trained through a process of trial-and-error to achieve optimal decisions or strategies in an environment of which the agent has zero to little prior knowledge [35]. Training an RL agent to attack the grid can yield novel unforeseen insight into system vulnerabilities and attack strategies. Establishing an RL problem within the context of synthesising attacks requires defining the environment (representing the cyber-physical system), and specifying what actions (representing attacks) the agent can execute in the environment and what environmental states it can observe and make decisions based on. A reward function is formulated to steer the agent into taking actions that achieve the attack goals.

The agent contains two components: a policy and a learning algorithm. The goal of the policy is to map environment observations to actions that maximize rewards. The policy can involve an actor, critic, or actor-critic function approximators. An actor $\pi\mathrel{\mathop{\ordinarycolon}}S\rightarrow A$ maps environment observations $S$ to actions $A$. A critic $Q\mathrel{\mathop{\ordinarycolon}}(S,A)\rightarrow R$ maps action-observation pairs to (predicted) discounted cumulative long-term rewards $R$. The learning algorithm continuously updates the policy to find the optimal policy. Learning happens in episodes: simulations that expire after the RL agent achieves a certain goal or a maximum simulation length.

In this work, we employ deep deterministic policy gradient (DDPG) RL as it is the simplest learning algorithm compatible with continuous actions and observations – to highlight the feasibility of implementing our work offensively as a cyberattacker or defensively for electric grid cyber-physical security.

We apply the swing equation [27] to model LFC. The following state-space system expresses the linear load-frequency dynamics:

where the state is

$e$ is governor-droop control signal, $P_{g}$, the governor output, $P_{m}$, the mechanical power, $\omega$, the system frequency, $\hat{\omega}$, the frequency measurement, and $\hat{\dot{\omega}}$, the rate of change of frequency measurement.

The input vectors $\bm{u}$ and $\bm{p}$ represent the inputs to the systems during normal operation and attacks, respectively. The input vector:

includes change in the demand, $P_{L}$, and tie-line power, $P_{tie}$, if any. The attack vector:

includes actions the attacker can execute as enumerated in our threat model, including corrupting frequency measurements to the control center ($p_{1}$), corrupting generation control signals ($p_{2}$), corrupting tie-line power measurements ($p_{3}$), and compromising load switching ($p_{4}$). The state matrices in (1) are elaborated in the Appendix.

Frequency relays protect generators from damage sustained during operation in unsafe frequency by ensuring the frequency and its rate of change are within a safe set

Frequency relay functions include under-(UF), over-(OF) and rate-of-change of-(ROCOF) frequency. Recommended settings per IEEE 1547 [37] are listed in Table LABEL:table:relaysettings.

We assume that the attacker can observe the system frequency. We model actions $\{p_{1},p_{2},p_{3}\}$, entailing corruption of communication, as continuous-value actions. In practice, the attacker’s ability to inject an attack signal is limited due to physical constraints, constraints imposed by the communication protocol, or the need to avoid detection by bad data detectors. Hence, we assume that the attack vector $\bm{p}$ is bounded. For load switching, if the attacker compromises an aggregate load $P_{sw}$ and can switch on and off portions of the total load, then $p_{4}$ can also be modelled as continuous-value action, with $p_{4}\in[0,P_{sw}]$. If the attacker can only switch on and off the entire load, then $p_{4}$ is a discrete-value action $p_{4}\in P_{sw}\times\{0,1\}$. We discretize the DDPG RL actions in this case as follows:

In Algorithm 1, we adapt DDPG to attack LFC. Here, the RL agent observes the system state $S=(\hat{\omega},\hat{\dot{\omega}})$ and influences the power system (1) by injecting an attack action $A$ through the input vector $\bm{p}$. The learning objective is for the RL agent to learn a policy $\pi\mathrel{\mathop{\ordinarycolon}}S\rightarrow A$ to force the states in $S$ outside the safe set $\mathcal{S}$ (5), effectively triggering a frequency protection device. The attacker can then execute this policy to attack and destabilize a real system.

We collect all episodes in a dataset for training the supervised-learning attack detector. The DDPG neural network architecture is detailed in Table LABEL:table:ddpgNNarchitectures in the Appendix.

Unsupervised machine learning methods detect potential cyberattacks by learning patterns and regularities in normal operation data and flagging anomalies. Due to the lack of labelled cyberattack datasets, unsupervised learning approaches, particularly autoencoder-based detectors, have seen increased application for attack detection [38].

Autoencoders consist of a deep neural network – partitioned into an encoder and decoder – that is trained to reconstruct its input data. The encoder maps the input normal operation data to a compressed hidden representation based on regularities in the data, and its decoder attempts to map this representation back to the input data. When the trained autoencoder is applied to new data, a large variation between the data and the autoencoder’s reconstruction indicates an anomaly in the data, which can then be classified as an attack.

To simulate normal operation data, we model the change in power demand as a random process following the work in [39, 40]. We model the change in demand as a 1 second-per-step random walk sampling from a Gaussian distribution $\mathcal{N}(0,\sigma_{1}^{2})$ superimposed on a 5 minute-per-step random walk sampling from a Gaussian distribution $\mathcal{N}(0,\sigma_{2}^{2})$. Fig. 1 illustrates the change in load power as simulated by this random process and the resulting frequency and rate of change of frequency deviation.

We develop an autoencoder to detect anomalies in the time-series consisting of the governor-droop control signal $e$ and the frequency measurement $\hat{\omega}$. To prepare the training dataset, we run a long simulation of normal system operation and then we randomly crop $N$ portions. The portions are sampled at 50 milliseconds per sample and have variable (time) length. Each portion is a vector $X_{i}\in\mathbb{R}^{2\times n_{i}}$ vector representing a time-series of ($e,\hat{\omega}$), where $n_{i}$ is the number of samples in the portion.

The overall autoencoder can be expressed as a mapping $f_{ae}\mathrel{\mathop{\ordinarycolon}}X_{i}\rightarrow\hat{X}_{i}$ between the input data $X_{i}$ and its reconstruction $\hat{X}_{i}$, where $f_{ae}(X;\phi)$ is a neural network with parameters $\phi$. We develop an LSTM neural network-based autoencoder, described in Table LABEL:table:lstmNNarchitectures of the Appendix, given the suitability of LSTM networks for time-series. Training the autoencoder seeks to minimize the mean square error between the data and their reconstructions:

Next, a threshold can be selected based on the maximum reconstruction error (seen in the validation set) to differentiate between normal and anomalous data. When the autoencoder is input new data, if the reconstruction error is below this threshold, the data is labelled normal; otherwise, anomalous.

Supervised methods are trained to distinguish between normal operation and attacks by training on labelled system data. To collect labelled data, we augment the normal operation dataset (used to train the autoencoder) with the RL-generated dataset and label the data into 4 categories: (1) normal operation, and attacks that (2) do not trigger protection, (3) trigger under or over-frequency protection, and (4) trigger rate of change of frequency protection. Each record in the dataset includes a $X_{i}\in\mathbb{R}^{2\times n_{i}}$ vector representing a time-series control signal and frequency measurement and a label $\ell_{i}\in\mathcal{C}=\{1,2,3,4\}$.

We train the supervised attack detector to classify the data to their correct labels. The attack detector consists of a neural network $f_{ad}\mathrel{\mathop{\ordinarycolon}}X_{i}\rightarrow\{P_{i}^{(c)}\}^{c\in\mathcal{C}}$ mapping the input data $X_{i}$ to the probability of $X_{i}$ belonging to each category. $P_{i}^{(c)}$ is the probability of $X_{i}$ belonging to category $c$. The category with the highest probability is chosen as the label for instance. Training the detector seeks to minimize the cross-entropy loss:

Again, we develop an LSTM neural network-based attack detector (see in Table LABEL:table:lstmNNarchitectures in the Appendix) given its suitability for time-series and for comparison between the supervised and unsupervised attack detection methods.

To deploy the anomaly and attack detectors, the generation’s governor intelligent electronic devices (IED) can be upgraded to collect measurements of the governor control signal and local frequency. The IED will store the last $n_{i-1}$ measurement samples, add the latest sample, and perform the neural network computations to classify the system data. The classification can be communicated to the grid operator SCADA system to alert on attacks, or actions can be programmed into the IED to autonomously mitigate detected attacks.

Recall that we employ a DDPG RL agent to inject false data into any of the inputs $\{p_{1},p_{2},p_{3}\}$ of a single-area system while observing the frequency and its rate of change. The following results are specific to corrupting the frequency measurement to the control center ($p_{1}$); but are generally applicable to the other inputs. Following experimentation, we list the following decisions we made in the design of the RL agent:

• •

  We bound the RL action space, representing the injected frequency bias, to $[-0.1,0.1]$ pu. The large action space makes it easier and faster for the RL agent to learn successful attack strategies and generate a larger variety of attacks. Smaller bounds lead to longer convergence times during training. After training, the RL action space can be scaled down to smaller, more practical bounds to destabilize vulnerable power systems. For example, the RL agent’s actions in the detailed model time-domain simulations (Fig. 4) are restricted to the range $[-3.5,2]/60$ pu to match the frequency range ($56.5$ to $62$ Hz) in which the generator regulates the system frequency (refer to Table LABEL:table:relaysettings).

• •

  The large action space allows the agent to quickly discover simple bias attack to trigger UF or OF protection. We employ reward function (17) to encourage the agent to discover more complex attacks. The reward function is illustrated in Fig. 3. The safety set $\mathcal{S}$, which the agent attempts to force the system to exit, is inside the area depicted by the red square. Function (17) rewards the agent for increasing the rate of change of frequency towards and beyond the ROCOF relay setting while maintaining the frequency deviation small. Additionally, the agent attains a high reward of $+20$ when the ROCOF relay trips and a high penalty of $-20$ when the either of the UF or OF relays trips. Without the penalization, the agent continues to prefer simple actions that trigger UF or OF protection. Our experiments show that generally following the above guidelines facilitates RL training.

• •

  We limit each episode to 15 seconds to encourage the agent to destabilize the system quickly, and end the episode when the agent succeeds in triggering protection.

The agent generates an oscillatory frequency bias to excite the mechanical eigenmode of the microgrid, leading to generation tripping in vulnerable microgrids. Fig. 4 shows (detailed-model) time-domain simulations of two vulnerable microgrid testbeds, each with a different eigenmode frequency. The top plot in each column depicts the injected attack signal, with the frequency and rate of change of frequency deviation plot below it. The horizontal red dashed lines indicate the ROCOF protection relay bounds, which trigger the corresponding relay function and cause generation tripping when exceeded. For demonstration purposes, the relay triggering is suppressed to continue to demonstrate the RL agent’s attack strategy. The agent successfully triggers the rate of change of frequency relay function in the systems.

Fig. 4 demonstrates RL agents’ adaptability. Being data-driven, they can easily adjust their attacks to destabilize different systems. The agent utilizes easily available system frequency measurements to tailor the frequency of its injected attack signal to the mechanical eigenmode of the system.

The RL agent learns to execute load switching attacks by manipulating the system load through $p_{4}$, while monitoring the frequency and its rate of change. We use reward function (18) to incentivize the agent to increase the frequency or rate of change of frequency deviations, with high rewards of $+20$ earned when any of the UF, OF, or ROCOF relays trip. The reward function is illustrated in Fig. 5. The change in the reward function (from (17)) is attributed to the difficulty of tripping UF/OF protection with switching attacks.

Fig. 6 shows (detailed-model) time-domain simulations of a fixed load switching attack, wherein 464 kW (0.18 pu) of MG1’s load is switched on (1) and off (0), thereby exciting the mechanical eigenmode and eventually tripping the generation. We present examples of aggregate load switching attacks later in Fig. 8.

In this section, the RL agent is trained to generate a large attack dataset to train the supervised-learning attack detector. To generate the data, the agent is trained with more emphasis on increasing exploration and delaying learning convergence. Figs. 7 and 8 showcase a variety of frequency measurement corruption and load switching attacks, respectively, collected during RL training that destabilized the LFC model. Although these attacks are generally not optimal (e.g., in terms of time-to-failure), they are still able to trigger protection relays and therefore warrant attention.

We gather 4000 records in the dataset, equally distributed among the 4 categories outlined in Section 4.4, and allocate 15% and 30% of the data for validation and testing, respectively. The detector’s performance on the test data is presented in the confusion matrix in Fig. 9, with an accuracy of 98.4%.

Fig. 10 demonstrates the reconstruction of the trained autoencoder. The left plot shows a portion of control signal and frequency measurement collected during normal operation. The right plot shows the autoencoder’s reconstruction of the data. By learning the patterns of data during normal operation, the autoencoder is able to reconstruct the data with low error. Fig. 11 is a frequency histogram of the autoencoder’s reconstruction mean absolute error on the validation data. For comparison purposes, the test and validation sets are the same as that used to compute the supervised detectors’ accuracy. Based on the histogram, we select an error value of $0.2$ to classify between normal and anomalous behavior, which yields $100\%$ accuracy classifying between normal and attack behavior.

When comparing the unsupervised anomaly detector’s accuracy to that of the supervised attack detector in classifying normal and anomalous (comprising successful and unsuccessful attacks) operation, the detectors are comparable – at 100% and 99.8%, respectively. If, however, we compare their accuracy in classifying behavior preceding relay triggering (comprising successful attacks) and behavior that is not (comprising normal operation and unsuccessful attacks), the anomaly detector’s accuracy is 74.5% compared to 98.6%. This echoes a major drawback of unsupervised methods, which is their high false alarm rate for safe anomalous events that are difficult to exhaustively include in their training data, even when these events do not have any impact on the system. We discuss consequent implication and opportunities further in the discussion.