Scalable Cyber-Physical Testbed for Cybersecurity Evaluation of Synchrophasors in Power Systems

The state of a power system can be expressed by the complex voltages of all buses[7]. State-estimation involves solving the network Quasi-Static model to find the states $x$ utilizing the digital and analog measurements $z_{i}$ from the system[25]. Power system measurement data $z_{i}$ can be expressed with relation to the system states $x$ and measurement error $e_{i}$.

Here, subscript $i$ represents the $i^{th}$ meter and $h(.)$ is the measurement function which expresses the relationship between measurements and state, $x$. For AC state estimation, this function is non-linear, requiring an iterative method like Newton-Raphson. On the other hand, this function is linear in the linear state estimation problem. As a result, the solution can be obtained just by matrix multiplication.

A Weighted Least Square(WLS) optimization problem can be formulated to solve this problem. By minimizing $J(x)$, we effectively choose $x$ that best ”fits” the measurements.

Here, $\sigma_{i}^{2}$ represents the variance of the meter measurement. For linear state estimation, both of the equation can be written in matrix form:

Here, $W=R^{-1}$ and $R=diag\{\sigma_{1}^{2},\sigma_{2}^{2},\sigma_{3}^{2},\ldots\}$ The optimization problem can be solved by first order optimal condition. So, the estimated value of $\vec{x}_{est}$ can be obtained by $\frac{\partial\vec{j}}{\partial\vec{x}}=0$ [2, 7]

Linear state estimate is just a matrix multiplication with contrast to AC state estimation but we need to build the $H$ matrix from system network[25].

Here $II$ represents voltage measurement bus incident matrix, $A$ current measurement bus incident matrix, $y$ represents series admittance matrix, $y_{s}$ shunt admittance matrix. These matrices were formed following the instruction from thesis[26].

One of the main goals of state estimation is to find out the bad meter measurements and eliminate that measurements. As the state estimation problem is intrinsically over-determined, there are more measurements than the state. So, bad measurements can be eliminated. For linear state estimation, the presence of bad data detection problem can be formulated as follows[27]:

$j$ follows the Chi-Square distribution, $\chi^{2}(K)$. Here, $K=N_{m}-Ns$, is known as degree of freedom, where, $N_{m}$ Number of total measurements and $N_{s}$, Number of total states [28]. The presence of bad data can be estimated by checking the condition if $j<T_{j}$ , No bad data in the system. $j\geq T_{j}$, else, Bad data exists. Threshold value, $T_{j}$ can be obtained from the Chi-Square table utilizing the degree of freedom(d.f) $K$ and meter accuracy, $\sigma$.

After detecting the presence of bad data in the measurement, it is also possible to identify the bad or poisoned meter by hypothesis testing.

if $\sigma_{Y}>3$, it identifies that $i^{th}$ meter measurement is bad. The same method that is used for bad meter detection can also be utilized to detect false data injection attack on meter measurements.

We developed the testbed focusing on the IEEE 13 Node test feeder, a three-phase unbalanced distribution system operating in 4.16kV[29]. The distribution system is designed in MATLAB Simulink and deployed in Opal-RT real-time simulator. Placing the $\mu$PMU is an optimization problem to minimize the number of PMUs. The paper [7] identified the minimum number of PMUs for IEEE 13 node test feeder. Following that, we placed 5 $\mu$PMUs in nodes 2, 3, 5, 9, and 10. These 5 $\mu$PMUs measure voltage at five buses and current at nine lines in a total of 13 voltages and 20 current measurements for three phases. One phase is considered in the case of a transmission system due to its balanced nature. However, due to the unbalance in phases, the distribution system poses another challenge. The authors in paper[7] demonstrated that the three-phase unbalanced system could be decoupled into three-state estimation problems and possibly calculated by parallel computing. Figure 2 demonstrates the $\mu$PMU placement with current and voltage measurements. The red diamond indicates the complex voltage measurements, and the green arrow indicates the complex current measurements. The yellow dashed border indicates the $\mu$PMU—a single $\mu$PMU measures multiple complex voltages and multiple line currents. We also added standard normal distributed noise with the simulated PMU measurements to mimic the real behavior.

Here, $\mathbf{N}$ represents the generated noise utilizing the standard normal distribution function $R(z)$, variance $\sigma$ and mean $\mu$. $S_{N}$ is the noisy measurement that mimics the real meter measurements.

Complex voltage and current measurements must be aggregated to the control center to perform linear state estimation. This task is accomplished through synchrophasor technology. it consists of Phasor Measurement Unit(PMU), Phasor Data Concentrator(PDC), communication network and control center[2]. Real-time GPS time-stamped electrical quantities are measured and transmitted to the control center through a suitable communication protocol. The latest synchrophasor standard has been split into two standards, IEEE Std 37.118.1-2011, which covers measurement provision, and IEEE Std 37.118.2™-2011, which covers data communication[30]. IEEE C37.118.2 protocol is an application layer protocol that is accepted by industries for phasor data transferring between PMU and PDC. Although IEEE standard C37.244-2013[4] deals with PDC, it did not impose strictly to use of IEEE C37.118.2 protocol for phasor data transfer. This standard recommended IEEE C37.118.2 or IEC 61850-90-5 for PDC data transfer. Both of the protocols have their unique features and limitations[2].

Four message types are defined in the IEEE C37.118.2 standard[30] such as data, configuration, header, and command. A sample communication scenario of IEEE C37.118.2 protocol has been presented in the figure 5. In the synchrophasor protocol, PMU functions as a server, and PDC functions as a client. In the server-client paradigm, a client initiates communication session. Therefore, PDC starts a session by sending a command that requests the PMU to send configuration message. Without a configuration packet, PDC cannot interpret the data packet. Then After receiving the configuration packet, PDC sends another command to start data transmission. PDC transmits synchrophasor data continuously at a fixed rate until further PDC commands stop data transmission.

We deployed all the $\mu$PMU, PDC, and network layer on the top of the virtual machine(VM), which gives us the advantage of scaling. So, We need to deploy four types of VMs to develop the whole system. Moreover, different types of VM has separate dependencies and setup mechanism, which is one of the challenges for scaling the system. To scale up and automate the VM deployment process, we leverage the power of Vagrant. It is developed by HashiCorp to create and manage a portable virtual machine environment. Vagrant utilizes Ruby programming language to write instructions for deploying all virtual machines in a single file named Vagrantfile[31].

We leverage an open-source Python library named pyPMU[21] to develop the $\mu$PMU. All the dependencies have been installed inside the virtual machine to mimic a real PMU. The $\mu$PMU is performing two significant tasks. First, it collects the phasor measurements from the distribution system that is running inside the Opal-RT; second, pyPMU encodes phasor measurements into the IEEE C37.118.2 packet. Finally, it is waiting for the command from PDC to initiate communication, just like in figure 5. Manual installing many PMUs is a cumbersome process, So we utilized the Vagrant automation tool to automate the virtual machine deployment process.

All the steps of $\mu$PMU installation is depicted in figure 3(a), and the script of the implementation can be found in the GitHub link: ”https://github.com/shuvangkar”. From VM deployment to dependencies installation, all the steps are handled by the vagrant script. As the Vagrant script is based on the Ruby environment, it supports basic programming syntax like for loop. The support of the loop in the Vagrant script elevates the scaling issue significantly. Hence, we implemented a base PMU device and utilized a ”for loop” to deploy the n number of PMUs. The command sequence of PMU deployment is essential because all the dependencies should be installed before changing the network card. Each PMU requires two network cards to work correctly. One network card works in bridge mode that connects with Opal-RT, and another network card works in internal network mode that connects the PMU with the smart-grid network. Then dependencies for the PMU, such as pyPMU, are installed in the script. Finally, static IP addresses are correctly set into both network cards to connect the physical layer with the cyber layer.

The pySynphasor module can be found in the link: ”https://github.com/shuvangkar”

As the pySynphasor can dissect and build IEEE C37.118.2 packets, we developed a simple PDC application utilizing that. We named the PDC application pyPDC. The pyPDc can receive data from multiple PMUs. The pyPDC is available at the following link: ”https://github.com/shuvangkar” The pyPDC being a python application gives us the advantage of deploying everything in a python environment utilizing vagrant script. The steps for PDC deployment is shown in the figure 3(b).

The vagrant script’s VM setting with dependencies is written down sequentially. Afterward, a simple vagrant command ”vagrant up” deploys the whole testbed except the CORE network. The script installs all the PMUs and PDC with dependencies and necessary setups within a few minutes. This automatic installation opens the real potential of the testing cyber-physical system in a hardware-in-loop and co-simulation environment as it does not require purchasing costly PMU and PDC hardware to test more extensive distribution or transmission systems. Both the PMU and PDC have critical network configurations in the testbed, which will be discussed in the subsection II-D.

Designing a highly scalable cyber-physical system in the virtual environment is one of the main goals of the research. So, a software-defined network that runs in real-time is required to connect all the virtual machines. CORE(Common Open Research Emulator) is an open-source network emulator tool that runs on top of Linux. It runs in real-time and connects many nodes, taking advantage of the Linux network namespace. CORE has python API and a graphical user interface for building the emulated network.

Figure 4 depicts the emulated network designed in the CORE emulator. The network is composed of routers, switches, and RJ45 connectors. Let us consider the router n13. The network under the router n13 is considered the substation network. Under the substation router, a switch connects all the devices in the substation, such as PMU, local PDC, relay, and RJ45 connector. The RJ45 connector is the interface between the virtual network and PMU. Therefore, the RJ45 connector is the main bridge between the independent host and emulated network. The independent host can be any physical or virtual machine like PMU VM.

The VirtualBox network needs to be configured in a particular way to connect any host with the CORE virtual network through the RJ45 connector. For that reason, configuring the network in VirtualBox is one of the crucial factors for modeling the network. VirtualBox has seven network modes such as (1) Not Attached, (2) NAT, (3) NAT Network, (4) Bridge Network, (5) Internal Network, (6) Host-only network, and (7) Generic Network[32]. Different networking modes has different application and different advantages and disadvantages. Internal network mode is suitable for this application as it allows connecting a particular VM with the CORE RJ45 ethernet port. While building the CORE VM, Promiscuous mode must be allowed while configuring the network interface cards.

Promiscuous mode allows the incoming traffic to pass the physical network adapter and reach the CORE virtual network adapter. So, without enabling this mode, CORE will not receive the packet from the rest of the PMU and PDC VMs. The IP address of the PMU VMs has to be set according to the network address of the substation router. Otherwise, the VM cannot connect with the CORE router. For example, the network IP of the link connecting the enpos9 ethernet port is 192.168.0.1. The allowable IP range is 192.168.0.2-192.168.0.255 for the PMU VM connected with the enpos9 RJ45 connector. The same rules apply to all the VM connected with the CORE network through the RJ45 connectors. Also, 192.168.0.1 will be the gateway address of the PMU. The network configuration mentioned above is automated in the Vagrant script. We only have to manually deploy the CORE network and configure and connect the VM with the RJ45 ethernet interface, just like the figure 3(c). It is also possible to automate the CORE VM deployment using CORE python API.

MAC address is the unique identifier assigned to the network interface card (NIC). The IP address is a 32-bit address that identifies a device in a local area network in the IPv4 system. The problem with IP addresses is that they might change depending on the network setting. So, to identify a device in the network, the MAC address plays a vital role because of its uniqueness. ARP protocol resolves MAC address from IP address in the local area network. Therefore, the ARP protocol is the bridge between layer 2(MAC address) and layer 3(IP address).

Consider a scenario in figure 6 where device D1 needs to send data to device D4. D1 only knows the IP address of the device D4. Therefore, D1 needs to know the MAC address of D4 to send data. Then, D1 sends a broadcast message to the local area network asking MAC address of D4. Afterward, all the devices in the local network will get the broadcast request. Only D4 will respond by attaching its MAC address. After receiving the MAC address, D1 caches the MAC address of D4 along with the IP address in its ARP table.

ARP protocol does not verify the responder address before caching into the ARP table. So, any device can send an ARP reply message without a query. That means it is not mandatory for D1 to ask for the D4 MAC address to get an ARP reply. Any host in the local area network can generate an ARP reply without inquiry. The adversary uses this vulnerability to pretend to be someone in the network. In the figure 7, attacker D3 sends an ARP reply to device D1, attaching the IP address of device D4 and MAC address of its own without any initial inquiry from device D1. So, the attacker, D3, is pretending to be D4. As a result, the ARP table of D1 will be poisonous. This technique is known as ARP poisoning. Because of ARP poisoning, whenever D1 sends any packet to D4, it will be forwarded to the attacker machine(D3).

Now, all the packets from D1 to D4 pass to the attacker machine(D3) after deploying the ARP poisoning script. We have written an ARP poisoning script in Scapy. The aprspoof is another ARP poisoning tool with the built-in command to perform ARP poisoning. Both tools work according to the mentioned method in figure 7. After deploying the ARP poisoning script, all the packets from D1(Alice) to D4(Bob) will be passed to the D3(Attacker). However, Bob(D4) will not receive the packet unless the Attacker enables the packet forwarding. Therefore, Alice and Bob can establish communication after enabling packet forwarding in the attacker machine. In the case of the Man in The Middle(MITM) attack, the attacker convinces two victims that they are directly transferring data with each other[11]. In this way, the MITM attack has been implemented in our testbed.

FDIA attack is the most dangerous attack that can disrupt the regular operation and lead to blackout, just like the 2015 Ukraine attack. In this attack, the adversary craft fully changes the meter measurement. We utilized Linux iptables, NetfilterQueue, and pySynphasor to implement FDIA in our testbed. The iptables is the basic firewall program of the Linux operating system. We developed a python script that combines all the tools and automates the FDIA attack process. The packet filtering mechanism provided by iptables is organized into three different kinds of structures; (1)tables, (2)chains, and (3)targets. The table allows processing packets in specific ways. Filter table decides whether a packet should be allowed to reach the destination, Mangle table allows to alter packet header, NAT table allows to route packet to different hosts on NAT network, Raw table is a stateful firewall. Tables have a chain attached to them, and the chain inspects traffic at a different point. Prerouting chain applies when a packet arrives. The input chain applies just before the local process. Forward chain applies to the packet that are routed through the current host. The Postrouting chain applies when the packet leaves the network interfaces. Finally, the target decides the fate of a packet, such as allowing, rejecting packets, or passing the packet to a queue. We created iptables rules such as way so that any packet passing through the FORWARD chain; will pass to the NetfilterQueue buffer(NFQ) just like depicted in figure 8(a).

After reading the packet from the NFQ buffer, the attacker inject false data in the measurements. As the pySynphasor module can intuitively build and dissect the IEEE C37.118.2 packet, It plays a vital role in injecting false data. It reads the network packet and builds an internal object structure in the Scapy framework. That is known as the internal representation of the packet. The internal representation is helpful in the Scapy framework as it allows the user to modify and redesign the packet. After modifying the packet, Scapy again builds the packet for the network. This network representation is the machine representation of a packet in Scapy terminology. Another critical point while rebuilding the packet after the injection is updating the packet fields that hold the packet’s signature in some form, such as IP length, TCP length, TCP checksum, and IEEE C37.118.2 CRC value.

Figure 8(a) depicts the direction and flow of the packet in Linux kernel space and user space. The red line indicates the path of the synchrophasor packets while passing through the attacker’s machine. Figure 8(b) presented the flowchart of the script that we have developed for implementing the FDIA attack in the synchrophasor system. The script begins with preparing the FDIA environment that includes (1) enabling packet forwarding, (2) resetting iptables (3) adding new iptables rules. Then, the script filters the TCP packet that is aimed at the target machine. Afterward, the script injected false measurements. The next part is a little tricky. As the packet changes after the injection, a few fields such as IP length, TCP length, IP checksum, TCP checksum, and IEEE C37.118.2 CRC need to recalculate again. Otherwise, the receiver machine will discard the packet. For a successful FDIA, the steps are (1) Capture packet from Linux Forward chain and pass it to NetfilerQueue buffer, (2) When packets are available on the buffer, accept them and inject false data using pySynphasor. (3) Recalculate checksum as packets are changing after injecting false measurements. (4) Release the packet to the Linux Forward chain.

The distribution system is intrinsically unbalanced. This unbalanced system can be realized as the three separate equations[7] for designing state estimation estimation problem. These three separate equations can be solved by parallel processing, reducing computation time. So, we formulated the three-state estimation problems and calculated the voltages and phase angles of the individual phases. Figure 9(a) and 9(b) presents the estimated and actual bus voltage and phase angle of phase A for IEEE 13 node test feeder. The results were generated based on the following scenarios. For phase A, buses 5,6 and 11 are not present. Therefore, 4 PMU measurements were used while calculating states. Seven current and four voltage measurements are comprised in total of $N_{m}=11$ measurements. Bus 7 and 8 are connected with a breaker, and bus 12 has no load. So, we need to estimate the voltage for bus 2,3,4,7,9,10,13. Therefore, the total number of states to estimate is $N_{n}=7$. The noise was added with the meter measurement utilizing the equation 8. $\sigma=0.05$ and mean $\mu=0$ were considered while adding noise for this result.

For detecting presence of bad data, we utilized the equation 6 and Chi-square distribution table. For degree of freedom $K=N_{m}-N_{n}=11-7=4$ and $\sigma=0.05$, the threshold value $T_{j}=9.49$ obtained from the Chi-square distribution table. Figure 9(c) represents the meter error for all 11 measurement calculated using the hypothesis testing equation 7. The calculated value of $j=7.7943<T_{j}$, that indicates there is no bad data in meter measurement. Also hypothesis testing results for detecting bad meter was presented in figure 9(c). For all 11 meters, the value of $\sigma_{Y}<3$ that indicates all meter measurements are good. It also concludes that there is no attack on meter measurement in this case.

pySynphasor module is built on top of Scapy. Scapy is an interactive packet manipulation program based on python. It has three types of packet representation: internal, machine, and human. Machine representation is the actual raw packet that will be sent through the network. Using internal representation Scapy manipulates the packet. Human representation is the human-readable representation in plain text. So, this representation is easy to deal with for injecting packets. Figure 10 represents the human-readable representation of IEEE C37.118.2 data packets. These packets were collected using the Wireshark tool. Wireshark stores these packets from PMU and PDC communication in a pcap file format. Then, pySynphasor reads the pcap file and dissects the packet just by one line of code. The figure 10 demonstrates that the packet dissection is done just by applying show() method on packet. It presents the packet in a human-readable format. The data packet has five sub-segments: an Ethernet Header, IP header, TCP header, IEEE C37.118.2 common frame, and IEEE C37.118.2 data frame. Data frame represent the phasors measurements in complex number format. Similarly, pySynphasor is capable of dissecting command, configuration, and header packet of IEEE C37.118.2 protocol.

The module is also capable of building the IEEE C37.118.2 packet from raw phasor measurements. The figure 11 demonstrated an example of intuitively building the IEEE C37.118.2 data packet just using a few lines of the python script.

The figure 12 presents an example of how CORE connects the PMU and PDC that were deployed in the virtual machine. Figure 12(a) depicts a sample CORE network deployed in VM for connecting PMU and PDC. Figure 12(b) presents PMU deployed in a VM that transfers phasor measurement after a specific interval. The PMU device deployed here is built using the pyPMU python module. This VM is connected to the RJ45 ethernet interface enp0s8 in the CORE network. Figure 12(c) is another VM connected with the enp0s10 RJ45 network interface. PMU Connection Tester application is deployed in this VM for verifying the connection with PMU through the CORE network. PMU Connection Tester plotted four phasors measurements when it received data from the PMU device. All the phasors measurements are flat, because PMU sends a constant value. So these results verify the successful connection of PMU and PDC through the CORE virtual network in our proposed testbed.

ARP poisoning mechanism is used to implement MITM attack in our smart-grid testbed. The primary technique of this type of implementation is that the attacker has to poison two victims’ ARP table. The figure 13 presents the results of ARP poisoning where PMU and PDC are deployed in a local network in two separate VMs. The figure 13 presents the ARP table of both victims before and after the poisoning. If we look into the ARP table of the PMU device, The IP address of PDC is 10.0.2.7, and the MAC address is 08:00:27:69:58:64 before the attack. The MAC address changed to 08:00:27:a7:1b:c3 in the PMU ARP table after the poisoning; that is the MAC address of the attacker machine. That means the PMU ARP table is poisoned, and all packets from PMU to PDC will go through the attacker machine. The same thing happened to the PDC ARP table.

After ARP poisoning, the attacker machine has access to the victim packet. It created a temporary queue for manipulating and forwarding the packet afterward. In the figure 8, we explained how the attacker gets access to the synchrophasor data from Linux kernel space to the user space through NetfilterQueue and iptables package. Afterward, utilizing the pySynphasor module, the attacker dissects the synchrophasor packet. The figure 14 presents scenarios of MITM and FDIA attack. Here, PMU and PDC are transferring packets normally. The left VM represents the PMU, and the right side represents the PDC and Center VM, which is the attacker machine. PMU and PDC are transferring data normally. By poisoning the PMU and PDC ARP table, the attacker now steals the session of the PMU and PDC. After deploying the attack script, it has access to the synchrophasor data. In this way, the attacker is eavesdropping on the synchrophasor packet.The center VM in figure 14 presents dissection of one of the synchrophasor data packet. pySynphasor module paved the way for real-time dissection of synchrophasor packets. The figure 14 also presents a scenario of false data injection attack. In the attacker VM, we can observe that the phasor measurements were [(2453+2444j), (2954+2780j), (2922+2079j)] before the attack, and after the attack, it was injected to [(2402+0j), (58218+ 2860j),(58218+12675j)]. The packet injection mechanism is just a few lines of pySynphasor commands, just like the figure in 11. As the packet contents are changing, it is mandatory to update the fields that keep track of the packet signature, such as TCP and IP packet length, checksum, and IEEE C37.118.2 CRC value. Updating of these signature fields is performed in the script. After the injection, the script forwards the packet to the PDC. This is how the testbed smoothens the different attack scenarios for testing large system deployment.

The testbed paves the way for testing different types of cyber attack detection in smart grid systems. The detection mechanism can be classified into packet-based and physics-based. Linear state estimation-based bad data detection technique is physics-based attack detection. Figure 9(d), 9(e) and 9(f) represents one of the scenarios of linear state estimation based bad data detection technique. Where the meter 1 measurement was poisoned by the attacker and bad data detection algorithms were performed in the control center. We mentioned earlier that for the degree of freedom $K=7$ and $\sigma=0.05$ The threshold value, $T_{j}=9.49$. But in the figure 9(f), the value of $j=19.6005>T_{j}$. That indicates that there is bad data present in the measurement. The 9(f) also demonstrates the result of hypothesis testing on meter measurements. The error value $\sigma_{Y}>3$ for meter 1. It proves that the attacker poisoned the meter 1 measurement.