Distributed Software-Defined Network Architecture for Smart Grid Resilience to Denial-of-Service Attacks

Cyber-related smart grid (SG) system security has been a growing concern over the last decade. The current focus of research for power grid cybersecurity is operation technology (OT) and information technology (IT) systems reliability, whereas the development of an interdependent SDN management framework for SGs communications is still emerging. Traditional power systems are often protected by isolated and uncoordinated equipment that provides ad hoc solutions to each protection challenge. Other solutions are based in ongoing research for power system state estimation [1]. Although these techniques can be effective, these technologies are not integrated to communicate online with one another, and thus they are vulnerable to distributed cyber-attacks, denial-of-service (DoS), distributed denial-of-service (DDoS), man-in-the-middle (MITM), and false data injection (FDI) [2]. These types of attacks can influence data from several layers of the grid’s physical structure and ultimately disrupt system service. A unified methodology to recognize multiple attacks conducted from various layers inside the SG is still to be realized.

In our previous studies [2, 3, 4], we have explored the use of cross-layered data from both the communication and power grid layer for a Cross-Layer Ensemble CorrDet with Adaptive Statistics (CECD-AS) model to identify and characterize various cyber attacks. We have also introduced software defined networks (SDNs) as a possible underlying architecture to facilitate SG cybersecurity [3, 5].

This work is a continuation of our previous study [3], and provides a statistical analysis of a distributed flat topology for the SDN. We further present a comparison with the common alternative controller, the POX controller [6, 7, 8]. Several studies such as [9, 10, 11] have suggested a distributed SDN architecture for smart grids, but none provide a comparison of performance between distributed SDN framework and the single controller approach used in the POX controller-based literature. This work seeks to provide such a comparative analysis, and makes the following specific contributions towards the state-of-the-art:

1. 1.

   A benchmark study that compares and analyzes distributed versus centralized SDN management layer solutions for Smart Grids.

2. 2.

   A benchmark study for Smart Grid SDN frameworks to evaluate throughput resiliency of distributed vs centralized SDN frameworks against DoS attacks.

The remainder of the paper is organized as follows. Section II presents background information on the SDN framework and the network statistics used, while Section III contains data flow information of the framework, and provides the implementation aspects. A case study is presented in Section IV, followed by concluding remarks in Section V.

Figure 2 illustrates the distributed SDN architecture to manage the framework and which incorporates the aforementioned characteristics. The selected distributed flat SDN architecture divides the network horizontally into multiple areas or clusters, and each cluster is handled by a single controller. The SDN controllers in this architecture are interconnected and each controller has a global view of the network. ONOS can deploy distributed SDN controllers that utilize the flat architecture to achieve a logically distributed software-defined network [18].

To emulate and implement the D-SDN, first the Atomix and ONOS clusters are built using Docker containers in a local virtual machine (VM) with 6GB of RAM and 5 logical cores of AMD Ryzen 7 1800x installed on a local personal device using ONOS version 2.3, Openflow version 1.3, and Atomix version 3.1.5. Then, 118 hosts and 45 Open vSwitches using OpenFlow 1.3 were constructed using a Python script and Mininet 2.3 APIs. Open vSwitch is an open-source distributed virtual multi-layer switch solution and one of the most widely used OpenFlow implementations.

The smart grid’s communication layer is a three-controller SDN framework. Several approaches could be used, such as a hierarchical model or vertical architecture. In this approach, a root controller manages the local controllers, and each local controller manages a network domain or cluster. The root controller coordinates local controllers. To perform inter-domain functions, each local controller queries the root controller for global network state information, which adds latency and still provides a single point of failure [19]. To avoid a single point of failure, a flat, distributed topology is used where the three controllers share the same network authority. We also use ONOS to control packet flow in the smart grid system. Unlike the hierarchical model, that does not connect local controllers, controllers in the flat model have direct connectivity, and every controller has equal access to the network. Therefore, ONOS can quickly redistribute controller load to optimize network performance. Furthermore, ONOS and Atomix can dynamically move workloads from a down controller to the remaining controllers to reduce single points of failure. Atomix is a reactive Java framework for constructing scalable fault-tolerant distributed networks and manages our ONOS controllers as illustrated in Figure 2. It is in charge of ONOS cluster management, service discovery, and data storage. Because of the Atomix framework, ONOS controllers may be rapidly discovered and removed if needed.

The design of a logically distributed framework with low latency and improved resilience of the communication layer of the IEEE-118 bus system for a smart grid is of interest. To incorporate these characteristics into the SDN framework, it was realized a need of a controller that could respond to network demands swiftly, provide global oversight to reroute traffic based on link status, and had a proven performance in both literature and industry. Given these factors, the ONOS controller was chosen.

A case study was performed to compare the performance of the proposed ONOS managed flat, three-controller, distributed SDN (D3-SDN), against the POX controller approach. The results are presented in Figure 3, Table I, and Figure 4.

First, to evaluate the scalability of D3-SDN we used CBench, a popular controller benchmark tool to test and record throughput performance. These results are shown in Figure 3. For this experiment, the controller configurations ranged from 3 - 192 switches. As shown in Figure 3, the ONOS controllers have a higher initial throughput and maintain a higher throughput as the demand for more switches increases. As the number of switches that must be accounted for by each controller increases, the throughput performance of both controllers decreases. However, ONOS performs more effectively than POX overall.

Next, we compared the latency induced by each approach, based on the analysis in Section II-B. We determined the propagation delay, $D_{PD}$, for the emulated testbed by estimating the line lengths of the IEEE 118-bus system based on the per-unit to total reactance transformation presented in [20]. From the same reference we obtained each of the 117 line lengths based on area voltage level, after which an average line length was obtained. Then $D_{PD}$ was calculated by dividing the average line length by the recorded link speed of $200,000$km per second [21] to obtain an average $D_{PD}$ of 203.307$\mu$s. We add variability by incorporating Gaussian noise to each $D_{PD}$ then assigning the value to each link’s delay in our emulated testbed.

To measure the latency performance of our proposed ONOS D3-SDN approach and the traditional POX approach, we used the Sockperf software to perform a ping pong test. A ping pong test is when computer A sends a packet to computer B (ping) requesting for computer B to send back an echo response (pong). Sockperf operates by the client sending packets to the server, which then returns all or a portion of the packets to the client. It measures roundtrip time, $RTT$, between the two machines along a specific network path with packets of varying sizes. The latency for a particular one-way link between two computers is the $RTT$ divided by two. As our two nodes, we chose node 1 and node 112, the nodes with that required the most hops to communicate, as the Sockperf server and client, respectively and measured the latency performance on both $UDP$ and $TCP$ packets. Table I displays the results for average latency. The D3-SDN ONOS architecture reduces $UDP$ latency by $\sim 24\%$ and $TCP$ latency by $\sim 31\%$ over the POX controller approach, demonstrating that ONOS responds to network demands faster than POX.

Finally, to measure the throughput resilience during and after a DoS attack for the D3-SDN ONOS approach versus the traditional POX approach, we conducted a DoS attack on the frameworks using iPerf3 and hping. A server on node 1 and a client on node 2 are emulated using iPerf. To keep the comparison fair between controllers, for each attack simulation for both controllers, the client and server are instructed to have a transfer rate of 18 Gbits/sec for a simulation time of 30 seconds. After 10 seconds, the attacker, node 3, is instructed to ”flood” the client, node 2, with approximately $80,000$ packets as fast as possible, with an IAT of $<100$ microseconds. After 20 seconds, the DoS attack is stopped, and the simulation continues for 10 additional seconds. This attack scenario is repeated three times for each controller configuration and the results were recorded.

Figure 4 shows the results. For the first 10 seconds, there is a natural initial decrease and variability in network throughput due to the controllers spending the necessary time to learn the network routes. At 10 seconds, the attack begins and both controllers experience throughput decreases as the attacker begins to attack the client. This is also expected behavior. During the DoS attack, however, the POX controller lost 41.11% percent of its throughput, while D3-SDN ONOS lost only 29.77% percent. Therefore, during our DoS attack scenario, ONOS retains an average $\sim 11\%$ greater throughput than POX.

After the attack, the POX controller experiences the worst throughput from having been overwhelmed with the increased DoS demand. At its lowest point of the throughput measurements, the throughput drops to $\sim 6$ Gbits/s, which is a throughput loss of throughput of $\sim 66\%$ compared to the virtual $\sim 0\%$ lost experienced by D3-SDN ONOS approach at the time of simulation completion. This demonstrates the D3-SDN ONOS approach’s resilience and its capacity to handle increased network demands, albeit fraudulent in the form of a DoS attack in this instance. Furthermore, ONOS is able to recover quickly after these attack instances, which is advantageous when developing robust communication layers for smart grids.

Since POX fails in terms of maintaining throughput and the D3-SDN ONOS approach accommodates the DoS traffic, it is important to be able to determine that an attack has occurred and identify which type. In previous work [2], an algorithm called cross-layer ensemble corrdet is presented for cyber threat detection. Its performance is proven to exceed state-of-the-art physics-based and machine learning-based methodologies. In our previous study [4], the algorithm’s pseudo code and implementation are described. Each row represents a point in time for the relevant measurements, whereas each column represents a grid measurement point. A a 691 x 10,000 matrix of data points is built in Mininet, where one row of data is considered a sample of measurements for the entire grid, and Mininet creates one data point in around 4 seconds, or 45 minutes for each network sample, or row of data. The ML model requires 10,000 rows of data, which poses a time restriction. To address this limitation, SimComponents, a network simulator tool, was used to reduce the amount of time spent obtaining the network data. The SimPy and component toolkit SimComponents are utilized in the communication layer in order to simulate the network traffic one would encounter in the ONOS-managed D-SDN. SimPy is a Python-based discrete-event simulation toolbox. Active components including packets, packet generators, packet sinks, switch ports, and port monitors may be simulated with this tool. The SimComponents toolkit is used to define and replicate various components and their features. SimComponent creates the data required for one data sample in around $0.80$ seconds, which is significantly faster than Mininet, allowing the completion of the necessary dataset to acquire the results for the CECD-AS algorithm technique, as shown in Table II.

The real-time CECD-AS algorithm works exceedingly well for a range of cyber threats mentioned in this research, as seen in Table II. The improved detection is the result of the integration (combined) data from communication and power grid.

SDN enables users to manage smart grid communications with greater visibility, control, and reactivity. Deploying a set of ONOS clusters with equal permissions reduces the possibility of a single point of failure that may arise when using a single controller. This study evaluates a distributed three ONOS controller managed SDN architecture for smart grids and compares it with the common alternative POX controller. The ONOS managed architecture significantly outperformed the POX controller framework by reducing latency by $>20\%$ and increasing controller throughput ten-fold, resulting in increased scalability and significant throughput resiliency during DoS attacks. These performance results demonstrate that the D3-SDN is the better option over POX implementations regarding cyber security of the smart grid.