Faster than Real-Time Simulation:
Methods, Tools, and Applications

Due to the complexity of power systems, it is challenging to simultaneously solve sets of nonlinear ordinary differential equations (ODEs) (e.g., required for electromagnetic transient (EMT) simulations), or differential-algebraic equations (DAEs) (e.g., necessary for transient stability (TS) simulations), for large-scale EPS without exceeding the real-time time-step. To improve the computational performance of simulations, a parallel computing mechanism is examined in [4]. The HPC-based model is designed to improve the execution time by implementing a parallelized computing process for the models of the EPS under test. The proposed FTRT implementation reaches a solution 3 times faster than a conventional real-time simulation. Namely, the benchmark used is the Western Electricity Coordinating Council (WECC) power system that is composed of about $4,555$ buses and $3,000$ generators supplying electricity to $71$ million users. Similarly, a fine-grained relaxation algorithm (FGRA) is demonstrated in [2], which can solve the nonlinear DAEs using parallelized and pipelined hardware. According to the authors, the hardware emulation of the two case studies is accelerated by $134$ times, with a total $7,462$ clock-cycle hardware latency.

A reconfigurable hardware-based simulator is developed in [5] to achieve EPS simulation in FTRT rates. The proposed method adaptively matches the hardware architecture of the computation engine with the power system structure representing the mathematical system model. The computation engine enforces parallelism at the sub-systems/components, equations, and primitive operations level. HPC can support the parallel general Norton multiport equivalent (PGNME)-based domain decomposition method, enhancing the performance of FTRT TS simulation for large-scale systems [6]. The PGNME technique decomposes the power system network into sub-systems that can be evaluated in parallel. The simulation results of the approximated WECC achieve speeds 10 times faster than the benchmark. Furthermore, a variable time-stepping scheme is utilized to accelerate FTRT in [7]. In this scheme, the time-step of each sub-system is different, reducing the overall processing latency. However, all the sub-systems need to be synchronized (regardless of their respective time-step). Therefore, the time-step of each sub-system needs to be linearly proportional to the global simulation time-step. For example, to achieve synchronization between sub-systems with longer time-steps (e.g., generator models, lighting transients), sub-systems with shorter time-steps (e.g., only transmission lines) might be computed several times.

Besides the improvement of computing efficiency, the prediction of the system behavior is another advantage of FTRT simulations. From the cybersecurity perspective, the authors in [8] introduce an intrusion detection scheme for maliciously altered data based on a distributed dynamic state estimator. The proposed detection scheme is achieved through a command authentication method that leverages a FTRT implementation. Thus, the prediction and protection against vulnerabilities are ensured by determining the security impact that specific commands can have on the system model. Another prediction-based example is presented in [3], where the authors predict FIDVR events that will occur in a short-term horizon (less than $30s$) via FTRT dynamic simulations. The proposed algorithm utilizes the collected data from phasor measurement units (PMUs) and monitors the topological changes of the system for potential faults that could trigger FIVDR events (e.g., line outage faults, three-phase short circuit faults, etc.). Once an event is detected, the prediction algorithm is activated to assess the post-fault behavior. Leveraging FTRT for the prediction of OOS condition of turbine generator units is demonstrated in [9]. An OOS condition occurs when one of the generators experiences a large angular difference, which could result in severe oscillations in the voltage, current, and torque measured at the system level. This study shows that the proposed mechanism can detect the OOS $5.86s$ faster than a traditional real-time simulation-based implementation for the New England IEEE 39-bus system.

Many research works focus on FTRT simulation for preventing threats targeting power systems. For example, the authors in [10] propose mitigation strategies to circumvent short-term voltage instabilities caused by FIVDR. Namely, an under-voltage load shedding scheme is utilized to respond to FIVDR events and maintain system stability. The minimum amount of load that needs to be shed is predicted leveraging FTRT simulations and a digital replica of the system. Researchers in [11] present a vulnerability named subsynchronous interaction (SSI) introduced by capacitive series compensation when boosting the capacity of transmission lines in long-distance power delivery. SSI can result in severe overvoltage events and trip the associated transmission facilities [19]. In order to negate the impacts of SSI, such as immediate oscillations in the shaft torque, the authors present a method to mitigate the influence of SSI in a hybrid AC/DC grid. Notably, the grid topology of this case study supports wind farm integration and utilizes FTRT dynamic simulations to predict crucial control parameters (e.g., active/reactive power flows and injections). The reported hardware computational overhead occupies $4,174$ clock cycles.

The benefits of FTRT simulations can assist other disciplines by ensuring awareness and enhancing the system operation. Typical applications include building emergency management strategies, prediction/detection of natural emergencies, and improving the performance of specific system mechanisms. For example, the authors in [12] develop a fast fluid dynamics method to compute the airflow and temperature distributions in buildings. The proposed method can predict the airflow patterns $4$ to $100$ times faster than real-time simulation. This, in turn, can be used to enhance the deployment of emergency management systems designed to predict fire, smoke, or contamination in the airflow of buildings. Moreover, in [13], the authors show that FTRT simulations can be used to model emergency navigation systems designed to provide ‘optimal’ evacuation routes and effectively increase human survival rates during unexpected disasters. A fire-related scenario is used to examine the efficacy of the proposed building evacuation algorithm. When a person discovers the fire, the current fire location can be identified by taking snapshots from the actual areas inside the building. The photos will then be transferred to image processing servers and activate the emergency navigation system. The fire spreading model is simulated on a cloud-based system that predicts the fire spreading path. Based on the initial distribution of evacuees, the optimal routes can be identified using FTRT simulations.

FTRT simulations have also been employed in wildfire propagation studies, forecasting both the size and the shape of the burnt area during natural disasters [14]. Although the computational overhead increases with the system size, FTRT simulation can predict the fire dynamics in $58s$ for every minute of the real-time wall clock. Another application of FTRT simulation is reported in [16], where a saliency-maximized audio spectrogram is proposed to enable audio browsing and detect acoustic events $10$ times faster than real-time simulations. Similarly, for image processing, FTRT simulation is used for 3D facial alignment, where models are capable of landmarking approximately $170$ faces per second [17]. FTRT simulation has also been employed in the acceleration of training of experienced or inexperienced personnel (e.g., employees in a water plant or laymen) using the drinking water treatment plant simulation models presented in [18].

Models represent mathematical equations and/or data that are used to explain and predict the behavior of systems. For complex systems, such as EPS, both nonlinear ODEs and DAEs are used to represent them. A general formulation of such system equations is shown in Eq. (1), where $\mathbf{x}$ represents the state variables vector (e.g., generator angle, generator speed), $\mathbf{y}$ is the vector of variables (e.g., bus voltage, bus angle, or load flow variables), $\mathbf{f}$ represents a derivative function of the state variable $\mathbf{x}$ (e.g., the dynamic behavior of generator flux linkages, rotor angles, control states, and dynamic load states [20]), and $\mathbf{g}$ denotes a function of the outputs. In this subsection, we present 4 modeling approaches leveraged by FTRT implementations. Although the mathematical formulation for FTRT algorithms does not differ from conventional real-time simulation modeling, different implementations are utilized by FTRT methods to leverage parallel architectures and enhance computation efficiency.

Due to the complexity of modeling large-scale dynamic networks, computational burden reduction is critical to ensure that the overall simulation processing can be achieved within the real-time interval. Evaluating system solutions sequentially or using single-processor computer systems might be insufficient to achieve FTRT timescales. For example, the authors in [22] report that it takes around $60s$ to run a $20s$ simulation for a WECC system composed by a detailed generator and controller model on a computer running PowerWorld with an Intel Core i7, 2.8 GHz CPU, and 8GB of RAM memory. The lack of tools that can solve complex system models motivates the development of FTRT simulation platforms. Following, we discuss prominent examples of FTRT-based simulation platforms used for power system studies.

Accuracy requirements for FTRT simulation consider the deviations between the simulated results, the actual system state, and the dynamic system performance. However, a performance-accuracy trade-off exists, i.e., if detailed system models are utilized, higher accuracy will be obtained but at the expense of longer execution time. However, simplified system models reduce computation time by sacrificing accuracy. Therefore, the FTRT results need to be validated by comparing them with results obtained from other simulation tools evaluating the same mathematical models. For instance, the FTRT simulation results provided using the FTRT GridPACK HPC system could be compared with results reported by PowerWorld (offline) or by an RTS (e.g., Opal-RT). In [22], the generator rotor speed results under fault conditions are similar in both systems. In [2], the authors validate their FTRT results with an offline TS simulation tool TSAT (DSAToolsTM) by comparing them to the oscilloscope waveforms. The FTRT simulation matches the results from TSAT when AC ground faults are performed. The authors in [7] and [5] compare the results of their proposed FTRT model with simulations performed in PSCAD/EMTDC. EMT simulations demonstrate that acceptable accuracy levels can be achieved without sacrificing accuracy.

Furthermore, an error analysis between FTRT results and offline Matlab/Simulink results is performed in [11] using $\epsilon=((R_{FTRT}-R_{\text{Simulink }})/R_{\text{Simulink }})\times 100\%$, where ${R_{FTRT}}$ and $R_{\text{Simulink}}$ refer to the results reported by the FTRT and the Simulink simulations, respectively. The maximum relative errors are $-0.97\%$ over 4 contingency simulation results including three-phase faults, fault mitigation, overloads, and mitigation after overloads. Thus FTRT simulation accuracy is verified under diverse system contingency scenarios.