Stability Constrained Reinforcement Learning
for Decentralized Real-Time Voltage Control

We consider the branch flow model [1] in a radial distribution network. Consider a distribution grid $\mathcal{G}=(\mathcal{N}_{0},\mathcal{E})$, consisting of a set of $\mathcal{N}_{0}=\{0,1,\ldots,n\}$ nodes and an edge set $\mathcal{E}$. In the graph, node $0$ is known as the substation, and all the other nodes are buses that correspond to residential areas. We also use $\mathcal{N}=\mathcal{N}_{0}/\{0\}$ to denote the set of nodes excluding the substation node. Each node $i\in\mathcal{N}$ is associated with an active power injection $p_{i}$ and a reactive power injection $q_{i}$. Let $V_{i}$ be the complex voltage and $v_{i}=|V_{i}|^{2}$ is the squared voltage magnitude. We use notation $\mathbf{p},\mathbf{q}$ and $\mathbf{v}$ to denote the $p_{i},q_{i},v_{i}$ stacked into a vector. $\mathbf{p},\mathbf{q}$ and $\mathbf{v}$ satisfy the following equations, $\forall j\in\mathcal{N},i=\textrm{parent}(j)$,

where $l_{ij}=\frac{P_{ij}^{2}+Q_{ij}^{2}}{v_{i}}$ is the squared current, $P_{ij}$ and $Q_{ij}$ represent the active power and reactive power flow on line $(i,j)$, and $r_{ij}$ and $x_{ij}$ are the line resistance and reactance. Equation (1a) and (1b) represent the real and reactive power conservation at node $j$, and (1c) represents the voltage drop from node $i$ to node $j$.

Following [36], if the higher order power loss term can be ignored by setting $l_{ij}=0$, we obtain the following linear approximation model,

We can rearrange the above equations into the vector form,

where matrix $R={[R_{ij}]}_{n\times n},X={[X_{ij}]}_{n\times n}$ are given as follows, $R_{ij}:=2\sum_{(h,k)\in\mathcal{P}_{i}\cap\mathcal{P}_{j}}r_{hk},X_{ij}:=2\sum_{(h,k)\in\mathcal{P}_{i}\cap\mathcal{P}_{j}}x_{hk}$ where $\mathcal{P}_{i}\subset E$ is the set of lines on the unique path from bus $0$ to bus $i$. Here we follow [17] to separate the voltage magnitude $\mathbf{v}$ into two parts: the controllable part $X\mathbf{q}$ that can be adjusted via adjusting reactive power injection $\mathbf{q}$ through the inverter-based control devices, and the non-controllable part $\mathbf{v}^{env}=R\mathbf{p}+v_{0}\mathbf{1}$ that is decided by the load and PV active power $\mathbf{p}$. Matrix $X$ and $R$ satisfy the following property, which is crucial for the stable control design.

We now introduce an abridged version of the branch flow model in three-phase distribution systems. For simplicity, it is first assumed that all buses are served by all three phases, so we can use 3-dimensional vectors to represent system variables. With slight abuse of notation, $\mathbf{P_{ij}}$ is a 3-dimensional vector such that $\mathbf{P_{ij}}=[P_{ij}^{a},P_{ij}^{b},P_{ij}^{c}]^{\top}$. $\mathbf{S_{ij}}$, $\mathbf{Q_{ij}}$ are defined in the same way. The vectors of power injections and complex voltages are denoted by $\mathbf{s_{i}}$ and $\mathbf{v_{i}}$, respectively. $\mathbf{Z_{ij}}\in\mathbb{S}^{3}$ is the phase impedance matrix for line $(i,j)$, where $\mathbf{Z_{ij}}=\mathbf{R_{ij}}+j\mathbf{X_{ij}}$.

We further assume that the phase voltages of arbitrary bus $i$ are approximately balanced with absolute value $\hat{v}_{i}$, then $\mathbf{v_{i}}$ can be estimated by $\hat{v}_{i}\mathbf{\alpha}$, where $\mathbf{\alpha}=[1\quad a\quad a^{2}]$, $a=e^{-j{2\pi}/{3}}$. Define $\mathbf{\hat{Z}_{ij}}=diag(\mathbf{\alpha}^{H})\mathbf{Z_{ij}}diag(\mathbf{\alpha})$, following (2), the linear approximate three-phase model is,

Notice that the vector variables can be arranged either by bus or by phase. For example, the voltage magnitude can be rearranged by phase as $\mathbf{\check{v}}=[\mathbf{\check{v}_{a}},\mathbf{\check{v}_{b}},\mathbf{\check{v}_{c}}]$, where $\mathbf{\check{v}_{a}}=[v_{1}^{a},v_{2}^{a},...,v_{n}^{a}]$, $\mathbf{\check{v}_{b}}$ and $\mathbf{\check{v}_{c}}$ share the similar definition. Recall that $\mathbf{v}=[\mathbf{v_{i}},i\in\mathcal{N}]$, which is ordered by bus. With a permutation matrix $T_{v}$, the transformation between two formats can be represented by $\mathbf{v}=T_{v}\mathbf{\check{v}}$. The three-phase branch flow model can then be arranged to a compact form, the same as the single-phase model,

For a detailed mathematical derivation, please refer to [37]. Notice that the single-phase and three-phase system dynamics share the same linear approximation model (3), $\mathbf{v}=X\mathbf{q}+\mathbf{v}^{env}$, allowing us to derive the Lyapunov equation and stability conditions based on the same analytical model.

It is denoted in [37], due to the structure of distribution lines, the diagonal dominance conditions in Assumption 1 are generally satisfied for multi-phase grids. According to Corollary 1 in [37], if Assumption 1 holds, $X$ is positive definite for three-phase distribution system.

Voltage stability is defined as the ability of the system voltage trajectory to return to an acceptable range after arbitrary disturbance. See Definition 1 below.

With high penetration of DERs, rapid changes of load and renewable generation often happen in a fast time scale, thus it is important to ensure the designed controller meets the stability condition. With the requirement for voltage stability, the optimal voltage control problem can be formulated as,

The goal of the voltage control problem is to reduce the total cost (7a) for time steps $t$ from 0 to $T$, which consists of two parts: the cost of voltage deviation and the cost of control actions. One can choose different cost functions (e.g., one-norm, two-norm, or infinity-norm), depending on the system performance metrics and control devices. Our stability-constrained RL framework can accommodate different types of cost functions mentioned above. In particular, in our experiment, we use $c_{i}({v}_{i}(t),u_{i}(t))=\eta_{1}\|\max(v_{i}(t)-\bar{v}_{i},0)+\min(v_{i}(t)-\underline{v}_{i},0)\|_{2}^{2}+\eta_{2}\|u_{i}(t)\|_{1}$. Here $\eta_{1},\eta_{2}$ are coefficients that balance the cost of action with respect to the voltage deviation. Voltage dynamics are represented by equations (7b)-(7c). (7d) specifies the decentralized policy structure $u_{i}(t)=-g_{\theta_{i}}(v_{i}(t))$ only depends on local voltage measurement $v_{i}(t)$. Here $\theta_{i}$ is the policy parameter for the local policy at node $i$, and $\theta=(\theta_{i})_{i\in\mathcal{N}}$ is the collection of the local policy parameters.

Transient cost vs. stationary cost. Our problem formulation in (7) is different from some of those in the literature, e.g., [38, 5, 39, 6], in the sense that the existing works typically consider the cost in steady-state, meaning the cost is evaluated at the fixed point or stationary point of the system. In contrast, our work considers the transient cost after a voltage disturbance, which is also an important metric for the performance of voltage control. An important future direction is to unify these two perspectives and design policies that can optimize both the transient and stationary costs.

In order to solve the optimal voltage control problem in (7), one needs the exact system dynamics, i.e., $X$. However, for distribution systems, the exact network parameters are often unknown or hard to estimate in real systems [7]. RL provides a powerful paradigm for solving (7), by training a policy that maps the state to action via interacting with the environment, so as to minimize the loss function defined as (7a). There are many RL algorithms to solve the policy minimization problem (7), and in this paper, we focus on the class of RL algorithms called policy optimization. We define the state space of each local controller as the nodal voltage deviation, represented by $v_{i}\in\mathbb{R}$ (single-phase) or $v_{i}\in\mathbb{R}^{3}$ (three-phase). The action space is defined as the range of potential reactive power changes, represented by $u_{i}\in\mathbb{R}$ (single-phase) or $u_{i}\in\mathbb{R}^{3}$ (three-phase).

Generally speaking, we parameterize each of the controllers, i.e., $u_{i}(t)=-g_{\theta_{i}}(v_{i}(t))$ as a neural network with weights $\theta_{i}$. The procedure is to run gradient methods on the policy parameter $\theta_{i}$ with learning rate $\alpha_{i}$, $\theta_{i}\leftarrow\theta_{i}-\alpha_{i}\nabla J(\theta_{i}).$ As we are dealing with deterministic policies and continuous state space, one of the most popular choices is the Deep Deterministic Policy Gradient (DDPG) [12], where the policy gradient $\nabla J(\theta_{i})$ is approximated by

where $g_{\theta_{i}}(v_{i})$ is the actor network, and $\{v_{i}[j],u_{i}[j]\}_{j\in B}$ are a batch of samples with batch size $|B|=N$ sampled from the replay buffer which stores historical state-action transition tuples of bus $i$. Here $\hat{Q}_{\phi_{i}}(v_{i},u_{i})$ is the value network (a.k.a critic network) that can be learned via temporal difference learning,

where $v_{i}^{\prime}$ is system voltage after taking action $u_{i}$ and realization of $v^{env}_{i}$. For more details of DDPG, readers may refer to [12].

In standard DDPG, stability is not an explicit requirement, it plays the role of implicit regularization since instability leads to high costs. However, the lack of an explicit stability requirement can lead to several issues. During the training phase, the policy may become unstable, causing the training process to terminate. Even after a policy is trained, there is no formal guarantee that the closed loop system is stable, which hinders the learned policy’s deployment in real-world power systems where there is a very strong emphasis on stability. Next, we will introduce our framework that guarantees stability in policy learning.

In order to explicitly constrain stability for RL, we constrain the search space of policy in a subset of stabilizing controllers from Lyapunov stability theory. In particular, we use a generalization of Lyapunov’s direct method, known as LaSalle’s Invariance theorem for deriving the stability condition.

The key to ensure stability is to find a controller $\mathbf{u}=-g_{\theta}(\mathbf{v})$ and a Lyapunov function $V$, such that the stability conditions in Proposition 2 can be satisfied. For the voltage control problem defined by (7), $\mathbf{v}(t+1)=\mathbf{v}(t)+I_{\Delta T}X\mathbf{u}(t)$, where $I_{\Delta T}$ is a diagonal matrix with diagonal entries equal to $\Delta T$. Since the control input $\mathbf{u}=-g_{\theta}(\mathbf{v})$ depends on state $\mathbf{v}$, the closed-loop system dynamics can be written as $\mathbf{v}(t+1)=\mathbf{v}(t)-I_{\Delta T}Xg_{\theta}(\mathbf{v}):=f_{u}(\mathbf{v}(t))$. We consider the following Lyapunov function,

where X is the network reactance matrix defined in (3), that is a positive definite matrix for both single-phase and three-phase distribution grids. $V$ is positive definite and is radially unbounded if $\|g_{\theta}(\mathbf{v})\|\rightarrow\infty$ as $\|\mathbf{v}\|\rightarrow\infty$. From LaSalle’s theorem in Proposition 2, if $V(f_{u}(\mathbf{v}))-V(\mathbf{v})\leq 0$ and $V(f_{u}(\mathbf{v}))-V(\mathbf{v})=0$ only when $\mathbf{v}\in S_{v}$, where $S_{v}$ is the voltage safety set defined as $S_{v}=\{\mathbf{v}\in\mathbb{R}^{n}:\underline{v}_{i}\leq v_{i}\leq\bar{v}_{i}\}$. Then we have for every initial voltage profile $\mathbf{v}(0)\in\mathbb{R}^{n}$, $\mathbf{v}(t)$ converges to the largest invariant set in $S_{v}$. Furthermore, suppose for all $i$, the control action satisfies $u_{i}=0$ for $v_{i}\in[\underline{v}_{i},\bar{v}_{i}]$. Then $S_{v}$ itself is an invariant set.

The key question now reduces to how can we design the controller $\mathbf{u}=-g_{\theta}(\mathbf{v})$ such that the closed-loop system satisfying these two properties:

1. 1.

   $V(f_{u}(\mathbf{v}))-V(\mathbf{v})<0$, $\forall v\notin S_{v}$

2. 2.

   $V(f_{u}(\mathbf{v}))-V(\mathbf{v})=0$ for $\mathbf{v}\in S_{v}$

Theorem 1 presents a sufficient structural condition for the above properties to hold, thus guaranteeing voltage stability.

Equation (11) shows that when the sampling time $\Delta T\to 0$, the stability condition reduces to the continuous time stability condition $\frac{\partial\mathbf{u}}{\partial\mathbf{v}}\prec 0$ as first shown in [8]. As the length of the sampling time increases, $\frac{\partial\mathbf{u}}{\partial\mathbf{v}}$ also needs to be lower bounded by $-\frac{2}{\Delta T}X^{-1}$ and upper bounded by $0$. As the typical sampling frequency of real-world inverters is in kHz scale[41], the left-hand side of (11) is naturally satisfied in most cases. Therefore, we focus on the right-hand side condition, $\frac{\partial\mathbf{u}}{\partial\mathbf{v}}\prec 0$. Because of the decentralized characteristic, $\frac{\partial\mathbf{u}}{\partial\mathbf{v}}$ is a diagonal matrix.

Thus, if each $g_{\theta_{i}}$ is strictly monotonically increasing, i.e., $\frac{\partial g_{\theta_{i}}(v_{i})}{\partial v_{i}}>0,\forall i$, the voltage stability condition $\frac{\partial\mathbf{u}}{\partial\mathbf{v}}\prec 0$ will be met. We note that a similar stability condition for the discrete-time voltage dynamics has been shown in [42], while our condition ensures globally asymptotic stability rather than the local stability guarantee in [42].

Combining the structural constraints for stabilizing controllers in Theorem 1 and the DDPG algorithm for solving voltage control in Section III-B, we now present the design of Stable-DDPG algorithm. The proposed stability-constrained policy learning algorithm is summarized in Algorithm 1.

As we notice in Algorithm 1, the general algorithm flow of Stable-DDPG is the same as DDPG, and the only difference is in the policy network parameterization. Since Theorem 1 restricts the class of stabilizing decentralized controllers to be strictly monotonically decreasing. Thus, we need to incorporate this structural condition into the policy design. Essentially, any monotone functions can be used for parameterizing the policy function, e.g., a linear policy $u_{i}=-k_{i}v_{i},\forall v_{i}<\underline{v}_{i}$ or $v_{i}>\overline{v}_{i}$ with $k_{i}$ is positive; and $u_{i}=0$ for $\underline{v}_{i}\leq v_{i}\leq\overline{v}_{i}$. To leverage the superior expressiveness of neural networks, we represent $u_{i}=-g_{\theta_{i}}(v_{i})$ as a monotone neural network. There are several existing designs for the monotone neural networks in literature, e.g. [43, 44, 31]. In this paper, we follow the monotonic neural network design in [31, Lemma 3], which guarantees universal approximation of all single-input-single-output monotonic increasing functions [31, Theorem 2]. This design uses a single hidden layer neural network with $d$ hidden units and ReLU activation, which is defined below.

For single-phase test feeders, we use the IEEE 13-bus feeder and IEEE 123-bus feeder as the test cases, which are modified from three-phase models in [46]. For three-phase test feeders, we test on the unbalanced IEEE 13-bus feeder and IEEE 123-bus feeder [46]. Simulations for single-phase systems are implemented with pandapower [47], and the three-phase system is obtained by OpenDSS. We simulate different voltage disturbance scenarios: 1) High voltages: the PV generators are generating a large amount of power, this corresponds to the daytime scenario in California where there is abundant sunshine that can result in high voltage issues. 2) Low voltages: the system is serving heavy loads without PV generation. It corresponds to late afternoon or night when there is low/no solar generation but still a significant load. For each scenario, we randomly vary the active power injections thus obtaining different degrees of voltage violations, i.e., $5\%$ to $15\%$ of the nominal value. We set $\Delta T=1s$ for all numerical experiments and verify the stability condition for all the trained policies. All experiments are conducted with an AMD 5800X CPU and an Nvidia 1080Ti GPU.

Baselines: We test the proposed stable-DDPG approach (Algorithm 1), against the following baseline algorithms. Details about the algorithm implementations are provided in Appendix B.

We now evaluate Stable-DDPG in three-phase systems. All simulations are built with the OpenDSS public models [48].

The above results also reveal an important trade-off between stability and the expressiveness of neural networks. DDPG algorithm with standard neural network policy obtains the best transient performance if it was able to stabilize the system (see the performance of DDPG*). However, without a stability guarantee, the DDPG controller can lead to unstable working conditions, thus incurring overall high costs compared to both optimized linear policy and Stable-DDPG policy. With the monotone policy network, Stable-DDPG maintains the voltage magnitude in all test scenarios at the cost of a less flexible neural network parameterization. The linear policy can be regarded as an extreme example of a restricted neural net with only one learnable parameter, its slope, and thus might get sub-optimal performance compared to the monotone neural network with more learnable parameters.