Learning a Local Trading Strategy: Deep Reinforcement Learning for Grid-scale Renewable Energy Integration

World-wide power sectors have seen ambitious decarbonization targets. Renewable energy sources, such as wind and solar power, offer low cost carbon-free electric power. However, their weather dependence means their output is intermittent – unlike traditional thermal power generation, whose output is controlled by fuel input. Energy storage, such as lithium-ion batteries, can ease the integration of renewables, charging when surplus energy is available and discharging when the renewables output is low [5].

Locational marginal prices (LMPs) describe the value of an additional unit of generation at a given location. This price encapsulates both global and local dynamics; e.g. prices across the network will generally be low when there is a high share of cheap renewable generation, but local congested power lines can cause high prices in small regions of the network. When batteries act to minimize their charging/discharging costs using LMPs, they should act to utilize renewable generation and resolve local constraints. However, without perfect knowledge of future prices and generation, achieving optimal operation is challenging.

Historically, there are two lines of research to solve the energy storage operation problem. The first and more common one is a model-based approach. Here, the description of the cost and transition kernel is known, and a mathematical program is formulated and then solved. For example, a deterministic linear program is formulated to optimize the scheduling of a Lithium-ion battery array, which is connected to a utility electric grid and photovoltaic array, to meet forecast customer load [19]. Similarly, a deterministic mixed-integer linear program (MILP) is used for price arbitraging of a battery in the German intraday markets [17].

It has, however, been acknowledged planning problems must account for uncertainties (e.g., random prices [27, 14, 28] and renewable energy [31, 23, 24]). Naturally, stochastic programming is used. One example is found in [4], where a microgrid involving a lithium-iron-phosphate battery with energy from an unreliable molten carbonate fuel cell and rooftop photovoltaic system, is solved by a two-stage stochastic MILP. Data was generated from historical data over a one-week period, and the goal is to minimize both economic (taxes, electric costs, etc.) and environmental costs. Other methods, such as solving a large-scale linear program [15] and an analytic value iteration method [34] have also been considered. A data-driven distributionally robust optimization framework was applied to the energy storage problem [20] and shown to be competitive with the best risk-neutral methods. We refer to [33] for more stochastic programming methods for energy storage problems and [29, 6] for methods to power systems more generally.

Another line of research to solve energy storage problems is by model-free methods. Here, the assumption for needing to know an explicit formulation of the model is removed, and one can instead work with a black-box simulator of the environment. In particular, we focus on a machine learning approach called deep reinforcement learning (DRL). DRL has recently garnered much attention in the power systems community; see [25] for an extensive review. Several works apply DRL to operating a battery storage for profit/value maximization [30, 31, 3, 9, 8, 11]. For instance, by discretizing the state (i.e., price data from the ISO New England real-time market and battery energy levels) and actions, Q-Learning is used and shown to outperform a greedy approach [30]. State discretization can be avoided by employing deep Q-Network (DQN), which generalizes Q-Learning with neural networks. It achieves higher profits than solving a linear program (LP) when considering battery degradation, suggesting RL-based methods can navigate noisy data more effectively than deterministic methods with forecast data [3]. Yet, the aforementioned DQN uses price forecasts from a hybrid neural network architecture, which takes about three hours to train. A follow-up paper considers the setting of a microgrid consisting of solar photovoltaics (PVs), wind turbines, and energy storage [9]. A DQN variant is shown to outperform the LP approach. A similar study on applying various DRL methods to PV-battery storage systems is evaluated in [11].

In spite of the recent success of DRL for energy arbitrage w.r.t. (with respect to) profit/value maximization, less attention has been paid on which methods best utilize a battery to mitigate the misalignment between variable renewable generation (VRG) and demand. Existing work in this area usually involve modelled methods, such as optimization problems with forecast data [12, 13]. Additionally, the majority of previous DRL work quantifies only a single deterministic period of operation. This does not allow us to quantify the robustness of the DRL approach; unlike formal optimization methods, the training process does not necessarily converge to a single model, and performance can vary drastically based on operating conditions. Furthermore, if a DRL approach were to be widely adopted, it is necessary to consider the aggregate effect of many systems deploying the same strategy.

In this paper, we extend previous work investigating the use of DRL for management of battery energy storage systems via extensive numerical simulation, which is available online¹¹1https://github.com/jucaleb4/battery-trading, in several ways. Firstly, that we consider the “upstream” effect of energy arbitrage as it relates to supply-demand misalignment. Second, that we quantify the robustness of DRL-based control for energy storage systems, analyzing both the sensitivity of the training process and the difference between various locations and seasons. Finally, that we consider the diversity of actions between different systems (with varied LMPs) using the same control policy – diversity between actions across the transmission system is important for stability and avoiding re-bound effects.

We will now formulate the mathematical problem of managing a grid-scale battery energy storage systems (BESS) co-located with photovoltaic (PV) generation.

The relevant parameters of a grid-scale battery are its energy capacity $E_{max}$, charging and discharging efficiency $\eta$, and maximum charging/discharging power $P_{max}$. We consider the energy stored within the battery, $E$, to change via the following dynamics equation:

$\displaystyle E_{t+1}$

$\displaystyle=E_{t}+\Delta_{t}\Big{(}\eta c_{t}-\frac{1}{\eta}d_{t}-s_{t}\Big{)}\,,$

(1)

where $\Delta_{t}$ is the time step size (in hours), $c_{t}$ is the charging rate (MW), and $d_{t}$ is the discharging rate (MW), and $s_{t}$ is the self-discharge rate (MW). However, the charging power $c_{t}$ can come from either the grid or from solar generation. We therefore define terms for the power imported from the grid, $p^{buy}$, and the power discharged to the grid, $p^{sell}$. The power charged to the battery is therefore the net from the grid plus any produced by the PV, which can be written as:

$\displaystyle c_{t}-d_{t}=p^{buy}_{t}+p^{solar}_{t}-p^{sell}_{t}\,,$

(2)

where $p^{solar}_{t}$ is the average power produced from solar (MW) at time-step $t$. Note that we have implicitly assumed that any excess solar must be sold, rather than curtailed. It is not physically possible for a battery to simultaneously charge and discharge. This could be explicitly enforced, e.g., with $c_{t}d_{t}=0$, however this should not be necessary as it would never be optimal for both to be non-zero. To see this consider a RHS of -1MW. The obvious LHS solution would be $d_{t}=1,c_{t}=0$, considering $\Delta_{t}=1$ and $\eta=0.9$ that results in a loss of energy of $1.1$MWh. An alternative solution would be $d_{t}=1.5,c_{t}=0.5$, resulting in an $1.2$MWh energy loss for the same profit. Given an efficiency of $\eta<100$% having one of the terms be zero will always be the most economic solution.

The self-discharge rate of batteries, $s_{t}$, is a function of battery state-of-charge (SoC), with significant self-discharge rates observed when the battery is full, but negligible values at low SoC. For modeling simplicity we assume that the battery discharges at a constant rate above 90% SoC, but that below that we can assume $s_{t}=0$. This could be mathematically constrained using, for example, the big-M formulation.

Finally, we consider our objective term $f$ which is the total profit made by the BESS and PV systems:

$\displaystyle f(p^{buy}_{t},p^{sell}_{t})=\sum_{t}\Delta_{t}\lambda_{t}\Big{(}p^{buy}_{t}-p^{sell}_{t}\Big{)}\,,$

(3)

where $\lambda_{t}$ is the locational marginal price (LMP) at time $t$. The price signal gives the value of an extra unit of generation at that location and time on the network, and hence can be seen as a measure of the balancing services provided to the grid; the higher the sell price, the more useful the battery discharge is for balancing the grid, and therefore the more effectively the solar was integrated.

In this section, we will introduce the model-free reinforcement learning approach to BESS control. Due to the non-linear and uncertain nature of optimal control of a grid-scale BESS, reinforcement learning is an attractive approach to manage BESS in an optimal way. Reinforcement learning (RL) is a model-free, machine-learning approach towards the optimal control of a Markov decision process (MDP). Unlike other machine learning paradigms like supervised and unsupervised learning, the agent in RL learns by receiving observations and rewards obtained by interacting with its environment. By properly choosing a reward signal (e.g., net profit for profit maximization), the agent can learn the optimal strategy – in an economical sense – for battery management.

The grid-scale BESS problem can be formulated as a finite-horizon discrete-time MDP. The time-inhomogeneous MDP consists of a quintuple ($T$, $\mathcal{S}$, $\mathcal{A}$, $r$, $\{\mathcal{P}_{t}\}_{t}$). $T$ is the number of time steps. A state $s_{t}\in\mathcal{S}$ at time $t$ contains both the state-of-charge ($E_{t}$) of the battery as well as exogenous variables derived from real-world datasets, which are the current and historical (up to four hours prior) locational marginal prices $\lambda_{t}$ and energy from solar power $p^{solar}_{t}$ (see Section 4). There are three actions $a_{t}\in\{\text{sell},\text{buy},\text{null}\}$, which respectively correspond to selling power to the grid, buying power from the grid, and null (i.e., neither buy nor sell). Using discrete actions is partially justified by the fact an optimal solution to (4) will charge or discharge at zero or max power. Energy transfers occur in fixed quantities except for battery depletion/full charge. More specifically, the transition kernel $\mathcal{P}_{t}$ at time $t$ defines the dynamics for the non-exogenous state-of-charge $E_{t}$. These dynamics are similar to (1) and standard battery behavior [3, 11], where the next state-of-charge $E_{t+1}$ is defined as

$\displaystyle p(\tilde{E}_{t}+c\Delta_{t}\eta\cdot\mathbf{1}[a_{t}=\text{buy}]-d\Delta_{t}\cdot\mathbf{1}[a_{t}=\text{sell}]).$

The projection $p(\cdot):=\min\{{E}_{max},\max\{0,\cdot\}\}$ ensures the state-of-charge is in the range $[0,{E}_{max}]$ (see Table 1). Self-discharge is highly non-linear but most significant at high state-of-charges. To penalize dormant fully charged batteries, we use the self-discharge rule:

$\displaystyle\tilde{E}_{t}=\begin{cases}\beta\cdot E_{t}&:E_{t}\geq 0.9\cdot{E}_{max}\\ E_{t}&:E_{t}<0.9\cdot{E}_{max}\end{cases}.$

The reward at time $t$ is the net profit from buying/selling power while using solar energy:

$\displaystyle\begin{cases}\big{(}\eta^{-1}[\tilde{E}_{t}-E_{t+1}]+p^{solar}_{t}\big{)}\cdot\lambda_{t}&:a_{t}=\text{buy}\\ \big{(}\eta[\tilde{E}_{t}-E_{t+1}]+p^{solar}_{t}\big{)}\cdot\lambda_{t}&:a_{t}\neq\text{buy}\end{cases}.$

The goal of reinforcement learning is to find a policy $\pi:\mathcal{S}\to\Delta(\mathcal{A})$²²2The probability simplex over a finite set $\mathcal{A}$ is denoted by $\Delta(\mathcal{A})$. that determines (possibly randomly) actions that can maximize the expected cumulative reward over the time horizon $T$. To solve the problem, we introduce the Q-function at time $t$,

$\displaystyle\textstyle Q_{t}^{\pi}(s,a):=\mathbf{E}[\sum_{\tau=t}^{T}r(s_{\tau},a_{\tau})~{}|~{}\begin{subarray}{c}a_{\tau}\sim\pi(s_{\tau})\\ s_{\tau+1}\sim\mathcal{P}(s_{\tau},a_{\tau})\\ s_{t}=s,a_{t}=a\end{subarray}].$

The Q-function quantifies the expected reward when starting at a given state-action pair at time $t$ and following the policy $\pi$ for the remainder of the time horizon. It is well-known the optimal policy $\pi^{*}$ satisfies Bellman’s optimality equation [2, 21], $Q_{t}^{\pi^{*}}(s,a)=r(s,a)+\mathbf{E}_{s^{\prime}\sim\mathcal{P}(s,a)}[\max_{a^{\prime}\in\mathcal{A}}Q_{t+1}^{\pi^{*}}(s^{\prime},a^{\prime})]$, at all time steps $t$ and state-action pairs $(s,a)$.

Q-Learning is a dynamic programming method to solve Bellman’s equation [32, 26]. However, because the state space $\mathcal{S}$ is continuous, the Q-function is high-dimensional and thus cannot be computed exactly. Deep Q-network, or DQN, generalizes Q-Learning to continuous state and discrete action spaces [26, 18]. It approximates the Q-function using a neural network with weights $\theta\in\mathbf{R}^{d}$, written as $Q^{\theta}_{t}$. Then Bellman’s equation is equivalent to solving the non-linear least squares problem³³3State-action indices are omitted to fit the equation inline.:

$\displaystyle\textstyle\underset{\theta}{\min}\sum\limits_{t=1}^{T}\underset{s_{t},a_{t}}{\mathbf{E}}\big{(}Q^{\theta}_{t}-[r+\underset{s^{\prime}\sim\mathcal{P}(s_{t},a_{t})}{\mathbf{E}}[\underset{a^{\prime}\in\mathcal{A}}{\max}~{}Q_{t+1}^{\theta}]\big{)}^{2},$

(5)

where the expectation is taken w.r.t. the trajectory of the state-action pairs when following policy $\pi^{\theta}$, defined as $\pi_{t}^{\theta}(s)={\mathrm{argmax}}_{a\in\mathcal{A}}\{Q_{t}^{\theta}(s,a)\}$. To explore actions, the policy $\pi_{t}^{\theta}$ will occasionally output a random action instead. To solve (5), DQN first replaces the expectation with a Monte-Carlo approximation, which can be done by generating multiple samples $(s,a,r,s^{\prime})$ (i.e., a state, action, reward, and next state) along a trajectory while following policy $\pi^{\theta}_{t}$. Then a gradient is taken w.r.t. the weights $\theta$ (e.g. by autodiff in deep learning libraries) followed by a gradient descent step [26]. While there are many other RL algorithms, we focus on DQN due to its relative simplicity.

In this section, we detail the simulation framework we used for training and testing. Exogenous data within the MDP (i.e., LMPs and solar) is gathered from the California ISO’s Open Access Same-time Information System (OASIS). We obtained real-time (in 15 min increments) LMPs and solar power (in 1 hour increments) across six months in 2023. The MDP (from the previous section) and data are then integrated in Gymnasium, a simulation library for training RL algorithms. Gymnasium provides a black-box model of the environment, outputting a reward and next state when given an action. The RL agent uses the state and reward to determine the next best action (see Fig. 1).

To train the RL agent, we used Stablebaseline3, a popular library of RL algorithms [22]. We used random search to tune the hyperparameters of deep Q-network (DQN). Most default values delivered good performance. Some non-default values we choose were a large exploration fraction (decay) of 0.9685 to reduce exploration and the maximum gradient steps to improve the policy often. The RL agent is trained for 200,000 time steps, taking 574s to run on a Macbook with a 2.3 GHz Dual-Core Intel Core i5 and 8GB of RAM. The rules-based’s genetic algorithm is trained for 64 generations, taking 515s. The training times were manually tuned to provide good solutions.

It is important that the algorithms not be evaluated on the data that is used for training. We therefore split our data chronologically into training and testing, such that the algorithm is trained on the period directly preceding the testing data. Since the genetic and DQN algorithm rely on randomness, we train them over 10 seeds and analyze the average and range of performances. This is important because some seeds may perform well by fluke, but the mean performance is more informative.

Our dataset covers three pnodes, described in Table 2 and profiled in Fig. 2 and 3, over three winter (January to March) and three summer (June to August) months in 2023. These three pnodes cover multiple geographies in California and provide different LMP and solar characteristics. We also provide three PV sizings, resulting in three different solar powers.

Looking closer at Fig. 2 and 3, we see winter months consist of steady, periodic LMPs and varying amounts of solar. In contrast, the summer months exhibit a less periodic LMP and the occasional huge LMP spike, along with consistently high solar power. We also split the LMP and solar into training and testing datasets: the first (approximately) 12 weeks of data are for training and the remaining week is for testing. This mimics the setup where the latest historical data helps train an agent how to manage a battery for the upcoming week.

We compared a reinforcement learning (RL)-based agent to a rules-based benchmark, receding-horizon control (RHC, i.e., advanced optimal control), and a near theoretical optimal solution with perfect price foresight for the operation of a grid-scale battery and PV system. By testing between locations, season, and PV sizings, we identified a variety of LMP behaviors. RHC tends to be sensitive to prices and performs poorly when they cannot be predicted well. A simpler rules-based has less degrees-of-freedom so it will not overfit to noise, but it cannot exploit slight variations in the data. RL is designed to handle noise during the training process, and this leads it to out-perform the benchmarks in most settings, achieving an average of approximately 61% of the approximate optimal profits. Our findings suggest that when future prices cannot be predicted well, RL is an attractive alternative to RHC.

We also demonstrate that the RL method had further benefits over the benchmark: (1) a greater diversity in the operational policies of systems in different areas – demonstrating a local approach and reducing the risk of superimposed similar actions of multiple systems; (2) that the resulting output from the system better aligns with the system-level demand.

Our experience with RL also unveiled some of its limitations. Mainly, it required some effort to tune hyperparameters, since using all the default values did not produce desirable results. We also only implemented deep Q-network, whereas there are many other RL algorithms. Finally, our work made some simplifications to the battery model, such as ignoring battery degradation. Future work can address these shortcomings.