Economic Battery Storage Dispatch
with Deep Reinforcement Learning from Rule-Based Demonstrations

Energy storages play a vital role in the transition towards “green” electricity grids. By smoothening the intermittent power production of renewable energies (RE) and by shaving peaks in electricity demand, storages facilitate the large-scale integration of renewables. Batteries, in particular lithium-ion batteries, have emerged as one of the most promising storage technologies due to significant technological progress and cost reduction [1]. This has led to the planning and realization of utility-scale projects with previously unimaginable capacities. A popular example here is Vistra’s Moss Landing Energy Storage facility in California, United States, consisting of batteries with 400 MW power and 1,600 MWh energy capacity that are currently expanded to 750 MW/ 3,000 MWh [2].

For the proper planning of such facilities, control algorithms are required that can model plant operation in uncertain conditions, for example regarding electricity prices and RE production, and optimize the dispatch of batteries. The objective thereby is usually either minimizing operational cost or maximizing profits and the results can help to dimension the system and to choose appropriate components and locations. In the past few years, reinforcement learning (RL) has gained attention as a tool for economic battery dispatch and research in this field has increased substantially [3].

Especially deep reinforcement learning (DRL) algorithms with actor-critic architectures such as proximal policy optimization (PPO) [4] and deep deterministic policy gradients (DDPG) [5] have thereby performed well. For example, Zhang et al. [6] model a hybrid plant consisting of wind, solar photovoltaic (PV), diesel generators, batteries, and reverse osmosis elements producing power and freshwater. Task of the employed control algorithms is to dispatch the diesel generators and batteries efficiently and minimize system cost. Three popular DRL algorithms, namely double deep Q networks (DDQN), DDPG, and PPO, are compared with the non-RL alternatives stochastic programming (SP) and particle swarm optimization. The temporal resolution of the task is 1h and an episode is 24h long. PPO was found to perform best and to achieve a cost reduction of over 14% compared to SP.

DDPG has been compared to model predictive control (MPC) in the work of da Silva André et al. [7] on battery dispatch for two different electric grid models. The duration between timesteps is 9s with 100 timesteps per episode. Despite not having access to forecasts, contrary to MPC, DDPG achieves similar cost reduction while significantly reducing the computational time. Zha et al. [8] propose a modified actor-critic architecture with a distributional critic network to optimize a grid-connected solar PV, wind, and battery hybrid system. The modifications include grouping four subsequent observations into a state and then using convolutional neural networks for function approximation as well as the pretraining of the critic for 200 episodes. The researchers train the algorithm using an hourly resolution and 720 timesteps (one month) per episode and report an increased performance of this variant over the tested alternatives. The wide-spread applications of reinforcement learning to battery control have been reviewed in detail by Subramanya et al. [3].

The main advantage of popular RL algorithms for battery control is their model-free optimization. Instead of requiring a full mathematical model of the investigated environment, only a sample model is needed [9]. This leads to a data-based approach to optimization that, besides simplifying implementations, allows to capture uncertainty. Modern power systems are complex in nature and subject to different types of uncertainty, for example regarding demands and renewable power production. Model-free RL can cope with this challenge by optimizing complex systems over long periods [10, 11, 12]. On the other hand, RL often struggles to find good policies on tasks with sparse or delayed rewards [13]. As the purpose of batteries is to store energy for later use, delayed rewards are a natural phenomenon. Moreover, agents must learn that investing energy by charging the battery is a requirement for receiving these rewards at later timesteps.

In our experiments, we observe that DRL algorithms struggle to learn useful policies across various algorithms, implementation choices, and environment configurations. Often, training does not exceed the performance of random agents or collapses to a single action, leading to de-facto ignoring the battery. By modeling entire years in 8760 hourly timesteps, our episodes are significantly longer compared to the related work referenced above. This aggravates the dispatch task due to larger state spaces and potentially longer intervals between charging and discharging cycles. However, longer modeling periods capture more uncertainty and lead to more expressive results. In order to obtain good results while keeping the episodes long, we resort to learning from demonstrations, a branch of RL that has previously shown to boost performance during the initial stage of training.

The idea that learning from demonstration, for example from human experts or previous control systems, facilitates learning problems in RL is well-established [14, 15, 16, 17]. One of the most popular applications to DRL has been deep Q-learning from demonstrations (DQfD), an approach that achieved state-of-the-art results and sample efficiencies on many of the tested Atari games when introduced by Hester et al. [15]. The researchers first pretrain the Q-network on demonstrations from a human player using an additional supervised loss term. Then, the agent starts to interact with the environment, storing its experiences in the same replay buffer but never overwriting the demonstrations. Sampling from the replay buffer occurs with prioritized experience replay [18]. The demonstrations are given an extra priority bonus to increase their sampling frequency.

Most approaches assume (near-) optimal demonstrations and force the agent to mimic the behavior of the provided examples. This is achieved either by additional supervised loss-terms as in DQfD, or by modifying the reward function to reward the agent only for behavior resembling the demonstrations. The latter is called imitation learning and has been applied to battery dispatched by Krishnamoorthy et al. [19]. The researchers deploy soft actor-critic (SAC) with imitation learning to control batteries in 34-bus and 123-bus systems over 450 timesteps with 10ms resolution. The demonstrations stem from previously solving the same system using mixed-integer linear programming (MILP) and are assigned a constant reward of +1, while experiences from agent-environment interactions receive a reward of 0. Both types of data are sampled with equal probability during training. This approach, originally proposed by Reddy et al. [16], greatly reduces the computational time compared to MILP, but it can only be applied in settings where perfect demonstrations exist. Gao et al. [17] demonstrate that RL can benefit from imperfect demonstrations as well. The researchers propose normalized actor-critic (NAC), an algorithm that does not conduct supervised pretraining and instead uses an additional normalization term for the gradient of the loss function to reduce the bias from demonstration data. The experiments on driving games show that NAC surpasses the demonstrator and other algorithms, even when fed with imperfect demonstrations.

In this study, we propose an approach to improve economic battery dispatch with RL through imperfect, rule-based demonstrations. We collect demonstrations by controlling the same environment via trivial if-then-else statements based on electricity prices. Our approach uses SAC [20] and only requires minimal changes to the algorithm, namely a second replay buffer for demonstrations and a linearly decaying sampling ratio between the two buffers that favors the demonstrations initially. We show empirically with a case study that despite the simplicity, the approach is robust to the choice of rules and able to outperform both the demonstrator as well as other DRL algorithms and traditional methods. To the best of our knowledge, this is the first application of both imperfect and rule-based demonstration to battery dispatch with RL.

The remainder of the paper is organized as follows. The next section formulates the problem of economic battery dispatch. Section 3 introduces the modified SAC algorithm. Section 4 describes and discusses the conducted experiments before section 5 provides our conclusion.

Figure 1 shows the key components and power flows of the modeled hybrid energy system. We assume that the microgrid is connected to the utility grid and supplies its power to a factory with varying demands. Besides, it consists of wind turbines, solar PV modules, and the battery storage.

The objective is to minimize the total cost of supplying power to the factory, $C_{Total}$:

subject to

In eq. 1, we assume that the operational cost of all components are fixed and thus not relevant for the optimization problem. This leaves the cost of interacting with the utility grid, $C_{G,t}$, as main cost factor. $P_{G,t}$ has a positive sign if power is bought from the grid and a negative sign if it is sold to the grid. Surplus electricity is sold at wholesale price, $c_{w,t}$, while purchasing deficit power adds fixed auxiliary cost, $c_{a}$, for example for transmission charges (see eq. 2).

Constraint 3 ensures that the power demand is met at all times. $P_{B,t}$ is negative if the battery is charged, and positive if it is discharged. Constraint 4 describes the behavior of the battery’s state-of-charge (SOC), with $\eta<1$ during charging ($P_{B,t}<0$) and $\eta>1$ during discharging. Constraint 5 restricts the permitted range of SOCs to prevent damage to the battery. Constraint 6 limits $P_{B}$ to the maximum charging and discharging rates. We further add constraint 7 to limit the battery charging to the available RE power.

To apply reinforcement learning to the battery dispatch problem, we resort to the common Markov Decision Process (MDP) framework [21]. The MDP is defined as the tuple $(\mathscr{S},\mathscr{A},\mathscr{P},\mathscr{R},\gamma)$, consisting of the state space $\mathscr{S}$, the action space $\mathscr{A}$, the transition function $\mathscr{P}$, the reward function $\mathscr{R}$, and the discount factor $\gamma$. At each timestep $t$, the agent finds itself in a state $s_{t}\in\mathscr{S}$ and chooses an action $a_{t}\in\mathscr{A}(s_{t})$. The agent then receives a reward $R_{t+1}$ for the state-action pair $\mathscr{R}:\mathscr{S}\times\mathscr{A}\rightarrow\mathbb{R}$ and the environment transitions to the next state $s_{t+1}$ as by the transition function $\mathscr{P}:\mathscr{S}\times\mathscr{A}\times\mathscr{S}\rightarrow[0,1]$. The goal is to maximize the discounted sequence of rewards observed after timestep $t$, also called the expected return:

with $\gamma\in[0,1]$ [21]. For this work, we define state space, action space, and reward function as follows:

State space: A state $s_{t}\in\mathscr{S}$ is defined as $s_{t}=(c_{w,t},P_{RE,t},P_{D,t},SOC_{t},sin(h),cos(h),\mathbbm{1}(workday))$ and conveys the current timestep’s information regarding wholesale price, renewable energy production, power demand, and battery SOC. In addition, $sin(h),cos(h),\mathbbm{1}(workday)$ are time-related features to help the agent recognize patterns, for example in the factory’s power demand. $sin(h)$ and $cos(h)$ are a cyclical encoding for the hour of the day. $\mathbbm{1}(workday)$ is a binary feature that returns one if the current day is a workday for the factory and 0 otherwise.

Action space: The action space is continuous and one-dimensional with $a_{t}=P_{B,t}$ and $a_{t}\in[P^{min}_{B},P^{max}_{B}]$. We prevent the agent from violating the constraints 5 - 7 by applying the following security layer to obtain a corrected action, $a_{c,t}$ before executing the action in the environment:

This serves as an action mask correcting invalid actions to the nearest possible valid action.

Reward function: The immediate reward, $R_{t}$, consists of two terms:

where $C_{G,t}$ is the cost of grid interaction as introduced by eq. 2 and $\omega\mathbbm{1}(a_{c,t},a_{t})$ is a penalty term with weight $\omega$ that is added only if $a_{c,t}\neq a_{t}$, i.e. when action correction is required. Both terms receive a negative sign to turn the cost minimization task into a reward maximization task. The penalty term is only used inside the control algorithm. The accumulated rewards reported in section IV are computed as $R=-\sum_{t=0}^{T}C_{G,t}$.

We apply SAC to this RL framework. The algorithm has been introduced by Haarnoja et al. [22] and improved by the same authors in [20]. We implement the improved version which includes entropy regularization by learning the temperature parameter. Proposed as off-policy algorithm for continuous action spaces, SAC is a natural choice for the battery dispatch environment and our objective to utilize demonstration data. Algorithm 1 shows our SAC variant in pseudocode, with focus on the modification we have made to gather demonstration data and incorporate it into the learning process. These modifications are:

• •

  Defining a deterministic, rule-based policy, $\pi_{r}$, that controls the environment through an if-then-else statement. For example, the rewards of the environment introduced in section II are greatly dependent on the current wholesale price, $c_{w,t}$. A straightforward rule could thus be:

  $$\pi_{r}\coloneqq a(s_{t})=\begin{cases}P^{max}_{B}&,c_{w,t}>\overline{c_{w}}\\ P^{min}_{B}&,c_{w,t}\leq\overline{c_{w}}\end{cases}\hskip 5.0pt\forall t\in T$$

  (11)

  which discharges the battery at $P^{max}_{B}$ whenever the current wholesale price $c_{w,t}$ is greater than the mean wholesale price $\overline{c_{w}}$, and charges at $P^{min}_{B}$ otherwise. Designing such rules requires some domain knowledge and applying them to control the environment produces imperfect demonstration data. However, as we will show in the conducted case study, SAC benefits even from highly suboptimal rules and reliably learns policies that surpass them.

• •

  Initializing a second replay buffer, $\mathcal{D}_{demo}$, and populating it with the transitions obtained from executing $\pi_{r}$ in the environment. In our experiments with episodes of 8760 timesteps, we conduct a single episode to collect demo data (see lines 2-6).

• •

  Composing a joint batch from demonstrations and experiences by sampling from both replay buffers during training (see lines 14-18). The agent interacts with the environment using its own policy and stores the observed experiences in the replay buffer $\mathcal{D}_{exp}$, just as in the original SAC algorithm. To update network weights and temperature, a batch of size $\mathcal{B}$ is set together by uniformly sampling from both replay buffers according to ratio $\rho$ (see line 9). $\rho$ is designed to initially favor demonstration data and then shift towards the agent’s own experiences as training progresses. We use a linearly decaying $\rho$ and compute its value as $\rho=\frac{\mathcal{E}-e}{\mathcal{E}}$ using the current episode, $e$, and the total number of episodes, $\mathcal{E}$.

The above modifications are easy to implement and insignificant regarding the added computational time. For the remainder of this paper, we will refer to algorithm 1 as SAC from Demonstrations (SACfD).

We propose an approach to improve battery dispatch with RL through self-generated, imperfect demonstrations that are generated before training via simple if-then-else statements. This makes the approach applicable to tasks where no expert demonstrations are available. The results with SAC on the case study show that sample efficiency and accumulated rewards dramatically improve. Despite the simplicity of the proposed method, the use of demonstrations is the key factor to obtain well-performing and stable policies. The results further indicate that our method is robust to different rules, as long as the demonstration data can give the agent a nudge into the right direction during initial training.

A key challenge in the field of battery dispatch is the wide range of applications and the lack of a suitable benchmarking environment [3]. Further research is required to assess the applicability of the approach to both other RL algorithms and other battery dispatch experiments. A natural extension of the approach would be experiments with prioritized instead of uniform data sampling, as well as alternatives to price-based rules for the generation of demonstration data.

This work is supported by the McGill Engineering Doctoral Award (MEDA) and MITACS grant number IT13369. The authors would like to thank Siemens Energy in Montreal for the support.