Demand Responsive Dynamic Pricing Framework for Prosumer Dominated Microgrids using Multiagent Reinforcement Learning

Demand-side management, also known as Demand Response (DR), is one of the most widely studied topics in the context of the smart grid [1]. In order to flatten the demand curve, conventional time-based DR methods such as the Real-Time Pricing (RTP) approach [2] rely on dynamically changing the electricity price to motivate the customers to alter their energy use profile [3]. This potentially improves the grid stability and reliability by shifting the peak demand and decreasing the need for peaking power plants while offering reduced energy bills to residential customers [4, 5].

Nevertheless, the conventional time-based DR methods typically assume that the pricing policies are deterministic and decided ahead of time [6, 7, 8], or consider that pricing policies follow a random process with known properties [9]. Thus, conventional DR methods are either not able to offer convergence to the optimal solution in presence of uncertainties in the environment or the mathematical formulations become cumbersome to model uncertainties [10], which makes them unsuitable for real-world implementations.

Moreover, the conventional time-based DR approaches almost exclusively focus on altering the preferred electricity consumption pattern of the customers, e.g., by changing the temperature set point on air conditioning systems or delaying the use of major appliances, in order to shift their load to off-peak periods. These approaches often lead to customer dissatisfaction (disutility), and, as a consequence, have not been widely adopted. Psychologically, the consumers value their comfort much higher than economic savings provided by traditionally proposed DR approaches [11].

Integration of energy storage into residential photovoltaic (PV) systems, EV and PV forecasting should provide energy consumers & producers, commonly known as prosumers, with more flexibility to participate in DR programs while minimizing their disutility[12, 13, 14]. In other words, prosumers should be able to shape their demand profile in real-time regardless of their consumption profile [15]. As a result, prosumers can potentially receive greater economic benefits by selling their excess energy to the grid at higher prices, while aiding grid support services [16]. Achieving the aforementioned benefits demands for a novel DR approach on the prosumers’ side. This DR approach considers various factors such as state-of-charge (SOC) of the storage device, household consumption profile, and real-time electricity price and PV generation levels. Furthermore, taking advantage of the flexibility provided by prosumers requires a modern grid management strategy which can treat the storage capacity of households as a grid asset that can be dispatched by properly incentivizing the households for DR participation, and leverage this asset for grid cost and performance optimization.

This paper proposes a multiagent deep Reinforcement Learning (RL) framework for implementing a new RTP-based DR technique in a prosumer microgrid that provides both, prosumers and the grid operator, with the aforementioned flexibility and greater economic benefits. The main contributions of the proposed framework are summarized as follows: a) Real-time learning: Grid and prosumer agents learn the optimal price and DR participation policy by interacting with the environment in real-time, rather than using a complex grid dynamic model for optimization. Therefore, the proposed method is applicable to the high dimensional and non-stationary environment of the grid with much less computational burden than traditional DR methods, allowing for real-world implementations; b) Altering the households’ grid injection patterns: The goal of the proposed prosumer-side DR algorithm is not to alter the consumption pattern of households which can typically lead to customer dissatisfaction. Rather, the prosumer-side DR algorithm provides cost savings by altering the households’ grid injection pattern, using the flexibility provided by the energy storage and PV generation; and c) Balanced market: The proposed framework makes better use of prosumers’ energy storage capacity, allowing for significant prosumer electricity bill reduction, as well as considerable grid economic benefit improvement, with a reasonably sized battery pack. Our results also show that after a certain threshold value, a larger battery size on the prosumer side does not necessarily yield a much higher profit. This facilitates a balanced market setup where trying to abruptly and unilaterally manipulate the pricing scheme is not in anyone’s financial interest.

Related Work: In recent years there has been a growing interest in application of RL to the problem of dynamic pricing and DR. A comprehensive survey of published works in this area is provided in [17]. Among the surveyed publications, the works in [18] and [19] are closely aligned with our work. The authors in [18] propose a RL-based dynamic pricing and energy consumption scheduling framework that can work without priori information and leads to reduced system costs. The proposed framework targets regular customers without grid injection capability and assumes that customer behavior is myopic and deterministic, i.e. each costumer is trying to minimize its cost in every single time slot. In contrast, our work takes advantage of the PV generation and storage capacity of prosumers, leading to greater flexibility for DR participation, and enables the prosumers to make decisions that lead to long-term optimization of their accumulative economic benefit, rather than minimizing their instantaneous cost. On the other hand, the authors in [19] present a RL-based dynamic pricing algorithm which can promote service provider profitability and reduce energy costs for the customers. However, similar to [18], this work only deals with regular electricity consumers, rather than prosumers with generation capability. A multi-Agent using deep reinforcement learning for a distributed energy resources in Smart Grids also provided in [20] which is highly related to our work.

The reminder of this paper is structured as follows: Section II details the proposed RL based DR approach. Section III implements the proposed method for a small-scale microgrid as a case study and provides simulation results to verify efficacy of the proposed method. Finally, our concluding thoughts are presented in Section IV.

Fig. 1 illustrates the envisioned microgrid system with conventional generation facilites, prosumers, and consumers. Prosumers are entities that can produce energy locally from renewable resources, store the excess energy in their battery, sell energy into the grid, or buy electricity from the grid for local consumption. The prosumers can make a profit by selling electricity to the grid at a dynamic $/kWh price of ($\delta^{b}$), referred to as buy price hereinafter. On the other hand, the prosumers incur a cost when buying electricity from the grid at a $/kWh price referred to as sell price ($\delta^{s}$), hereinafter. The goal of prosumer agents is to maximize its long term profit by determining an optimal charge/discharge policy for their energy storage devices. Similarly, the grid can buy energy from prosumers at a price of ($\delta^{b}$) and incur a cost, or sell electricity to prosumers at a price of ($\delta^{s}$) to make a profit. The goal of the grid agent is defined as maximizing the long term profit of the grid by determining an optimal buy price ($\delta^{b}$) policy. Therefore, we define the following optimization problem for grid agents,

where $\text{M}(T)$ is the grid profit defined as follows:

In addition, we have:

In which $V({P^{dm}}(T))$ represents the accumulative grid revenue by selling electricity to the households in time horizon $T$, ${{C_{j}}({P_{j}^{inj}(T)})}$ represents the accumulative grid cost of buying excess energy from $j^{th}$ prosumer in time horizon $T$, ${{F_{i}}({P_{i}^{G}(T)})}$ represents the accumulative cost of buying electricity form $i^{th}$ generation facility time horizon $T$, $N_{c}$ , $N_{g}$ and $N_{p}$ are the number of consumers, generation facilities and prosumers respectively. The first equality constraint in (1) represents the grid’s power balance requirement which needs to be maintained at all times, $P^{dm}$ is the total demand of the network, ${P_{q}^{d}\left(t\right)}$ is the demand of household $q$ , and $P_{j}^{inj}$ is the power injection into the grid by $j^{th}$ prosumer.

Similarly we define the following optimization problem for $j^{th}$ prosumer,

where ${{\rm U}_{j}}(T)=V_{j}^{p}\left({P_{j}^{inj}\left(T\right)}\right)-C_{j}^{p}\left({P_{j}^{inj}\left(T\right)}\right)$ is the $j^{th}$ prosumer accumulative profit in time horizon $T$, $V_{j}^{p}(P_{j}^{inj}(T))$ represents the $j^{th}$ prosumer accumulative revenue for selling excess electricity to the grid in time horizon $T$ calculated using the same equation as (3) considering $V_{j}^{p}(P_{j}^{inj}(T))=C_{j}(P_{j}^{inj}(T))$, and $C_{j}^{p}(P_{j}^{inj}(T))$ represents the $j^{th}$ prosumer accumulative cost of buying electricity from the grid in time horizon $T$ calculated as,

$P_{j}^{pv}(t)$ is the PV generation with the peak generation of $P_{j}^{pv,\max}$ for $j^{th}$ household, $P_{j}^{c}(t)$ is consumption for $j^{th}$ household, $P_{j}^{inj,\max}$ represents the maximum allowable power injection for $j^{th}$ household, $P_{j}^{batt}(t)$ represents the energy storage charge/discharge power with maximum allowable charge/discharge power of $P_{j}^{batt,\max}$ for $j^{th}$ household, $So{C_{j}}\left(t\right)$ is the state of charge for $j^{th}$ prosumer where $SoC_{j}^{\min}$ and $SoC_{j}^{\max}$ are minimum and maximum allowable state of charge. The state of charge is calculated as,

where $So{C_{j}}\left(0\right)$ is the initial state of charge and ${\eta_{j}}$ is the energy storage capacity of $j^{th}$ prosumer.

Each agent uses RL for solving the defined optimization problems in real-time. The agents interact with the environment in order to maximize a specified reward. The agents receive positive rewards for taking desirable actions and negative rewards for taking undesirable actions. This learning process is modeled as a Markov Decision Process (MDP) for a Multi-agent Reinforcement Learning (MARL) environment defined by a tuple of $\{{\rm{N,\ S,\ \{}}{{\rm{A}}_{\rm{i}}}{\rm{\},\ P,\ \{}}{{\rm{R}}_{\rm{i}}}{\rm{\},\ \gamma}}\}$ where $\rm{N}$ is the set of agents, $\rm{S}$ is the set of states of the environment, ${\rm{A}}_{\rm{i}}$ is the set of actions for $i^{th}$ agent, P is the set of transition function, ${\rm{R}}_{\rm{i}}$ is the immediate reward function set for $i^{th}$ agent, and ${\rm{\gamma}}\in\left[{0,1}\right]$ is the discount factor. The agent-environment interaction for an MDP is shown in Fig 2. At each discretized time index $k$, the agent $\rm{N_{i}}$ selects an action ${a_{i,k}}\in{{\rm{A}}_{\rm{i}}}$ based on observing the current state of the environment denoted by $s_{k}$. Subsequently, the agent receives a numerical feedback signal known as reward, ${r_{i,k}}\in{{\rm{R}}_{\rm{i}}}$, and transitions to a new state $s_{k+1}$. In the proposed framework, each agent takes actions only based on its own local. In other word, each agent can only observe partial features of the entire environment. Thus, the grid agent can observe the following environment states,

where ${{\mathbf{F}}_{k}}={\left[{\begin{array}[]{*{20}{c}}{{F_{1,k}}}&{...}&{{F_{i,k}}}\end{array}}\right]^{T}}$ and ${{\mathbf{\Omega}}_{k}}={\left[{\begin{array}[]{*{20}{c}}{{C_{1,k}}}&{...}&{{C_{j,k}}}\end{array}}\right]^{T}}$ are the vectors of the grid cost for buying electricity from the generation facilities and prosumers at time slot k respectively, ${{P}}_{1,k}^{dm}$ represents the total grid demand at time slot k.

Similarly, each prosumer agent can observe the following environment states,

where ${SoC_{n,k}}$ is the state of charge of energy storage device of $n^{th}$ prosumer at time slot k, $P_{n,k}^{pv}$ and $P_{n,k}^{c}$ are the PV generation and local consumption of $n^{th}$ prosumer at time slot k, $\delta_{k}^{b}$ is the buy price at time slot k.

In this work, the grid agent controls the buy price to incentivize the DR participation of the prosumers and maximize its own reward. Therefore, the buy price is the action of the grid agent denoted by ${a_{1,k}}=\delta_{k}^{b}\in{A_{1}}$. On the other hand, the prosumer agents control the household’s energy storage charge/discharge state in response to the dynamic buy price changes. Therefore, the action for $n^{th}$ prosumer’s agent is defined as ${a_{n,k}}\in{A_{n}}$ which denotes the charge/discharge command to the energy storage.

Finally, the immediate reward function for the grid and prosumer agents are defined as,

where $\rho\in\left\{{0,1}\right\}$ in which $\rho=0$ when $P_{n,k}^{inj}\leqslant 0$ and $\rho=1$ when $P_{n,k}^{inj}>0$.

According to the above terminology, the MDP trajectories for the grid agent and $n^{th}$ prosumer’s agent begin with ${s_{1,1}},\ {a_{1,1}},\ {r_{1,2}},\ {s_{1,2}},\ {a_{1,2}},\ {r_{1,3}},...$ and ${s_{n,1}},\ {a_{n,1}},\ {r_{n,2}},\ {s_{n,2}},\ {a_{n,2}},\ {r_{n,3}},...$, respectively. The primary goal of each agent is to maximize the accumulative reward sequence formulated by ${G_{1}}={\sum}\limits_{t=0}^{\infty}{{{\left({{\gamma_{1}}}\right)}^{t}}}\,{r_{1,k+t+1}}$ and ${G_{n}}={\sum}\limits_{t=0}^{\infty}{{{\left({{\gamma_{n}}}\right)}^{t}}}.\,{r_{n,k+t+1}}$ where $0\leqslant{\gamma_{1}}\leqslant 1$ and $0\leqslant{\gamma_{n}}\leqslant 1$ are discount factors and $n=2,3,...,{N_{p}}+1$. In this work, to find the optimal policy, Deep Q-Network (DQN) [21] is deployed to approximate the optimal action-value function which satisfies the Bellman Equation as,

where ${\hat{Q}}$ is the approximation of $Q$ that is estimated by a deep neural network. In this work, we use the updating mechanism provided in [22] for Q-values.

The proposed DR scheme is implemented on a small-scale microgrid similar to Fig. 1. The case study system is comprised of $(N_{p}=3)$ prosumers equipped with solar rooftop panels, energy storage system and smart agent, a conventional consumer representative of non-generational consumer $(N_{c}=1)$, and two generation facilities $(N_{g}=2)$ where one acts as a baseline generation facility and another acts as a reserve generation capacity. The prosumers’ details are provided in Table. I. Employed generation and consumption profiles for the prosumers are provided in Fig. 3, where the generation and consumption profiles are derived from California ISO [23].

Two scenarios have been simulated for analyzing the efficacy of the proposed DR scheme. In the first scenario, referred to as conventional scenario hereinafter, no DR scheme is applied to the microgrid. The prosumers simply inject power to grid when there is an excess energy generation in the household and the battery capacity is full. For the second scenario, the proposed DR scheme is implemented on the microgrid system and results are compared with the first scenario to evalute the effectiveness of the proposed method.

Fig. 4 compares the time domain profiles of the SoC of the prosumers and the buy/sell prices over a 24 hour period between the conventional and agent-based scenarios after fully training the DQN agents. Comparing Fig. 4 (b)-(d) with Fig. 3 (a), it can be observed that in the conventional scenario the changes in the SoC of the battery of each prosumer is very closely synced with the prosumer’s PV generation profile. This is expected since as mentioned above in the conventional scenario prosumers inject power to grid when there is an excess energy generation. On the other hand, in the agent based scenario, the battery SoC is changed as a result of charge/discharge commands issued by the prosumer agents based on the identified optimal charge/discharge policy. To compare the effect of this significant change in battery usage between the two scenarios, prosumers’ average daily electricity bill, grid daily profit and reserve power consumption for the two scenarios have been calculated and plotted in Fig.5. According to this figure, the prosumers’s average daily electricity bill is reduced significantly in the agent-based scenario. Similarly, the grid profit is considerably higher in the agent-based scenario which can be attributed to the significant drop in the reserve generation usage in this scenario.

According to the findings discussed above, it can be concluded that the grid and prosumer agents are leveraging the battery capacity of the prosumers to maximize their profits and reduce their costs.

In the next experiment, the effect of battery capacity on reduction of daily electricity bill of prosumers and raising the grid profit is investigated by running several agent-based simulations and increasing the battery capacity of the prosumers for each simulation. The results are shown in Fig. 6 and Fig. 7, respectively. As pictured, the battery capacity is increased from 2 kWh to 25 kWh and the daily energy bill of prosumers and the grid profit are measured at the end of simualtion and plotted against the battery capacity. Fig. 6 and Fig. 7 show a downward trend in the daily electricity cost of prosumers and an upward trend in the grid profit as a function of battery capacity. However, these trends seem to slow down around 15 kWh battery capacity, meaning that the improvements when using batteries with larger than 15 kWh capacity seem negligible. Therefore, for a given PV generation capacity (i.e., $P^{pv,\max}$), it can be concluded that the proposed DR scheme can provide maximum benefits with a reasonable battery pack size of around 15 kWh in the households.

This paper proposes a new multiagent RL-based decision-making environment for implementing a DR scheme in a microgrid dominated by prosumers. The proposed technique implements a Real-Time Pricing scheme that can mitigate several shortcomings common to traditional DR methods while providing important economic benefits to the grid operator and prosumers. To showcase the better efficacy of the RL-based method, this work includes a comparison to a baseline traditional operation scenario in a small-scale microgrid system. Results showed significant daily bill reductions (e.g., 38% , 46%, and 26%) for prosumers in the proposed RL-based marketplace . Moreover, experiments on the use of prosumers’ energy storage capacity in this microgrid setup highlight the advantages of the proposed method in establishing a fair and balanced market setup.