Distributed Energy Trading and Scheduling among Microgrids via Multiagent Reinforcement Learning

We illustrate the reference system of interconnected microgrids for energy trading and scheduling in Fig. 1. Several microgrids are interconnected with each other through power delivery system. Each microgrid consists of the following components: electrical loads, batteries, and renewable energy generators. The electrical loads are equipments which consume electricity, the batteries are utilized to store surplus electricity. The renewable energy generator can generate electricity from renewable energy sources, such as solar energy or wind energy. A microgrid can purchase electricity from the main grid on the wholesale market or from other microgrids on the hour-ahead energy market. It can also its sell its surplus energy to other microgrids on the hour-ahead energy market.

We consider two types of energy markets, namely, the wholesale market and the hour-ahead bidding market. We illustrate the trading flows for the microgrids on the hour-ahead market in Fig. 2. At the beginning of each time slot, a microgrid needs to determine how to schedule the local electrical loads, energy storage, and the bidding price and quantity on the hour-ahead market. The electricity from the power station is sold on the wholesale market and the price is known in advanced. A microgrid can trade with other microgrids on the hour-ahead bidding market. Each microgrid determines its bidding (buying or selling) price and quantity. The bidding price and quantity of each microgrid will be submitted to the bidding system without disclosing to others. When the bidding period elapses, the bidding system will calculate the final market clearing price and the trading quantity of each microgrid based on the adopted double-auction trading algorithm. The payment for each period will be made based on the real electricity usage of each microgrid.

We consider $N$ interconnected microgrids. The $i$-th microgrid is denoted as $M_{i}$, where $i=1,2,...,N$. We adopt a discrete time system in which the time is denoted as $t=0,1,2,...$. The duration of a time slot is one hour. Each microgrid consists of the following components.

During a time slot, a microgrid may charge or discharge the battery for energy storage or utilization. The changes of the battery level of the batteries in a microgrid due to the charging and discharging operations can be modeled as

where $\beta_{i}^{c}$ and $\beta_{i}^{d}$ are the charging and discharging efficiency, $c_{i}(t)$ is the amount of energy charged to the batteries in microgrid $i$ during time slot $t$, and $d_{i}(t)$ is the amount of energy discharged from the batteries in microgrid $i$ during time slot $t$. We assume that either charging or discharging can be performed to the batteries of a microgrid during a time slot.

At the beginning of a time slot, each microgrid determines whether it should participate in the energy trading of the next time slot on the hour-ahead market to sell or to buy electricity. Each microgrid determines the trading price and the trading quantity that will be submitted to the bidding system. The final market clearing price and trading quantity will be determined based on the double auction mechanism.

We formulate the energy trading and scheduling among microgrids as a Markov Game with continuous action spaces. Each microgrid is an agent, which aim is to learn the optimal policy to maximize its overall rewards.

We first briefly introduce why we choose MADDPG for learning the optimal control policy for distributed energy trading and scheduling among microgrids.

We illustrate the network architecture of MADDPG in Fig. 4. Each agent consists of an actor and a critic. The actor maps the state to the control action. The critic evaluates the advantage of the actions under the given states compared to the average actions. We denote the actor network of agent $i$ as $\mu_{i}(a_{i}|s_{i})$, where $s_{i}$ is the state of microgrid $i$ and $a_{i}$ is the corresponding control action for microgrid $i$. The actor network of an agent inputs the local state and outputs a control action. We denote the critic network of agent $i$ as $Q_{i}(s_{1},a_{1},...,s_{i},a_{i},...,s_{N},a_{N})$. The critic network inputs the states $(s_{1},s_{2},...,s_{N})$ and the actions $(a_{1},a_{2},...,a_{N})$ of all agents and outputs a real-valued evaluation.

Each agent has an individual critic network, which allows the agents to have different reward structures, and a microgrid can learn its optimal policy based on its objectives in a cooperative, competitive, or mixed environment. We only use the critic networks for training the actor network of each agent. After training, the critic networks are no longer required, and we only use the actor network of an agent for making control action for the corresponding microgrid. Thus, it will not relay on the states and actions of other microgrids.

We illustrate the implementation of the actor network and the critic network in Fig. 5. The outputs of an actor network include the selling price, the bid price, the trading quantity, and the charging quantity. Because an agent can only be either a seller or a buyer within one time slot, we use the value of the output of trading quantity to determine its role. Specifically, if the output value of trading quantity is larger than zero, the agent will be a buyer, and the bid price and the trading quantity will be submitted to the hour-ahead market for biding. If the output value of the trading quantity is less than zero, the agent will be a seller, and the selling price and trading quantity will be submitted for bidding on the hour-ahead market.

In the critic network of an agent, the states of all agents will be input into a Fully-Connected (FC) layer. The output vector of the FC layer will be concentrated with the control actions of all actors, and then input the second FC layer. Note that the main grid price and the last clearing price are public information, which are same for each agent. Therefore, they will be input into the critic network of an agent only once to avoid duplication and reduce the input dimension of the critic networks. The activation function of the actor network is Tanh function, and the range of each output is from $-1$ to $1$, and we will convert them to the actual range of each control action. The activation function of the critic network is a linear function and the output is a real value.

We now describe how to train the actor network and the critic network of each agent. The learning process can be conducted in a simulation environment. The details of the training algorithm are illustrated in Algorithm 1.

At the beginning of each time slot $t$, each agent $i$ first observes the state $s_{i}$ of microgrid $i$. State $s_{i}$ will be input into the actor network $\mu_{i}$ of agent $i$ and output a control action. For the exploration of the state space, we will add an exploration noise to the control action, and the final control action is

where $\mathcal{N}$ is the exploration noise. We adopt an Ornstein-Uhlenbeck process [23] to generate the exploration noise.

After obtaining the control action for each agent $i$, the control actions will be applied in the corresponding microgrids. The simulation environment will simulate the energy scheduling process and the energy trading process according to the specified control action for each microgrid. At the end of a time slot, each agent $i$ can calculate its reward $r_{i}$ during the time slot by Eq. (11) and observe new state $s^{\prime}_{i}$. The state, action, reward, and new state information of each agent during a time slot will be stored as a transition in the replay buffer as $(\mathbb{s},\mathbb{a},\mathbb{r},\mathbb{s^{\prime}})$, where $\mathbb{s}=(s_{1},...,s_{N})$, $\mathbb{a}=(a_{1},...,a_{N})$, $\mathbb{r}=(r_{1},...,r_{N})$, and $\mathbb{s^{\prime}}=(s^{\prime}_{1},...,s^{\prime}_{N})$.

At the end of each time slot, the actor network and the critic network of each agent will be trained with the transitions sampled from the replay buffer. To stabilize training, a copy of the actor networks and the critic networks will be created as target networks for slowly tracking the learned networks. We denote the target actor network and the target critic network of agent $i$ as $\mu^{\prime}_{i}$ and $Q^{\prime}_{i}$, respectively. To train the actor network and the critic network of an agent, we randomly select $S$ transitions from the replay buffer. For transition $j$, we denote it as $(\mathbb{s}^{j},\mathbb{a}^{j},\mathbb{r}^{j},\mathbb{s^{\prime}}^{j})$. The target Q-value of this transition for agent $i$ will be calculated as follow,

where $\gamma$ is the discounting factor and $s^{\prime}_{k}$ is the $k$-th item in $\mathbb{s^{\prime}}^{j}$ (i.e., the next state of agent $k$). The critic network of agent $i$ will be updated by minimizing the following loss,

where $\theta_{i}$ denotes the network parameters of agent $i$. The actor network of agent $i$ will be updated using the policy gradient (see details in [17]). The target network parameters of each agent $i$ will be updated using the following equation.

where $\theta^{\prime}_{i}$ represents the network parameters of the target network of agent $i$ and $\tau$ is the learning rate.

We denote the actor network of agent $i$ after training is $\mu_{i}^{*}$. The control action $a_{i}$ for microgrid $i$ can be directly obtained by observing the state $s_{i}$ of microgrid $i$ and inputing $s_{i}$ into the actor network, mathematically denoted as $a_{i}=\mu_{i}^{*}(s_{i})$. Thus, the computational complexity for obtaining a control action is only determined by the complexity of the actor network, which include the number of hidden layers and the number of neurons in each hidden layer. In this work, the actor network of an agent consists of two hidden layers. We suppose that each hidden layer of the actor has $n$ neurons, the complexity for making a control action for an agent is $O(n^{2})$.

We simulate the energy trading and scheduling among four microgrids with different load characteristics and renewable energy generation capacities. The electrical load and the renewable energy generation of each microgrid during each time slot are scaled from real-world datasets. We illustrate parts (500 hours) of the load profiles and the renewable energy generation profiles of each microgrid in Fig. 6 and Fig. 7, respectively. Microgrid 1 and 2 have more renewable energy generations and surplus energy to sell. Microgrid 3 and 4 have heavier loads and less renewable energy generations.

The electricity price on the wholesale market is 22.79 cents/kWh in Singapore. The minimum bidding price on the hour-ahead market is 15.0 cents/kWh, and the maximum bidding price is 22.79 cents/kWh, which is equal to the wholesale market. We use the solar radiation data of one year at four locations of Singapore, which are obtained from Solcast [24], to simulate the solar radiation in each microgrid. We use the scaled historical electricity system demand data for every half-hour period in Singapore to simulate the electrical loads of the microgrids [25]. The maximum battery capacities of Microgrid 1-4 are 100MWh, 100MWh, 20MWh, 10MWh, respectively. The maximum bid quantity for each microgrid on the hour-ahead market is 7.5MWh.

The initial exploration noise scale is 1.0, and the exploration noise linearly decreases over 1000 training episodes till zero. The models are trained over 1500 episodes, and the performances are evaluated over 1000 episodes. Each episode consists of $7*24$ time steps, and each time step in the simulation is one hour. The batch size is 1024. The learning rate is 0.01. The discount factor of the reward is 0.8. Each critic network has three hidden layers, the first layer has 1024 neurons, the second layer has 512 neurons, and the third layer has 256 neurons. Each actor network has two hidden layers, the first layer has 512 neurons, and the second layer has 128 neurons. The activation function is ReLU function.

To evaluate the convergences of different agents in MADDPG, we train the models five times under the same settings with different random seeds. We illustrate the rewards of different agents at different training episodes in Fig. 8, respectively. The shadow areas in the figures represent the standard deviations of the rewards of the five training times at the different episodes. We can observe from these figures that the rewards of different agents all converge.

The converged rewards of Microgrid 1 and 2 are larger than zero. This is because Microgrid 1 and 2 have larger capacities for renewable energy generation than their own consumptions. Therefore, they can sell their surplus energy on the hour-ahead market and gain revenues. Their revenues are larger than their cost, therefore, their converged rewards are larger than zero. The rewards of Microgrid 3 and 4 also increase during the training, because they can save more cost by improving the trading policy on the hour-ahead market. By trading on the hour-ahead market, they can save more cost compared with buying energy from the wholesale market.

The results illustrated in Fig. 8 verify the feasibility of adopting the multiagent approach for learning the optimal control policy for each microgrid. The four microgrids in the studied case are mixed cooperative-competitive. Microgrid 1 and 2 are competitors for energy selling, and Microgrid 3 and 4 are competitors for energy buying. Microgrid 1 and 2 also collaborate with Microgrid 3 and 4 for energy trading for mutual benefit. The convergences of the agents verify that the agents can find the equilibrium in the mixed cooperative-competitive environment for maximizing their rewards.

We analyze each agent’s performances by illustrating the characteristics of the system states and the control actions of each microgrid. We collect the control actions and the system states during each time slot over 1000 testing episodes. The distributions of the system states and the control actions are estimated with Kernel Density Estimation (KDE) and Histogram (Hist) to analyze the characteristics of each microgrid.

We illustrate the distribution of the bid price and the trading quantity of different microgrids on the hour-ahead market in Fig. 9 and 10, respectively. In our definition of the control action of each agent, if the price of an agent during one time step is larger than zero, the agent is a seller during the time slot. If the price is less than zero, the agent is a buyer during the time slot. In Fig. 9 and 10, we can observe that Microgrid 1 and 2 are mainly as the sellers on the hour-ahead market, because they have more surplus energy, which incentives them to sell to obtain revenues. Microgrid 3 and 4 are mainly as buyers on the hour-ahead market due to lower capacities of energy generation. The distributions of bid price and trading quantity are in line with the characteristics of each microgrid’s energy consumption and generation capacity.

The distributions of the selling price of Microgrid 1 and 2 are sharp, while the buying price of Microgrid 3 and 4 are flatter, because Microgrid 1 and 2 are more competitive on energy selling, and the renewable energy may be wasted if batteries have been fully charged and the generation exceeds the consumption. On the contrary, Microgrid 3 and 4 may change buying price based on their varying demands.

The distribution of the market clearing price on the hour-ahead market is illustrated in Fig. 12. The market clearing price mostly lies within the range (15, 20), which is much lower than the wholesale market price. Therefore, Microgrid 3 and 4 have the incentives of buying from the hour-ahead market to reduce cost. In some cases, there is no trade made on the hour-ahead market because the selling price is too high or the buying price is too low.

We illustrate the distribution of the overall trading quantity during a time slot on the hour-ahead market in Fig. 13. The microgrids prefer to submit the trading quantity at the maximum allowed quantity, because Microgrid 3 and 4 can always save cost by buying more from the hour-ahead market if they can consume. On the contrary, Microgrid 1 and 2 can get more revenues by selling more, if the trading quantity does not exceed the surplus. Therefore, the trading quantity of a microgrid has a large probability of being on the threshold of the maximum allowed trading quantity.

We illustrate the distribution of the battery levels of different microgrids in Fig. 11. The battery levels of Microgrid 3 and 4 are low during most of the time slots, because they have low capacity of energy generations. For Microgrid 1 and 2, their batteries may be fully charged, if they have too much surplus energy and cannot sell on the energy market, which incentives them to sell on the hour-ahead market. Therefore, the battery capacity is more important for the microgrids with larger capacity of renewable generation.

We compare the performances of our proposed method with the following baseline methods.

1) Isolated: the microgrids operate in an isolated mode, in which each microgrid can only supply the generated renewable energy to itself or charge the surplus energy to its battery. A microgrid can only buy energy on the wholesale market to meet its demand when local generation is deficient.

2) DDPG: the microgrids can trade with each other, yet the agent of each microgrid is trained independently with DDPG, which is a single-agent reinforcement learning approach. The agents will not share information with each other.

We illustrate the comparisons of the cost, revenue, and the overall reward per time slot for Microgrid 1-4 in Fig. 14, 15, 16, 17, respectively. In the X-axises of these figures, Wholesale represents the cost for buying from the wholesale market, Buying represents the cost for buying on the hour-ahead market, Selling represents the revenue for selling on the hour-ahead market, Penalty represents the penalty for violating the selling contract, and Overall represents the overall reward of the sum of the costs and revenue.

As illustrated in Fig. 14-17, the four microgrids can obtain a higher overall reward under our proposed method compared with the baselines. The results show that the multiagent deep reinforcement learning approach can effectively improve the reward of each agent by collaboratively learning the policy for each agent compared with the baselines.

As illustrated in Fig. 14 and 15, Microgrid 1 and 2 can obtain revenues by energy selling on the hour-ahead market. Thus, the overall rewards under our method and DDPG are higher than Isolated. With Isolated, the cost, revenue, and the overall reward are all near to zero, because Microgrid 1 and 2 have enough renewable energy generation to meet its demands, but cannot gain revenues without energy trading.

With our method, Microgrid 1 and 2 can gain more revenue compared with DDPG. As illustrated in Fig. 18, this is partly due to that the successful trading ratio and the average trading quantity per time slot with our proposed method are higher than that of DDPG. Therefore, Microgrid 1 and 2 can obtain more revenues on the hour-ahead market, and Microgrid 3 and 4 can save more cost via energy trading.

Fig. 16 and 17 verify that our proposed method can reduce more cost for Microgrid 3 and 4 compared with the baseline methods. This is because Microgrid 3 and 4 can successfully buy more energy from the hour-ahead market compared with DDPG, and the price on the hour-ahead market is much lower than the wholesale market, thus it can save more cost compared with DDPG. On the contrary, with the Isolated method, all energy must be brought from the wholesale market with a higher price, it will incur more cost.