BESS Aided Reconfigurable Energy Supply using Deep Reinforcement Learning for 5G and Beyond

The power demand pattern of a BS is mainly determined by its location and associated with the behavior of users there. Usually the demand also show a periodic pattern (e.g., with a one-day or one-week period). As shown in Fig. 2, in this paper, we mainly consider three types of BSs at the areas of resident, office, and comprehensive, which account for nearly ninety percentage of the total demands [13]. To be detailed, the characteristics of these power demand patterns are as follows.

• •

  Power Demand of BSs at Resident Area: The power demands of this type of BSs increase rapidly in the evening, as most people stay at home after work. Compared with those in weekdays, the power demands keep at high-levels in weekends.

• •

  Power Demand of BSs at Office Area: The power demands of this type of BSs keep at the high-level in the day time, when most people work during the time. Besides, due to the fewer people work on the weekends, the weekend power demands are much lower than those on the weekdays.

• •

  Power Demand of BSs at Comprehensive Area: Due to the diversity of the requests, compared to the above two BSs, the power demand patterns of this type of BSs are more stable: constantly keep at a high-level in the day time and evening and drop down to the valley in late night and early morning.

The first two types of power demand patterns change relatively dramatically, leading to a huge energy-saving potential, especially for the demand charge, which will be discussed in the next section.

The energy cost of the mobile operator typically makes up of two components: i) energy charge, i.e., the total consumed electricity amount (in kWh) throughout the entire billing cycle (e.g., one month), and ii) demand charge, i.e., the peak power demand (in kW) during the billing cycle period. Specifically, the demand charge is regarded as a penalty due to the caused extra load burden to the power grid.

For example, for a commercial data center consuming 10 MW on peak and 6 MW on average, the monthly energy charge and demand charge amounts to around $24,000 and $165,500, respectively [14]. The demand charge could be up to 8x the energy charge, therefore, effectively cutting down the demand charge could remarkably reduce the energy cost. However, there seems no practical way to flatten the peak power demands of 5G BSs, e.g., shifting the real-time demands from mobile users to the off-peak hours could lead to the long delay for some of the classes of jobs [15].

As illustrated in Fig. 3, the proposed BESS aided renewable energy supply solution deployed at each 5G BS mainly includes: i) a renewable energy generator, e.g., the PV panel and wind turbine, which is deployed near the 5G BS system and generates renewable energy for the system, ii) a battery storage, which stores the surplus renewable energy and acts as the power source for the BS as needed, and iii) a controller, which can obtain the environment state (i.e., the measurement data) so as to control the battery discharging/charging operations through the control signals. In addition to the standard meter, as shown in Fig. 3, an additional generation meter is installed for the BS power supply system to measure the renewable energy generation. Furthermore, with commands from the controller, the distribution panel takes responsibility of power switch between the renewable energy and grid energy and ensures continuous and stable electricity supply for the BS.

As the essential component of the BESS aided renewable energy supply solution, the controller determines how efficient this paradigm is. Specifically, at each scheduling point, the controller needs to decide the amount of power supply from either the battery or the power grid. The scheduling operations should be made upon the power demands and battery states in real-time, so that the utilization of renewable energy can be enhanced and the total energy cost can be minimized.

Note that the feasibility of such an implementation as illustrated by Fig. 3 has been preliminarily verified in practice. According to [16], small integrated renewable energy generators are provided by some commercial companies for the BS system, which are easily deployed in both open rural and crowded urban environments.

The power of each 5G BS is mainly supplied by three parts: power grid, generated renewable energy, and storage energy. In particular, i) when the generated renewable energy is more than the power demand (e.g., during the off-peak hours), each 5G BS is only supplied by the renewable energy (i.e., off-grid) and the surplus renewable energy is stored in the battery storage, ii) when the generated renewable energy is less than the power demand (e.g., during the peak hours), each 5G BS is supplied by all three parts in a cooperative way.

In this paper, we consider a discrete time model, where the entire billing cycle (e.g., one month) is equally spilt into $T$ consecutive slots with length of $\Delta t$ and denoted by $\mathcal{T}=\{1,2,\cdots,T\}$. For an arbitrary 5G BS, the power demand during the entire billing cycle can be represented by a power demand vector:

where $d(t)$ is the power demand in time slot $t$, which can be obtained by power meter readings at each BS.

By harvesting energy from renewable energy resources, the BSs could be powered in an environmentally-friendly and cost-efficient way. In this paper, in order to make the model extensible, we denote the renewable energy generation vector as:

In this work, we choose two typical renewable energy as the auxiliary way of power supply, i.e., solar energy (i.e., $g^{s}(t)$) and wind energy (i.e., $g^{w}(t)$). Accordingly, for an arbitrary time slot $t$, the renewable energy generation vector can be represented by:

We assume that if the total generated renewable energy is beyond the power demand (i.e., $g(t)>d(t)$), the power is supplied in proportion to the renewable energy generated. The generation of both varies during a certain period (e.g., one day) and is affected by a some similar factors such as weather, temperature, wind speed, and so on.

At an arbitrary time slot $t$, the state of the battery is modeled as follows:

where the notations of SoE, SoC, and DoD represent the state of effective capacity state of charge, and depth of discharge of the battery, respectively. Specifically, i) SoE indicates the current effective capacity of the battery, as a percentage of its initial capacity (denoted as $\pi$), ii) SoC indicates the current energy stored in the battery, as a percentage of the current effective capacity, and iii) DoD indicates how much energy the battery has released, as a percentage of the current effective capacity.

For simplicity to tackling the optimization problem, we discretize the SoC of a battery into $M$ equal-spaced states (e.g., $M=10$, i.e., $\{10\%,20\%,\cdots,100\%\}$). Accordingly, the DoD are also discretized (e.g., release $10\%$ from $90\%$, i.e., $90\%\to 80\%$). Besides, for an arbitrary time slot $t$, in order to prevent the battery from over-discharging/charging, we use $SoC_{max}$ and $SoC_{min}$ to indicate the upper and lower bounds of SoCs, respectively, which is shown as follows.

The billing policy of the energy cost for the mobile operators throughout the entire billing cycle typically make up of two components, energy charge and demand charge, which is widely applied in previous [14, 15, 20]. And we will introduce them in detail as follows.

• •

  Energy Charge: the total consumed electricity amount (in kWh) throughout the entire billing cycle (in the unit $kWh and denoted by $\lambda_{e}$).

• •

  Demand Charge: the peak power consumption supplied by power gird (in kW) during the entire billing cycle (in the unit $kW and denoted by $\lambda_{d}$).

Therefore, the incurred cost of energy charge of the whole system in each time slot $t$ can be represented by:

Accordingly, the incurred cost of demand charge of the whole system in each time slot $t$ can be represented by:

where $p_{max}$ records the peak power consumption during the past $t-1$ time slots. For any arbitrary time slot $t$, if $p(t)-p_{max}>0$, $p_{max}$ will be updated to $p(t)$ accordingly.

Every usage of this equipment (solar PV, wind turbine, and battery storage) incurs a certain reduction of its lifetime, which is essential for the investor. Therefore, it is significant to understand, detail and quantify the various factors influencing the performance loss curves. For the accuracy of our model, we quantify the investment cost in every time slot as follows.

The battery discharging/charging operations is controlled by the controller. Given the state (i.e., $\chi(t)$) of the battery storage in time slot $t-1$, the state in time slot $t$ can be updated by:

For the entire billing cycle $\mathcal{T}$, we need to find the optimal battery discharging/charging controlling policy to solve the optimization problem, so as to minimize the total electricity bill during the entire billing cycle, which is defined as follows.

When solving the above optimization problems, however, we are faced with the following three challenges.

A typical DRL framework consists of five key components: agent, state, action, policy, and reward. The concept and design of each component in our DRL-based battery discharging/charging controlling policy is explained as follows.

• •

  Agent: The role of the agent is to make decisions in every episode by interacting with the environment. Specifically, at the beginning of each time slot, it determines the discharging/charging operations (i.e., $b(t)$) according the current state (e.g., $d(t)$, $g(t)$ and $\chi(t)$) of the environment. The objective is to find an optimal battery discharging/charging controlling policy to minimize the total electricity bill during the entire billing cycle.

• •

  State: At each episode, the agent first observes the state of the current environment to take action. In order to take the optimal action at each episode, the current state should cover as much information as possible. In this paper, we define the state vector of the current environment as $s(t)=[d(t),g(t),\chi(t),p_{max}]$, which concludes the current information of the power demand, the renewable energy generation, the battery storage and the peak power consumption.

• •

  Action: After observing the state of the environment, the agent will take an action accordingly. In our problem, the action is to control the battery discharging/charging operations in each time slot, i.e., $b(t)$, specifically, i) whether the battery should be discharged or charged, and ii) how much energy should be discharged or charged. We denote the action taken at time $t$ by $a(t)$, which is equivalent to $b(t)$.

• •

  Policy: The battery discharging/charging controlling policy $\psi(s(t)):\mathcal{S}\to\mathcal{A}$ defines the mapping relationship from the state space to the action space, where $\mathcal{S}$ and $\mathcal{A}$ represent the state space and the action space, respectively. Specifically, the controlling policy can be represented by set of $a(t)=\psi(s(t))$, which maps the state of the environment to the action at time slot $t$.

• •

  Reward: After interacting with the environment, the agent will receive a reward $r(t)$ (calculated by the reward function $R(s(t),a(t))$), which indicates the effect of the action in this episode, so as to update the controlling policy. The objective of the agent is to find a policy $\psi$ to maximize the total reward through continuous interacting with the environment. The design of the reward function significantly affect the performance of the DRL-based algorithm, and we will introduce its detail in the next subsection.

To sum up, at each episode, the agent observes the state $s(t)$, takes an action $a(t)$ generated by the policy $\psi$, and receives a reward $r(t)$ calculated by the reward function $R(s(t),a(t))$. The objective of the proposed DRL-based battery discharging/charging controlling policy is to take the optimal action in every episode so as to maximize the total reward.

At the end of each time slot, the agent evaluates the performance of the action using a reward function, which transforms the performance statistics to a numerical utility value. For an arbitrary time $t$, the agent observes the state $s(t)$, takes the action $a(t)$ and adopts the following reward function to access the performance of the controlling action:

in which:

• •

  $V^{e}(t)=-\mathcal{C}^{e}(t)$, measures the reward of the incremental energy charge caused by the action in time slot $t$.

• •

  $V^{d}(t)=-\mathcal{C}^{d}(t)$, measures the reward of the incremental demand charge caused by the action in time slot $t$.

• •

  $V^{u}(t)=-\mathcal{C}^{u}(t)$, measures the reward of the investment cost caused by the action in time slot $t$.

At the end of each time slot, the agent evaluates the performance of the action by the reward $r(t)$ calculated by the reward function $R(s(t),a(t))$. In the DRL-based framework, the objective is to maximize the expected cumulative discounted reward:

where $\gamma\in(0,1]$ is a factor discounting future rewards.

The learning process of the algorithm adopts a deep neural network (DNN) called Deep Q-Network (DQN) to derive the correlation between each state-action pair $(s(t),a(t))$ and its value function $Q(s(t),a(t))$, which is the expected discounted cumulative reward. If the environment is in state $s(t)$ and follows action $a(t)$, the value function of the state-action $(s(t),a(t))$ can be represented as:

After obtaining the value of each state-action $(s(t),a(t))$, the agent selects the action $a(t)$ with the $\epsilon$-greedy policy $\psi$, that is, randomly selects the action with the probability of $\epsilon$, and chooses the action with the maximum of $Q(s(t),a(t))$ with the probability of 1-$\epsilon$, i.e., $\mbox{argmax}_{a(t)}Q(s(t),a(t))$.

As illustrated in Fig. 5, two effective techniques were introduced in [24] to improve stability: replay buffer and target network. Specifically,

• •

  Replay Buffer: Unlike traditional reinforcement learning, DQN applies a replay buffer to store state transition samples in the form of $\langle s(t),a(t),r(t),s(t+1)\rangle$ collected during learning. Every $\kappa$ time steps, the DRL-based agent updates the DNN with a mini-batch experiences from the replay buffer by means of stochastic gradient descent (SGD): $\theta_{i+1}=\theta_{i}+{\sigma}{\bigtriangledown}_{\theta}Loss(\theta)$, where $\sigma$ is the learning rate. Compared with Q-learning (only using immediately collected samples), randomly sampling from the replay buffer allows the DRL-based agent to break the correlation between sequentially generated samples, and learn from a more independently and identically distributed past experiences. Thus, the replay buffer can smooth out learning and avoid oscillations or divergence.

• •

  Target Network: There are two neural networks with the same structure but different parameters in DQN, the main net and the target net. $Q(s,a;\theta)$ and $Q(s,a;\tilde{\theta})$ represent the current Q-value and target Q-value generated by the main net and the target net, respectively. The DRL-based agent uses the target net to estimate the target Q-value $\tilde{Q}$ for training the DQN. Every $\tau$ time steps, the target net copies the parameters from the main net, whose parameters are updated in real-time. After introducing the target net, the target Q-value will remain unchanged for a period time, which reduces the correlation between the current Q-value and the target Q-value and improves the stability of the algorithm.

Accordingly, the DQN can be trained by the loss:

where $\theta$ is the network parameters of the main net, and $\tilde{Q}$ is the target Q-value and calculated by:

where $\tilde{\theta}$ is the network parameters of the target net and it updates every $\tau$ time slots by coping from the main net.

To sum up, the learning process is depicted by the pseudo-code in Alg. 1. The controller first initializes the replay buffer and the parameters (i.e., $\theta$ and $\tilde{\theta}$) of the main net and target net, respectively. After obtaining the value of each state-action $(s(t),a(t))$, the agent selects the action $a(t)$ with the $\epsilon$-greedy policy $\psi$, and then performs the action $a(t)$ and interacts with the environment. Next, the agent will receive the reward $r(t)$ and observe the next state $s(t+1)$ of the environment, meanwhile store the state $\langle s(t),a(t),r(t),s(t+1)\rangle$ into the RB. Every $\kappa$ time steps, the agent updates the DNN by Eq. 30 with a mini-batch experience from the replay buffer by means of stochastic gradient descent (SGD). The target net will copy the parameters of the main net by every $\tau$ time steps. During the learning process, we set the learning rate $\sigma$ is 0.001, the $\epsilon$ in $\epsilon$-greedy method is 0.9, the discount accumulative factor $\gamma$ is 0.9, and the step parameters $\tau$ and $\kappa$ are both 2000.

As is shown in Fig. 6, the output power patterns of the solar PV and wind turbine are both divided into three types under different weather conditions. Accordingly, the weather pattern can be divided into nine types: clear & high-wind day, clear & middle-wind day, clear & low-wind day, partial cloudy & high-wind day, partial cloudy & middle-wind day, partial cloudy & low-wind day, cloudy & high-wind day, cloudy & middle-wind day, and cloudy & low-wind day.

The power supply patterns under different weather conditions in one day period of 5G BS at the area of resident, office, and comprehensive are illustrated in Fig. 8, Fig. 9, and Fig. 10 (in the appendix), respectively. As we can see, the BESS aided renewable energy supply solution could significantly reduce the power from the grid (i.e., energy charge and demand charge). Specifically, with the increase of radiation and wind velocity, renewable energy generation increased accordingly. It could cover most of the power demand and reduce the power supplied from the power grid. Especially, under high-wind days, the power demand could be totally supplied by the renewable energy and battery storage and need 0 power from the grid.

After calculating the power supply paradigm under different weather patterns, we can derive the electricity bill of these three types of BSs during the billing cycle in different cities (i.e., different weather patterns, which is illustrated in Fig. 7), and the results from all the set scenarios are summarized in Table III.

Specifically, for a single 5G BS without the proposed power supply paradigm, the energy charge and the demand charge are $45.6 and $22.8, respectively. However, after utilizing the BESS aided renewable energy supply solution on the 5G BS, the electricity bill is significantly reduced. Especially in Shanghai, which has relatively more clear and high-wind days, the energy charge and the demand charge can be reduced to $3.8 and $9.1, respectively. Although there exists equipment degradation during the discharge/charge cycles, the investment cost still keeps at a well accepted level. The highest cost saving for the BS which utilized the proposed power supply paradigm in Beijing, Shanghai, and Guangzhou in one billing cycle is $50.4, $50.7 and $49.5, respectively. Accordingly, the saving ratio can be up to 74.4%, 74.8% and 73.2%, respectively.

As the different types of BSs has diverse power demand, resulting in different energy charge and demand charge, thus the performance of deployment of the BESS aided renewable energy supply solution could be different.

Specifically, as is shown in Table III, the type I BS has the highest cost saving compared to other two types of BSs, i.e., $50.4 in Beijing, $50.7, and $49.5. This is because that type I BS has the biggest power demand and peak value (near 1450 watts), making it has great potential in energy-saving and peak power shaving. Besides, as type II BS’s power demands are relatively small, the generated and stored renewable energy can effectively reduce the power grid supply. Therefore it has the highest saving ratio, i.e., 76.4% in Beijing, 77.9% in Shanghai, and 75.6% in Guangzhou.

The return of investment (ROI) is a financial metric defined by the benefit (cost saving in our case) divided by the total investment. It indicates the probability of gaining a return from an investment and has been widely used to evaluate the efficiency of an investment [30]. Typically, a bigger ROI value indicates a higher investment efficiency. With the costs of renewable energy generator and battery storage (given in Table II), the total investments can be calculated. Accordingly, the ROIs can thus be derived with the results in Table III.

The ROIs of different types of BSs deployed in different cities are shown in Table IV. Specifically, type I BS has the highest ROI, which could reach to 5.43% in Beijing, 5.46% in Shanghai, and 5.33% in Guangzhou, respectively, indicating a relatively high investment efficiency for the operators. This is because that type I BS has the biggest cost saving.

As the equipment’s cost is estimated to decrease dramatically in the future [31], and the ROI could rise significantly in 5G and beyond. Additionally, as we can see, the city with more clear and high-wind days will obtain a bigger ROI value, thus the proposed solution is more suitable for those cities with more sunny and windy days.

It is worth noting that, we assume the deployed renewable energy generator and the battery storage only supply power to one single 5G BS, and thus the surplus renewable energy (when the battery is full) will be discarded. This actually leads to a relatively low utilization, as given in this work. In practice, the generated renewable energy could supply to multiple BSs [7], so that the ROI and utilization of the renewable energy could be further improved.

With the increase of the BS power consumption, the energy-efficient design of cellular networks has recently received significant attention. Typically, the BS energy-saving methods are divided into three levels, equipment-level energy saving, site-level energy saving and network-level energy saving [32].

On the equipment-level, researchers propose new schemes (e.g., the scheme in this paper), new materials (e.g., semiconductor material), and new functions so as to achieve the energy-efficient goal. Besides, some liquid heat dissipation, high power amplifier efficiency, and high integration applications are applied in the BS so as to reduce the power consumption of the whole machine year by year.

On the site-level, one common scheme is to switching-on/off the BS related with the traffic load [33, 34, 35]. To be detail, switch-on the BS when the traffic load is large, and switch-off the BS when the traffic load is low. In addition, by combining with AI, the accuracy of the traffic load prediction can be improved so that the corresponding energy-saving policies can be elaborately formulated.

On the network-level, With the deployment of the 5G network, multiple networks (e.g., 4G/LTE and 5G) coexist in the current network, so the energy efficiency can be improved through the application of network-level energy-saving technology. Based on the basic data (e.g., configuration and performance) of the network and the built-in strategy, the multi network cooperative energy-saving technology [36] can realize the goal of reducing energy consumption by turning off the BS under the condition of ensuring service quality.

Peak power (i.e., the demand charge) is a sensitive factor for the power grid, as it occurs occasionally and takes place only for a small percentage of the time in a day [37]. The traditional solution is to increase the grid capacity, leading to uneconomic and inefficiency to supply peak power. The peak power shaving is a preferable approach to overcome these disadvantages, making the load curve flatten by reducing the peak amount of load and shifting it to times of lower load [38]. Typically, the related works are divided into the following three categories:

Peak shaving using energy storage system (ESS): Integrating energy storage systems to the grid is the most potent strategy of peak shaving due to its economic benefits [39, 40, 41]. Specifically, peak power shaving is achieved through the process of charging ESS when demand is low (off-peak period) and discharging when demand is high (one-peak period).

Peak shaving using electric vehicles (EV): Since the storage energy of electric vehicles is usually not fully utilized each day, this technology has the potential to provide peak shaving service [42, 43, 44]. For example, Alam et al. [45] proposed an effective strategy to utilize PEV batteries for both traveling and peak shaving purpose.

Peak shaving using demand side management (DSM): Demand side management refers to the programs that may influence the customers to balance their electricity consumption with the power supply system’s generation capacity [46, 47, 48, 49]. For example, Rozali et al. [50] aimed to achieve maximum peak shaving through DR under power shaving analysis.

The optimal control of energy storage has been extensively studied in the past. Most related works formulate an optimization problem that aims to maximize the revenue generated by the battery storage co-located with renewable energy generator.

Babacan et al. [51] proposed a convex program to minimize the electricity bill of operators. Ratnam et al. [52] aimed to maximize the daily operational savings that accrue to customers while penalizing large voltage swings stemming from reverse power flow and peak load. Kazhamiaka et al. [53] studied the profitability of residential PV-storage systems in three jurisdictions and set up an integer linear program to determine the battery controlling policy. These works assume the generations of renewable energy and the power demand are known in advance and can be optimized in an offline way. However, these assumptions are unpractical in the real world.

Several papers study the optimal control of batteries under uncertainty and randomness. Guan et al. [54] utilized a reinforcement learning method to minimize the homeowner’s cost by taking an action that yields the best expected reward. EnergyBoost [19] could provide a predictable ability of the renewable energy generation and power demand. However, these works are only applied in the home scenario, which generates a few demands compared to 5G BS. Therefore, we propose the DRL-based method to tackle the problem of large and constrained state- and action-space and the uncertainty of renewable energy generation and power demand.