Confidence Estimation Transformer for Long-term Renewable Energy Forecasting in Reinforcement Learning-based Power Grid Dispatching

In order to deal with renewable energy fluctuations caused by external factors in advance, this paper adopts the enhanced Transformer [27] to efficiently predict the long-term maximum active output of every renewable energy generator. The enhanced Transformer-based renewable energy prediction model, shown in Fig. 2 a, is divided into encoder and decoder.

For all renewable energy generators, ${\mathop{\textit{PR}}\nolimits_{{\mathop{t}\nolimits_{1}},{\mathop{t}\nolimits_{2}}}}$ represents real maximum active power from ${\mathop{t}\nolimits_{1}}$ to ${\mathop{t}\nolimits_{2}}$, and the sequence of the maximum active power predicted by the enhanced Transformer at time $t$ is noted as ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{1}},{\mathop{t}\nolimits_{2}}}^{t}}$. ${\mathop{\widehat{\textit{PR}}}\nolimits_{{t}+1,{t}+10}}$ represents the final prediction result for the next 10 days output by Conformer.

The encoder mainly carries out coding for known sequences and extracting features. Suppose that the current time is ${\mathop{t}\nolimits_{0}}$. In the enhanced Transformer, the input of encoder includes the real data in the past 56 days ${\mathop{\textit{PR}}\nolimits_{{\mathop{t}\nolimits_{0}}-55,{\mathop{t}\nolimits_{0}}}}$ and the corresponding four-dimension time code, whose sizes are $56\times\mathop{n}\nolimits_{new}$ and $56\times 4$, respectively. ProbSparse self-attention is adopted in the encoder, achieving $O(LlogL)$ time complexity and $O(LlogL)$ memory usage on dependency alignments. For tuple inputs, i.e, query $Q$, key $K$ and value $V$, ProbSparse self-attention allows each key to only attend to the $u$ dominant queries:

where $\overline{Q}$ is a sparse matrix of the same size as $Q$, and it only contains the Top-$u$ queries under the sparsity measurement. Meanwhile, self-attention distilling operation is introduced. Conv1D layers and Maxpooling layers are added after each ProbSparse self-attention layer to privilege dominating attention scores and help receiving long sequence input.

The decoder is responsible for predicting. The input of the decoder includes historical data and corresponding time coding besides the sequence characteristics output by the encoder. The data section includes real data in the past 28 days ${\mathop{\textit{PR}}\nolimits_{{\mathop{t}\nolimits_{0}}-27,{\mathop{t}\nolimits_{0}}}}$ and that in the next 15 days replaced by 0. Given the input, the decoder predicts the sequence in the next 15 days ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+1,{\mathop{t}\nolimits_{0}}+15}^{{\mathop{t}\nolimits_{0}}}}$. Therefore, the sizes of the input data and time encoding are $43\times\mathop{n}\nolimits_{new}$ and $43\times 4$ respectively. A generative style decoder is adopted to acquire long sequence output with only one forward step needed, simultaneously avoiding cumulative error spreading during the inference.

Combining the prediction results from different time can offset the contingency of single prediction result. This paper designs a confidence estimation method to represent the effectiveness of a prediction sequence as the combining weight.

$P\left(e_{i}\mid B\right)$ is the condition probability that the $i^{th}$ result in the prediction sequence is correct given the condition $B$ which means the accuracy of partial results is observed. $P\left(e_{i}\mid B\right)$ can be derived as

To facilitate the update of effectiveness, we use beta distribution to represent confidence of the prediction sequence according to the accuracy of historical prediction results, which is defined as

where $m>0\text{ and }n>0$ represent the number of correct prediction results and incorrect prediction results, respectively. In this paper, the prediction result is judged to be correct, when its root mean square error is less than ${\mu}$. $\Gamma(x)$ is the Gamma function, which could be written as

When the prediction model is initialized without prior knowledge, the confidence of the prediction sequence is expressed as a uniform distribution on $(0,1)$:

Suppose that $t={t_{0}}-d\left({1\leqslant d\leqslant 5}\right)$ and the prediction sequence at time $t$ is ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t+1}},{\mathop{t+15}}}^{{\mathop{t}}}}$. As shown in Fig. 2 b, the confidence estimation method can evaluate the effectiveness of prediction sequence at time $t$ by comparing ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t+1}},{\mathop{t}\nolimits_{0}}}^{{\mathop{t}}}}$ with ${\mathop{\textit{PR}}\nolimits_{{\mathop{t+1}},{\mathop{t}\nolimits_{0}}}}$. It is assumed that $m$ correct prediction results and $n$ incorrect prediction results are identified after comparison. The estimated confidence of the prediction sequence is defined as

Finally, the confidence is normalized as

The proposed Conformer algorithm is shown in Algorithm 1. Firstly, the enhanced Transformer-based renewable energy prediction model predicts the sequence in the next 15 days ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+1,{\mathop{t}\nolimits_{0}}+15}^{{\mathop{t}\nolimits_{0}}}}$ on the basis of the real data in the past 56 days ${\mathop{\textit{PR}}\nolimits_{{\mathop{t}\nolimits_{0}}-55,{\mathop{t}\nolimits_{0}}}}$. And the prediction results are stored to the storage pool $D_{s}$. If ${\mathop{t}\nolimits_{0}}<5$, ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+1,{\mathop{t}\nolimits_{0}}+15}^{{\mathop{t}\nolimits_{0}}}}$ is directly intercepted as ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+1,{\mathop{t}\nolimits_{0}}+10}}$. If ${t_{0}}\geqslant 5$, the past 5 days’ prediction sequences are taken out from the storage pool $D_{s}$, and the confidence of the sequences is calculated via (6) and (7). Finally, the predicted active output results from ${\mathop{t}\nolimits_{0}}+1$ to ${\mathop{t}\nolimits_{0}}+10$ in the past 5 days’ prediction sequences are multiplied by the corresponding estimated confidence respectively and then accumulated to obtain the effective prediction ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+1,{\mathop{t}\nolimits_{0}}+10}}$.

In the prediction process of Conformer, this paper combines the enhanced Transformer prediction results from different time to get the final prediction, thereby reducing the prediction error. The final prediction is delivered to RLpatching to assist the power grid dispatching and abate the impact of new energy uncertainty.

The optimization objectives consist of security, economic and environmental indicators. The security objective considers branch overflow ${\mathop{S}\nolimits_{b}^{t}}$, reactive power output overrun ${\mathop{S}\nolimits_{r}^{t}}$ and voltage overrun ${\mathop{S}\nolimits_{v}^{t}}$. The economic objective aims at minimizing the cumulative cost of the power system ${\mathop{C}\nolimits^{t}}$, which includes the operation cost and startup cost of the generators. And the environmental objective targets to maximize URRE $\mathop{R}\nolimits^{t}$. The formula of the optimization objectives is

Here, ${\mathop{N}\nolimits_{steps}}$ is the maximum number of steps in which the dispatching strategy can make the simulator run safely under the constraints. $\mathop{\omega}\nolimits_{R}$ is the weight coefficient greater than one to improve URRE. ${\mathop{n}\nolimits_{gen}}$, ${\mathop{n}\nolimits_{branch}}$, ${\mathop{n}\nolimits_{bus}}$ and ${\mathop{n}\nolimits_{new}}$ represent the number of generators, branches, busbars and renewable energy generators, respectively. ${{\mathop{u}\nolimits_{i}^{t}}}$, ${\mathop{q}\nolimits_{i}^{t}}$ and ${{\mathop{P}\nolimits_{i}^{t}}}$ are the on/off status, reactive power output and active output of generator $i$ in period $t$, respectively. ${\mathop{q}\nolimits_{i}^{\max}}$ and ${\mathop{q}\nolimits_{i}^{\min}}$ represent the maximum and minimum reactive power output of generator $i$, and ${\mathop{P}\nolimits_{i}^{\max}}$ is the maximum active output of generator $i$. ${\mathop{I}\nolimits_{j}^{t}}$ and ${\mathop{T}\nolimits_{j}}$ separately represent the current and thermal limits of branch $j$. ${\mathop{v}\nolimits_{k}^{t}}$ is the voltage of busbar $k$ in period $t$, ${\mathop{v}\nolimits_{k}^{\max}}$ and ${\mathop{v}\nolimits_{k}^{\min}}$ represent the maximum and minimum voltage of busbar $k$, respectively. ${\mathop{a}\nolimits_{i}}$, ${\mathop{b}\nolimits_{i}}$ and ${\mathop{c}\nolimits_{i}}$ are operation cost factors of generator $i$. ${\mathop{c}\nolimits_{i}^{start}}$, the factor of startup cost, is a fixed value for generator $i$. In addition, we use a normalization function ${\zeta(x)=\mathop{e}\nolimits^{x}-1}$ to limit the ranges of ${\mathop{S}\nolimits_{r}^{t}}$, ${\mathop{S}\nolimits_{v}^{t}}$ and ${-\mathop{C}\nolimits^{t}}$ to $\left({-1,0}\right]$.

In this paper, three constraint conditions are set to limit optimization.

RLpatching adopting DDPG determines the active power output of all generators for the next day according to the current power grid operation state and long-term prediction of renewable energy from Conformer. It includes ${actor}$ and ${critic}$, which are used for decision-making and scoring, respectively, as shown in Fig. 3.

In RLpatching, the input of ${actor}$ includes the current power grid observations $obs_{t}$ and the final prediction ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}}+1,{\mathop{t}}+10}}$ output by Conformer, which are marked as the state space $\mathop{s}\nolimits_{t}$. $obs_{t}$ consists of 13 selected groups of current power grid parameters to represent the operation state, including reactive power of all generators, voltage, active power and reactive power at the starts and ends of branches, load ratio and current of branches, active power and reactive power of the loads and the grid loss. ${actor}$ outputs active power of all generators for the next day $\mathop{P}^{t}$. ${critic}$ takes $\mathop{s}\nolimits_{t}$, $\mathop{P}^{t}$ as its input and outputs the ${value}$ ${Q}$.

Since the grid operation control is modeled as a time-continuous decision-making process, the reward function is defined as the optimization objectives at each time step. The reward function at time $t$ is

The ${actor}$ and ${critic}$ parameterized by ${\mathop{\theta}\nolimits^{\mu}}$, ${\mathop{\theta}\nolimits^{Q}}$ are used to represent the deterministic policy ${\mathop{P}^{t}=\mu(\mathop{s}\nolimits_{t}\left|{\mathop{\theta}\nolimits^{\mu}}\right.)}$ and the critic function ${Q(\mathop{s}\nolimits_{t},\mathop{P}^{t}\left|{\mathop{\theta}\nolimits^{Q}}\right.)}$. ${\mathop{\theta}\nolimits^{\mu}}$ and ${\mathop{\theta}\nolimits^{Q}}$ are optimized by stochastic gradient method. The loss functions of $actor$ and $critic$ are

In power grid dispatching, adjusting numerous generators would decrease the stability sharply. To deal with this problem, this paper designs a dispatching necessity evaluation mechanism to quantify the necessity of each generator and select $\mathop{n}\nolimits_{dis}$ generators to dispatch. For generator $i$, three factors are considered to evaluate its necessity, including active power adjustment value for the next day $\mathop{\Delta P}\nolimits_{i}$, utilization rate ${R_{i}}=\frac{{\mathop{P}\nolimits_{i}^{t}}}{{\mathop{P}\nolimits_{i}^{\max}}}$, and maximum load rate of all branches around it $\mathop{L}\nolimits_{i}^{max}$. $\left|{\mathop{\Delta P}\nolimits_{i}}\right|$ is normalized as ${\mathop{\left|{\mathop{\Delta P}\nolimits_{i}}\right|}\nolimits^{{}^{\prime}}}$ by the min-max method. We select $\mathop{n}\nolimits_{dis}$ generators with the highest necessity to dispatch, and the active output of other generators is consistent with that of previous time step. The necessity formula of dispatching is defined as

Equation (20) is a polynomial composed of three parts. $\mathop{\mathop{\omega}\nolimits_{P}\left|{\mathop{\Delta P}\nolimits_{i}}\right|}\nolimits^{{}^{\prime}}$ measures the adjustment value, with which the dispatching necessity is positively correlated. $({\mathop{\varphi}\nolimits_{R}}-\mathop{R}\nolimits_{i})\mathop{\Delta P}\nolimits_{i}$ is designed to control the utilization rate of generator $i$ within a reasonable range. When utilization rate of generator is less than ${\mathop{\varphi}\nolimits_{R}}$, the necessity of adjustment increases with the increase of $\mathop{\Delta P}\nolimits_{i}$. When utilization rate of generator is greater than ${\mathop{\varphi}\nolimits_{R}}$, the increase of active power of generator $i$ is restrained. $({\mathop{\varphi}\nolimits_{L}}-\mathop{L}\nolimits_{i}^{\max})\mathop{\Delta P}\nolimits_{i}$ maintains the load rate of branches around generator $i$, featuring the same characteristics as $({\mathop{\varphi}\nolimits_{R}}-\mathop{R}\nolimits_{i})\mathop{\Delta P}\nolimits_{i}$.

The decision-making and training process of the proposed Conformer-RLpatching for the hybrid power grid is shown in Algorithm 2. At the beginning of each epoch, $obs_{\mathop{t}\nolimits_{0}}$ is obtained from the initialized simulator, together with which the prediction from Conformer initializes the state space $\mathop{s}\nolimits_{{\mathop{t}\nolimits_{0}}}$. In each step of dispatching, active power output of all generators for the next step is determined by the $actor$, the dispatching necessity evaluation mechanism selects $\mathop{n}\nolimits_{dis}$ generators to dispatch, and then the simulator executes the dispatching strategy. Next, $obs_{{\mathop{t}\nolimits_{0}}+1}$ and ${\mathop{\widehat{\textit{PR}}}\nolimits_{{\mathop{t}\nolimits_{0}}+2,{\mathop{t}\nolimits_{0}}+11}}$ consists $\mathop{s}\nolimits_{{\mathop{t}\nolimits_{0}}+1}$. Afterwards $(\mathop{s}\nolimits_{\mathop{t}\nolimits_{0}},\mathop{P}^{\mathop{t}\nolimits_{0}},\mathop{r}\nolimits_{\mathop{t}\nolimits_{0}},\mathop{s}\nolimits_{{\mathop{t}\nolimits_{0}}+1})$ is stored to $D_{e}$. Finally, samples are selected from $D_{e}$ to update all networks. This epoch will end when the power flow calculation of the simulator fails to converge or the operation of the grid does not meet the constraints mentioned in Section III-A.

Conformer-RLpatching is evaluated on SG-126 power grid simulator. The simulator includes 54 generators and 117 branches, which conforms to the characteristics and operation mode of provincial grid. The simulation parameters and generator parameters are illustrated in TABLE I and TABLE II, respectively.

This subsection compares the predictive effectiveness of Conformer with that of state-of-the-art methods, including Informer [27], LSTMa [28], Prophet [29], convolutional neural network (CNN), and CNN-LSTM [26]. The data set used in the experiment is the maximum active power output of 18 renewable energy units in 106820 days, which is provided by China Electric Power Research Institute. This paper selects 90$\%$ of the data as the training set and 10$\%$ as the testing set. The root mean square error (RMSE) is used to evaluate the effect of the models:

We set up a series of comparative experiments, w.r.t variable input length and output length, to predict the long-term maximum active output of all renewable energy generators. In these experiments, historical data of the past 48 days, 56 days, 64 days, 72 days, 80 days, 88 days, and 96 days are fed into models separately, and the length of predicted values ${t_{pre}}$ is set as 15, 20, 25 and 30. We only demonstrate the crutial results in TABLE III, the complete results are exhibited in APPENDIX A.

Conformer achieves much better performance than all compared methods in the overall experiments, which demonstrates its superiority. Specifically, the average RMSE of Conformer in 12 groups of comparative experiments is 19.07$\%$ lower than that of Informer with the second-best performance, which means Conformer has less prediction error. The results show that the confidence estimation method offsetting the contingency of a single prediction result facilitates the predictive effectiveness. In addition, the prediction accuracy of Conformer is 39.65$\%$ higher than other algorithms on average.

When the input length is fixed, all methods show a consistent trend, that is, the gap between the prediction and real data widens slightly as the output length increases. For example, when the input length is 56 and the output length rises from 15 to 30, the RMSE of Conformer and CNN increases by 23.57$\%$ and 26.95$\%$, respectively.

Given the fixed output length, the prediction accuracy of all methods tends to increase with the input length rising, and reaches a peak as the input length is approximately 56. The phenomenon illustrates that more historical data endows the model with better predictive ability, while excessive input will bring more interference of invalid data to the model, thus reducing the prediction accuracy. When the output length is fixed at 20, despite the predictive error of Conformer increasing by 2.40$\%$ as the input length rises from 56 to 64, Conformer still maintains its superiority with the RMSE 19.94$\%$, 28.56$\%$ lower than that of Informer and CNN, respectively.

This paper further observes the robustness of Conformer in different conditions of variable input and output lengths compared with Informer and CNN. Short-term prediction with more historical data makes it much easier for models. Therefore, three scenarios of variable difficulty are set as shown Fig. 4. It is clear that CNN has the worst performance in all scenarios. As depicted in Fig. 4 (a), both Conformer and Informer are able to describe the correct trend of real data, specially the prediction from day 9 to day 13 output by Conformer achieves surprisingly little error. In the moderate scenario Fig. 4 (b), Conformer performs more stably than Informer, and achieves more accurate long-term prediction. Conformer significantly outperforms Informer in the hard scenario Fig. 4 (c), where Informer deviates real data as output length rises, whereas Conformer describes the correct trend of real data accurately. In summary, Conformer still achieves the best performance despite the prediction accuracy decreasing as the difficulty rises, which proves its advantageous robustness.

According to Section IV-B, the length of historical data input to Conformer-RLpatching is set as 56 with the lowest prediction error. Conformer outputs the prediction for the next 10 days in all experiments except for Conformer-RLpatching II, which only provides prediction for the next day.

In order to compare the impact of long-term and short-term renewable energy prediction on dispatching effect, this paper designs a comparative experiment between Conformer-RLpatching I and II. As shown in TABLE IV, Conformer-RLpatching I enables the simulator to run 117 more steps than Conformer-RLpatching II whilst satisfying the constrains, and the total reward of Conformer-RLpatching I is 52.352$\%$ higher.

Conformer-RLpatching I and III-VIII test the impact of different parameters in the dispatching necessity evaluation mechanism, including ${\mathop{\varphi}\nolimits_{R}}$, ${\mathop{\varphi}\nolimits_{L}}$, $\mathop{n}\nolimits_{dis}$, on active power flow dispatching of hybrid power grid. According to TABLE IV, overall, when ${\mathop{\varphi}\nolimits_{R}}$ and ${\mathop{\varphi}\nolimits_{L}}$ are both 0.8 and $\mathop{n}\nolimits_{dis}$ is 40, Conformer-RLpatching performs best, and its total reward reaches 527. When ${\mathop{\varphi}\nolimits_{R}}$ changes, the security score and renewable energy utilization rate show a downward trend, but the average cost of each step changes little. For example, when ${\mathop{\varphi}\nolimits_{R}}$ increases from 0.8 to 0.9, the utilization rate of renewable energy decreases by 4.499$\%$. When the critical value of branch current ${\mathop{\varphi}\nolimits_{L}}$ increases, the security of power grid is seriously affected. As ${\mathop{\varphi}\nolimits_{L}}$ increases from 0.8 to 0.9, the security score decreases by 7.429$\%$. As $\mathop{n}\nolimits_{dis}$ rises to 40, the stability of the power grid reaches a peak. When $\mathop{n}\nolimits_{dis}$ increases from 40 to 45, the number of operation steps and security score decrease by 2.113$\%$ and 5.843$\%$, respectively, despite the utilization rate of renewable energy increasing by 1.348$\%$.

In addition, ablation experiments are carried out to explore the separate contributions of Conformer and dispatching necessity evaluation mechanism. It can be seen that the security score decreases by 3.952$\%$ without C. This indicates that the dispatching necessity evaluation mechanism can ensure the stable operation of the power grid system. Besides, the total reward significantly reduces by 54.190$\%$ discarding A, which illustrates that Conformer can assist the grid dispatching and abate the negative impact of renewable energy fluctuations.

This paper compares Conformer-RLpatching with DDPG, distributed classification replay twin delayed deep deterministic policy gradient (DCR-TD3) [30] and proximal policy optimization (PPO) to further verify the dispatching effect of Conformer-RLpatching. The simulation results are shown in TABLE IV. The security score of Conformer-RLpatching is 25.758$\%$ and 63.319$\%$ higher than that of DDPG and DCR-TD3. The total reward is improved by 76.149$\%$ compared with DCR-TD3 having the second best performance. The renewable energy utilization rate of Conformer-RLpatching is 4.540$\%$ higher than that of other methods on average.

More intuitively, Fig. 5 depicts the average renewable energy utilization rate, the average cost, and the cumulative reward of each step of the above dispatching methods in an epoch. As shown in Fig. 5 (a), despite PPO reaching relatively high renewable energy utilization rate in the beginning, its dispatching is highly unstable with an end at only step 50. In contrast, Conformer-RLpatching, which enables superior stability and efficient renewable energy utilization rate, is noticeably the best performer on the renewable energy utilization issue, albeit at the cost of slightly high cost as presented in Fig. 5 (b). Fig. 5 (c) shows the overall performance in dispatching. The rewards obtained by PPO and DCR-TD3 improve fast initially whereas coming to an end quickly, which indicates their instability. DDPG runs relatively more steps than PPO and DCR-TD3, but it obtains the lowest rewards of them in the epoch. To sum up, Conformer-RLpatching achieves the most running steps and the highest cumulative reward, verifying its superiority among the state-of-the-art dispatching methods.

TABLE V compares the total reward of Conformer-RLpatching and the top three teams in State Grid Dispatching AI Innovation Competition. The results show that Conformer-RLpatching achieves a remarkable total reward up to 527.318 superior to 510.091 obtained by the best team.