Large Language Model Assisted Optimal Bidding of BESS in FCAS Market: An AI-agent based Approach

In Australia’s NEM, the Australian Energy Market Operator (AEMO) ensures that the system frequency remains within the normal operation band by purchasing sufficient capacity from FCAS providers’ bids to match the FCAS demand [20]. FCAS is divided into regulation and contingency services: regulation FCAS is dispatched to correct minor frequency deviations while contingency FCAS restores larger frequency deviations when the frequency exceeds the standard band [1]. Depending on the directions of frequency correction, they can be further classified as raised services and lower services. The specific market operation has the following stages.

At each trading interval $t$, AEMO will deploy sufficient capacity from $N$ bids to meet the demand $d_{t}$ in the order of the cost. $bc_{t,n,k}$ and $bp_{t,n,k}$ denote the $k$-th bidding capacity and bidding price respectively of the $n$-th bidder at time $t$. Then AEMO will determine the enabled capacity $ec_{t,n}$ and the clearing price $cp_{t}$ which will be settled to the $n$-th bidder. Let $m\in\{lr,rr,lc,rc\}$ denote the lower regulation, raise regulation, lower contingency, and raise contingency FCAS markets respectively. The market clearing model is as follows:

where the optimization objective (1a) is to minimize the costs across all FCAS markets. (1b)-(1c) aim to determine the bidding capacity $BC_{t,n}^{m}$ of each bidder by selecting at most one bidding band from ten bands. (1d) means that AEMO can flexibly enable the capacity $ec_{t,n}^{m}$ from the selected bidding capacity $BC_{t,n}^{m}$. Similar to (1b), (1e) determines the corresponding bidding price $BP_{t,n}^{m}$ at the selected band. (1f)-(1h) aims to use the big M method to determine the maximum enabled bidding price (i.e. the clearing price $cp_{t}^{m}$), where M is a very large value. (1i) ensures the overall enabled FCAS capacity plus the overall market supply variation $s_{t}^{m}$ is equal to the FCAS demand $d_{t}^{m}$. (1j) ensures the joint enabled capacity of each bidder in regulation lower, contingency lower FCAS markets plus the charging capacity in energy market $Pc_{t,n}$ never exceed its maximum charging capacity $Pc_{t,n}^{max}$. Similarly, (1k) ensures the joint enabled capacity of each bidder in regulation raise, contingency raise FCAS markets plus the discharging capacity in energy market $Pd_{t,n}$ never exceed its maximum discharging capacity $Pd_{t,n}^{max}$. As shown in (1l), $u_{t,n,k}$ and $v_{t,n}$ are binaries.

Before using the above models, it is necessary to present the model that calculates the overall market supply variation mentioned in the previous models, i.e. $s_{t}^{m}$. The clearing price is determined by both the FCAS supply and demand, taking a real-world scenario in the FCAS market as an example, bidders typically put more capacity into arbitrage during peak energy prices, and hence the clearing prices in FCAS markets will be higher to purchase capacity that can meet the demand. However, the data on clearing prices and demands are given by AEMO whereas the data on supply is incomplete. Hence, it is hard to describe the above scenarios caused by the market supply variation. To better describe the relationship between price and supply-demand balance under actual scenarios, we build a model to estimate the overall market supply variation $s_{t}^{m}$ under given clearing prices $cp_{t}^{m}$ and demands $d_{t}^{m}$.

where the optimization objective (4a) is to find the supply variation $s_{t}^{m}$ at the supply-demand balancing point. (4b)-(4e) can find the available capacity in market $m$ under the given price $cp_{t}$. Since $cp_{t}$ and $d_{t}$ are given, the only unknown variable $s_{t}^{m}$ can be calculated by (4f)-(4h). $s_{t}^{m}$ can be both positive and negative, which can be divided into a positive component $s_{t}^{+,m}$ and a negative component $-s_{t}^{-,m}$.

As shown in Fig. 2, we formulate the problem as a bi-level game model, where the upper-level model is the market clearing model as a leader, and the lower-level model is the BESS bidding model as a follower. First, the unknown parameter $s_{t}^{m}$ in the problem can be obtained based on the predicted clearing price and demand by solving the supply estimation model (4a). Then, in the pre-dispatch stage, the bidding price and capacity of the BESS are obtained by solving the day-ahead bidding model (2a). In the real-time stage, the bidding price and capacity of the BESS are introduced into model (1a) to obtain an initial clearing price. Then, the clearing price is introduced into model (3a) to obtain the bidding capacity. In the two-level game, model (1a) and model (3a) iteratively update their strategies until Nash equilibrium or exceed the iteration limit. Note that all the above models take the predicted market information as inputs. Finally, in the actual dispatch stage, model (4a) is solved based on the actual clearing price and demand to obtain the parameter $s_{t}^{m}$, and then the bidding strategy from model (3a) is introduced into model (1a) to obtain the actual clearing price and enabled capacity, as well as the final actual profit of the BESS.

To formulate the proposed problem as a DRL problem, a Markov Decision Process (MDP) including a state space $\mathcal{S}$, an action space $\mathcal{A}$, a transition probability function $\mathcal{P}$, and a reward function $\mathcal{R}$ is necessary to be defined in the form of a DRL environment for the interaction with the DRL agent.

• •

  State space: A state space contains a set of states $s_{t}\in\mathcal{S}$ that the agent can observe from the environment. At time $t$, the SoC of BESS $SoC_{t}$, the predicted energy price $p_{t}^{e}$, FCAS prices $p_{t}^{m}$ and demands $d_{t}^{m}$ in market $m$ are defined as states.

• •

  Action space: An action space contains a set of actions $a_{t}\in\mathcal{A}$ which is the bidding strategy of the agent. At time $t$, the bidding capacities $bc_{t,k}^{lr},bc_{t,k}^{rr},bc_{t,k}^{lc},bc_{t,k}^{rc},k=1,...,9$ in the FCAS markets as well as the output $Pc_{t}$ and $Pd_{t}$ in the energy market are defined as actions.

• •

  Transition probability function: $\mathcal{P}(s_{t},a_{t},s_{t+1})=\Pr(s_{t+1}\mid s_{t},a_{t})$. The state $s_{t+1}$ at $t+1$ is uncertain under given observation $s_{t}$ and bidding strategy $a_{t}$ at $t$.

• •

  Reward function: $\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. When the agent takes an action $a_{t}$ and transforms $s_{t}$ to $s_{t+1}$, it receives a reward $r_{t}(s_{t},a_{t},s_{t+1})\in\mathcal{R}$. We define the profits of the BESS including the revenues in the joint energy and FCAS markets and the degradation costs as the rewards. Similar to the objective function of model (3a), the basic reward function can be written as:

  $$\begin{split}r_{t+1}=\sum_{m}(cp_{t}^{m}ec_{t,i}^{m})+p_{t+1}^{e}(SoC_{t+1}-SoC_{t})\\ -\alpha|SoC_{t+1}-SoC_{t}|\end{split}$$

  (5)

  where $cp_{t}^{m}$ and $ec_{t,i}^{m}$ are the solutions of the market clearing model (1a), which means that there are interactions between the agent and the market.

Unlike the instantaneous rewards in FCAS, arbitrage rewards tend to be sparse and delayed. To encourage the BESS to charge before discharging and receiving rewards, the potential-based reward shaping (PBRS) [23] is employed to assist the agent in learning strategies for joint markets. The reward function (5) is modified by adding an extra reward based on prior knowledge:

where potential function $\Phi(s_{t})$ measures the gap between charged energy and energy threshold $\underline{E}$. $[\bullet]^{-}$ is the projector of $\min(\bullet,0)$ to avoid overcharging. In general, the dense artificial reward signals drive the agent to reach the sub-goal of charging at least a certain amount of energy in each episode.

In order to ensure that the bidding strategy of the agent complies with market standards and rules, any action out of specification is limited, for $k=1,...,9$ we have:

where the maximum function is used to ensure the bidding capacities are monotonically increasing in each FCAS market. The minimum function is used to ensure the joint capacities in energy and FCAS markets are within the limitations of the maximum charging or discharging capacity. In addition, when $SoC_{t}$ is larger than $SoC_{max}$, $Pc_{t},bc_{t,k}^{lr},bc_{t,k}^{lc},k=1,...,10$ are limited to 0, and when $SoC_{t}$ is smaller than $SoC_{min}$, $Pd_{t},bc_{t,k}^{rr},bc_{t,k}^{rc},k=1,...,10$ are limited to 0, which ensures $SoC_{t}$ is always within the limitations of the SoC. Once the MDP is defined, the DRL agent will interact with the environment to obtain experience samples and learn strategies to maximize the cumulative rewards of the scenarios. After offline training, the trained model will be applied to the online bidding system as the decision engine.

The epistemic uncertainty leads to a declined bidding performance of the DRL agent in unfamiliar scenarios outside the training set. Thus, we propose an LLMs-assisted AI-agent interactive decision-making framework to decompose the complicated bidding task into more manageable sub-tasks and deploy multiple LLMs-based agents to address the components in an orderly modular pattern. Compared to the DRL model trained on historical data, LLMs can analyze the latest market conditions in real time, enhancing the strategy timeliness. To reduce profit loss in unfamiliar scenarios, LLMs can adjust inferior strategies by conditional hybrid decisions and incrementally integrate new knowledge into DRL through MDP reshaping, thereby improving the strategy reliability. Additionally, unlike the ”black box” nature of DRL networks, LLMs can act as interpreters to provide explanations to users, thereby increasing the interpretability. An external evaluation and self-reflection mechanism are designed in the “DRL-LLMs” loop to mitigate the hallucination problem in LLMs.

In addition to epistemic uncertainty, aleatoric uncertainty also impacts the profitability of the BESS. Due to the prediction errors, the observation of DRL agent exists certain noises in our problem instead of fixed in some traditional problems. To reduce the risk of uncertain MDP and improve the algorithm performance, the CVaR-DRL algorithm is proposed, where the proximal policy optimization (PPO) architecture is employed as the basic decision-making engine, CVaR [24] is introduced as risk metrics and the maximum entropy principle [25] is applied for model robustness enhancement.

A standard DRL aims to find a policy $\pi_{\theta}$ parameterized by $\theta$ that maximizes the expected cumulative reward:

where $\gamma\in(0,1)$ is a discount factor. Deterministic policy often struggles to cope with uncertainty in MDPs and tends to fall into local optimum, hence, we first combine the concept of entropy with DRL to introduce randomness into the policy so that DRL agents can fully explore the environment. Mathematically, the objective function can be modified as follows:

where the term $\mathcal{H}_{\pi_{\theta}}$ maximizes the policy entropy under coefficient $\delta$ to encourage the agent to adopt the action distributions with high entropy and improve the generalization capabilities of policy under the uncertainty.

In addition, we introduce CVaR to quantify the expected loss under uncertainty in the worst $1-\alpha$ scenarios:

where $\alpha\in(0,1)$ is the confidence level. $[\bullet]^{+}$ denotes the projector of $\max(\bullet,0)$. $\mu$ denotes the threshold in the loss distribution. $R(\pi_{\theta})$ is the trajectory return under policy $\pi_{\theta}$. Based on the optimization method in [26], (15) can be constrained by CVaR to avoid profit loss in the worst scenarios. The rewritten objective function is as follows:

where $\beta$ is the reward tolerance. Then the Lagrangian relaxation is employed to unconstrain the problem:

where $\lambda$ is the Lagrange multiplier. Based on the PPO architecture, the objective function (17) can be optimized by the following gradients (18)-(20) in terms of $\lambda$, $\theta$ and $\mu$:

where $\omega_{\pi_{\theta}}$ is the scenario under policy $\pi_{\theta}$. $\textbf{1}[\bullet]$ is the indicator function. $\hat{A}^{\pi_{\theta_{k}}}$ is the estimated advantage function, $\frac{\pi_{\theta}(a\mid s)}{\pi_{\theta_{k}}(a\mid s)}$ is the importance sampling ratio, $\epsilon$ is the clip parameter and the clip function $\operatorname{clip}(\cdot)$ is used to remove the overestimation and underestimation by upper limit $1+\epsilon$ and lower limit $1-\epsilon$. Typically, the advantage function is estimated by the Generalized Advantage Estimator (GAE):

where $\nu$ is the GAE parameter and V(s) is the value function.

In summary, Algorithm 1 presents the implementation and procedure of the LLMs-assisted AI-agent CVaR-DRL.

The role of the market clearing model is to acquire sufficient capacities from various bidders to satisfy the market demand and establish clearing prices. Fig. 3 illustrates the clearing results for all participants in the market clearing model when the BESS bids 10MW, 30MW, and 50MW respectively. When the BESS bids 10 MW, the market clearing model has to purchase additional capacities from other bidders to ensure the supply-demand balance, resulting in higher clearing prices. As the bidding capacity of the BESS increases to 30MW, fewer capacities are needed from other bidders, leading to a decrease in clearing prices. As the BESS bids 50MW, the clearing prices are significantly lower. Thus, as a price maker, the BESS’s bidding capacities influence the market clearing prices.

To assess the profitability of the mathematical and DRL models, we evaluate their benefit performance in the joint energy and FCAS markets. Fig. 4 displays the bidding results of the mathematical model, wherein the BESS bids higher capacities during price spikes in the contingency markets, while mainly allocating capacities to the regulation markets under normal conditions. Fig. 5 shows the bidding results of the DRL model, which tends to distribute its capacities more uniformly across both the contingency and regulation markets.

Fig. 6 provides statistical data on the clearing prices under both strategies, which aggregates the maximum, mean, and minimum values of each FCAS market into a 24-hour profile, along with their averaged values. Generally, the market trends show that raised markets typically have higher prices than lower markets, and regulation markets are priced above contingency markets. Nevertheless, the contingency markets exhibit greater price volatility, offering rare but large profit opportunities. In both bidding strategies, the models generally allocate more capacities to the regulation market than to the contingency markets to secure stable revenues. However, notable differences exist in handling contingency capacities: the mathematical model tends to shift regulation capacities to the contingency markets after predicting the price spikes, whereas the DRL model consistently bids a portion of capacities into the contingency market to pursue potential profit opportunities. Additionally, the clearing results are different between the models, where the peak prices achieved by the DRL model in both contingency raise and lower exceed $1,000/MW, significantly surpassing those of the mathematical model.

Table II provides a summary of the profits of the BESS under the strategies of the DRL model and the mathematical model. In the energy market, the arbitrage profits of the BESS utilizing the DRL model show an increase of 7.69% over those achieved by the mathematical model. In the regulation markets, profits from the DRL model both lower and raise regulation are increased by 62.23% and 21.90%, respectively. Additionally, the DRL model enhances profits in the contingency markets by 8.77% and 7.67% and achieves a 5.63% reduction in degradation costs. Overall, the BESS under the DRL model generates $37,163 more in profit than the mathematical model, which demonstrates the superior performance of the DRL model in maximizing bidding benefits.

To assess the role of risk management in bidding strategies, we analyze the detailed distribution of bidding bands of each market for both the CVaR-DRL and the original DRL algorithms. Fig. 7 presents the box plot representation of the BESS bidding distribution for the CVaR-DRL model. The bidding bands are monotonically increasing and range from band 1 to band 10. Band1 represents the maximum capacity that can be purchased at the lowest clearing price, whereas band10 corresponds to the maximum capacity at the highest price. A higher concentration of capacity in the lower bands suggests that the BESS prioritizes having its bids accepted and disregards the risk of lower clearing prices and profits. Conversely, a concentration in the higher bands indicates a pursuit of higher clearing prices, with less consideration of the risk of bid failure. In terms of median, bidding capacities tend to be low in band1 across all markets, gradually increasing up to approximately band3, which illustrates the BESS attempts to elevate clearing prices while maintaining stable revenue. Beyond band 3, the lower extremes of capacity bids stabilize above 10MW, with the upper extremes continuing to rise slightly, indicating that in some scenarios, the BESS might incrementally increase bids to between 30MW and 35MW to further elevate the clearing prices. The outliers highlight the capacity distribution in rare cases.

Fig. 8 shows the case of the original DRL model. Compared to the proposed algorithm, the original DRL has a wider distribution both in terms of band dimension and capacity dimension. Specifically, the median on regulation capacity ceases to increase at around band 8, with the lower extreme dropping below 10MW and the upper extreme reaching up to 40MW. A similar pattern is observed in the contingency capacity, where the outliers occur significantly. This wider distribution suggests that the original DRL’s strategies are more aggressive, often disregarding the risks associated with bidding failure and potential profit loss. In contrast, the CVaR-DRL algorithm adopts a more conservative strategy by integrating the CVaR, which avoids the heavy profit loss in the worst bidding failure scenario. In terms of bidding profits, the CVaR-DRL in the same testing scenarios shows an increase of $7,879 and $1,240 in the regulation and contingency markets respectively, reflecting the profit advantages of risk management in bidding strategies.

The offset of FCAS market patterns over time may cause the DRL agent’s original strategy to no longer adapt well to new scenarios and lead to a decline in profitability. Thus, LLMs aim to detect the scenario changes in market patterns online for better timeliness, adjust strategies within a safe profit margin for better reliability, and provide explanations to users for better interpretability.

Fig. 9 compares the bidding results under the LLMs-assisted strategy and the DRL original strategy in the new scenario outside the training set. Typically, in the price axis, the volatility of contingency market clearing prices is significantly higher than that of the regulation market because the demand is often more urgent. However, in this scenario, the regulation market experienced greater volatility. Agent-1 detected the profit opportunities arising from this abnormal market condition and hence in the capacity axis, Agent-2 partially shifted capacity from the contingency markets to regulation markets. In contrast, the original strategy of reserving more contingency capacity based on historical experience would result in profit losses in this scenario, indicating that the LLMs’ bidding strategy is more reliable in terms of profitability in such uncommon scenarios.

In the time axis, the capacity shifting occurred only during the first half of the scenario. For the rest of the period, the regulation prices had returned to common market conditions, thus the original strategy remained essentially unchanged to avoid over-reliance on LLMs’ decisions and ensure a safe profit margin considering the possibility of hallucinations. This reflects the ability of LLMs in terms of timeliness to monitor dynamic changes in market conditions online.

To better demonstrate the workflow of LLMs, we partially excerpt the prompt and answers during interaction with LLMs as shown in Fig. 10. In this process, multiple agents iteratively update their strategies through interactive collaboration based on feedback and affect the interaction between the DRL agent and the environment, thus assisting the DRL agent in making better bidding strategies in new scenarios. In addition, LLMs provide the motivation and rationale for each agent’s decision, which enhances the interpretability of the strategies.