Machine Learning architectures for price formation models with common noise

In this work, we extend the use of machine learning (ML) techniques for the numerical solution of the mean-field games (MFGs) price formation models, introduced in [8], to incorporate the common noise model from [9] (see also [10]). The goal is to determine the price $\varpi$ of a commodity with a noisy supply $Q$ traded among rational agents within a finite time horizon $T>0$, under a market-clearing condition. More precisely, we assume the supply function $Q$ satisfies the following stochastic differential equation (SDE)

where $q_{0}\in{\mathbb{R}}$ and $W$ is a one-dimensional Brownian motion acting as common noise. The coefficients $b^{S}$ and $\sigma^{S}$ satisfy the usual Lipschitz conditions for existence and uniqueness of solutions (see [7]). Because of (1), our model explains the price formation for commodities with continuous and smooth fluctuations, such as stocks, bonds, currencies, and continuously produced or consumed goods such as oil or natural gas. Additional sources of noise can be considered. For instance, sudden and discontinuous fluctuations can be modeled by adding Poisson jumps to (1).

Let $(\Omega,\mathcal{F},\mathds{F},\mathds{P})$ be a complete filtered probability space supporting $W$. Progressive measurability refers to the measurability with respect to this filtration, which we require for all stochastic processes. In this context, the MFG with common noise characterizing the price is the following.

The previous problem generalizes the one introduced in [11], which corresponds to the case $\sigma^{S}=0$. The numerical solution of (2) presents additional challenges compared to the deterministic counterpart, as the state space becomes infinite-dimensional. [10] showed that (2) is well-posed when $b^{S}$ and $\sigma^{S}$ are linear and $H$ is quadratic, obtaining semi-explicit solutions. Section II presents the derivation of (2).

In the absence of common noise, several numerical schemes have been proposed: Fourier series [17], semi-Lagrangian schemes [4], fictitious play [12], and variational methods [3]. [6] proposes an ML-based approach to solve bi-level Stackelberg problems between a principal and a mean field of agents by reformulating the problem as a single-level mean-field optimal control problem. [15] and [5] survey deep learning and reinforcement learning methods applied to MFGs and mean-field control problems. However, these methods cannot handle general forms of common noise as the state space becomes infinite-dimensional. Recent works have circumvented this issue. [2] reduces continuous-time mean field games with finitely many states and common noise to a system of forward-backward systems of (random) ordinary differential equations. [16] used rough path theory and deep learning techniques. However, the coupling in the price formation problem with common noise is given by an integral constraint in infinite dimensional spaces, which is beyond what standard methods can handle. In [1], the price formation model with common noise was converted into a convex variational problem with constraints and solved using ML, enforcing constraints by penalization. This approach, however, introduces numerical instabilities. In contrast, our method includes the balance constraint in the loss functional as a Lagrange multiplier instead of a penalization.

Our method employs two recurrent neural networks (RNN) to approximate $\varpi$ and the optimal vector field agents follow using a particle approximation and a loss function that the RNNs optimize by adversarial training. We develop a posteriori estimates to confirm the convergence of our method, which is of paramount importance when no benchmarks are available. Given $f:[0,T]\times\Omega$, we write $\|f\|=\left({\mathds{E}}\left[\|f\|_{L^{2}([0,T]}^{2}\right]\right)^{1/2}$. We introduce additional notation in Section III, where we prove the following main result:

We present our algorithm in Section IV and numerical results in Section V for the linear-quadratic setting. Nonetheless, our method can handle models outside the linear-quadratic framework. Moreover, the ML is well suited for higher-dimensional state spaces, where, for instance, several commodities are priced simultaneously. Section VI contains concluding remarks and sketches future research directions.

Price formation is a critical aspect of economic systems. One example is load-adaptive pricing in smart grids, which motivates consumers to adjust their energy consumption based on changes in electricity prices. MFGs provide a mathematical framework for studying complex interactions between multiple agents, including buyers and producers in a market. Here, we revisit the underlying optimization problem in Problem 1.

A representative player with an initial quantity $x_{0}\in{\mathbb{R}}$ at time $t=0$ selects a progressively measurable trading rate $v:[0,T]\times\Omega\to{\mathbb{R}}$ to minimize the cost functional mapping $v$ to

where $X$ solves

with $x_{0}\sim m_{0}$. The Hamiltonian $H$ in (2) is the Legendre transform of the Lagrangian $L$ in (3):

Here, we assume that $v\mapsto L(x,v)$ is uniformly convex for all $x\in{\mathbb{R}}$. Moreover, we assume that $H$ satisfies the assumptions of Theorem 1, which guarantees the Lagrangian satisfies the convexity requirement. Considering the value function

we obtain the first equation in (2). The distribution starts with $m_{0}\in{\mathcal{P}}({\mathbb{R}})$ and evolves according to the stochastic flow controlled by $v^{*}(t,x)=-H_{p}(x,\varpi(t)+u_{x}(t,x))$, as described by the second equation in (2), while the third equation imposes a market-clearing condition. [1] discusses further details. In our method, we approximate $\varpi$, which allows the decoupling of the equations in (2).

The particle approximation involves a finite population of $N$ players with independent, identically distributed initial positions $x_{0}^{n}\in{\mathbb{R}}$, $n=1,\ldots,N$, with distribution $m_{0}$. Each player selects $v^{n}:[0,T]\times\mathbb{R}\times\Omega\to\mathbb{R}$, $1\leqslant n\leqslant N$, determining its trajectory $X^{n}$ according to (4) and aiming at minimizing the functional mapping $v^{n}$ to

The existence of ${v^{*}}^{n}$ minimizing (II) for $1\leqslant n\leqslant N$ corresponds to the existence of ${\bf v}^{*}=({v^{*}}^{1},\ldots,{v^{*}}^{N})$ minimizing the functional

subject to the market-clearing constraint

We rely on the existence and uniqueness result for the $N$-player price formation model, presented in [10], which determines the price ${\varpi^{*}}^{N}:[0,T]\times\Omega\to{\mathbb{R}}$ in (II) through the Lagrange multiplier associated with the market-clearing constraint. Our goal is to extend the ML algorithm introduced in [8] to cover the case of common noise, providing a solution to the price formation problem in random environments. Relying on the particle approximation of the model to approximate the price solving (2), we approximate stationary points of the functional mapping $({\bf v},\varpi^{N})$ to

by minimizing w.r.t. ${\bf v}$ and maximizing w.r.t. $\varpi^{N}$. The approximation is done in the ML framework, and we guarantee its accuracy using a posteriori estimates of the $N$-player model, which we present next.

In this section, we use optimality conditions for the $N$-player game to obtain a posteriori estimates to verify our approximation’s convergence. We extend the proof presented in [8] to the common noise setting with minor modifications.

The optimality conditions for (II) give rise to a Hamiltonian system comprising the following backward-forward stochastic differential equation

for $1\leqslant n\leqslant N$, where $H$ is given by (5). Notice that $Z^{n}$ is part of the unknowns. Moreover, ${\bf v}^{*}$ and ${\varpi^{*}}^{N}$ solving the $N$-player price formation problem define a solution of (9) by

for $1\leqslant n\leqslant N$, defining a saddle point of (II) that satisfies the market-clearing constraint (7). Let $\tilde{P}^{n}$, $\tilde{Z}^{n}$, $\tilde{X}^{n}$, and $\tilde{\varpi}^{N}$ satisfy

where $\epsilon^{n},\epsilon_{B}:[0,T]\times\Omega\to{\mathbb{R}}$, $\epsilon_{T}^{n}:\Omega\to{\mathbb{R}}$, for $1\leqslant n\leqslant N$. We write ${\bf X}=(X^{1},\ldots,X^{N})$, and, analogously, for all $n$-indexed stochastic processes. We denote $\epsilon_{H}=(\epsilon^{1},\ldots,\epsilon^{N},\epsilon_{T}^{1},\ldots,\epsilon_{T}^{N})$ and $\mathds{1}=(1,\ldots,1)\in{\mathbb{R}}^{N}$.

This section details the RNN architectures we use to estimate $v^{*}$ and $\varpi$. Section V presents some numerical experiments. RNNs, commonly used in natural language processing, generate outputs that sequentially depend on inputs. This architecture has a cell that iterates through input sequences and has a hidden state tracking historical dependencies; see [14] for details. RNNs have also been used in the context of control problems with delay in [13], but here our motivation comes from the impact of the common noise on the mean-field term.

In our architecture, the RNN takes as inputs an ordered sequence, such as a discrete realization of the supply $\mathrm{Q}=\left(\mathrm{Q}^{\langle 0\rangle},\ldots,\mathrm{Q}^{\langle K\rangle}\right)$. The RNN features a hidden state $\mathrm{h}$, initialized as zero, that captures the temporal dependence. Inside the RNN, a weight matrix $\mathrm{W}^{[l]}$ and a bias vector $\mathrm{b}^{[l]}$ determine layer $l$, where $1\leqslant l\leqslant L$. Their dimensions depend on the number of neurons $n^{[l]}$ per layer. The activation function of layer $j$ is denoted by $\sigma^{[l]}$. The cell parameters (weight matrices and bias vectors) are denoted by $\Theta$.

We use two RNNs for approximating the control variable $v$ and the price $\varpi$. As usual in the ML framework, a trade-off must be made between computational cost and accuracy. Deep-RNN employs several layers and neurons in their cell. After multiple numerical experiments, we select $L=5$ layers, with $n^{[1]}=16$, $n^{[2]},n^{[3]},n^{[4]}=32$, and $n^{[5]}=1$ for the RNN approximating $\mathrm{v}^{\langle k\rangle}$, and $n^{[1]},n^{[2]},n^{[3]},n^{[4]}=16$, and $n^{[5]}=1$ for the RNN approximating $\varpi$. As a common practice for RNN, the activation functions are hyperbolic tangent for the first layer, which computes the hidden state, and sigmoid for layers two to four. The last layer has an activation function equal to the identity, representing any real number as an output. Although we do not address it, an interesting research question is how sensitive the results are to the choice of parameters of the RNN. Moreover, a comparison regarding the accuracy and computational efficiency against other methods, such as forward-backward SDEs methods, can be formulated based on the adaptability of those methods to the price formation MFG problem with common noise.

We denote by $\Delta t=T/K$ the time-step size and discretize (4) according to

for $k=0,\ldots,K$, where $\Theta_{v}$ is the parameter of the RNN approximating $v$. The second RNN, with parameter $\Theta_{\varpi}$, computes $\varpi^{\langle k\rangle}$ for $k=0,\ldots,K$. More precisely, the inputs and outputs of the two RNNs are as follows. For the RNN computing $\varpi(\Theta_{\varpi})$, the input consists of a supply realization and the time; that is, $\left((Q^{\langle k\rangle})_{k=0}^{K},\quad(t^{\langle k\rangle})_{k=0}^{K}\right)$. The output is $(\varpi^{\langle k\rangle})_{k=0}^{K}$. For the RNN computing $v(\Theta_{v})$, the input consists of the time, the state variables (which the RNN updates according to (16) as it iterates in the temporal direction), and the current price approximation; that is, $\left((t^{\langle k\rangle})_{k=0}^{K},(X^{\langle k\rangle})_{k=0}^{K},(\varpi^{\langle k\rangle})_{k=0}^{K},\right)$. The output is $(\mathrm{v}^{\langle k\rangle})_{k=0}^{K}$. Because we consider a population of $N$ agents, we add the superscript $(n)$ to denote the position and control sequence of the agent being considered; that is, $(X^{(n)\langle k\rangle})_{k=0}^{K}$, and $(\mathrm{v}^{(n)\langle k\rangle})_{k=0}^{K}$, for $1\leqslant n\leqslant N$.