Linear Quadratic Mean-Field Games with Communication Constraints

In distributed real-world applications, like networked control systems [1], ecosystem monitoring [2], and energy harvesting [3], we rarely have the luxury of pure persistent communication. Hence, in this work, we study multi-agent systems under a constrained communication structure. The communication constraints may appear in the form of limited sensor energy levels [3], noisy transmission medium [4, 5], limits on the communication frequency [6, 7] or some combination thereof, as we investigate in this work. Additionally, scalability becomes a huge challenge with increasing number of agents in a multi-agent system.

In this paper, we consider a discrete-time multi-agent game problem. Each agent is coupled with other agents through its cost function, which incentivizes the agent to form consensus with other players. In addition to a plant and a controller, the agent’s control system (See Figure 1) also consists of a scheduler and an Additive White Gaussian Noise (AWGN) channel. The scheduler controls the flow of information using a fixed scheduling policy. The communication through the AWGN channel is regulated by an encoder/decoder pair which constitutes a predictive encoder [8] to encode sequential data and a minimum mean-square estimation (MMSE) decoder to produce the best estimate of the plant state.

A key difficulty in multi-agent systems is that of scalability. To alleviate this challenge, a mean-field game (MFG) framework was proposed in [12, 13] by Huang, Malhamé and Caines and simultaneously in [14], by Lasry and Lions. The essential idea in the MFG framework is that as the number of agents goes to infinity, agents become indistinguishable and the effect of individual deviation becomes negligible (that is, the effect of strategic interaction disappears). This leads to an aggregation effect, which can be modelled by an exogenous mean-field (MF) term. Consequently, the game problem reduces to a stochastic optimal control problem for a representative agent along with a consistency condition.

Linear Quadratic MFGs (LQ-MFGs), which combine linear agent dynamics with a quadratic cost function, serve as a significant benchmark in the study of MFGs. Recent works on LQ-MFGs [15, 16] in the discrete-time setting are free of communication constraints or consider partially observed dynamics involving packet drop-outs [17], thereby making the underlying communication link unreliable. Furthermore, Secure MFGs [18] capture the setting where the agents deliberately obfuscate their state information with the goal of subverting an eavesdropping adversary. In these works, however, communication occurs at every time instance, in contrast to our setting here, where the communication is intermittent and the channel adds noise to the incoming signal.

The paper is organized as follows. Following this introduction, Section II introduces the finite-agent game formulation of the multi-agent system and the underlying information structures of each of the its entities (see Fig. 1). In Section III, we formulate the LQ-MFG problem, characterize its MFE and demonstrate the $\epsilon$-Nash property of the MFE. In Section IV, we provide simulations to analyze the performance of the MFE and conclude the paper in Section V sharing some highlights.

Notations: Let $X_{k}^{i}$ denote the $i^{th}$ agent’s state at time instance $k$ and $X_{k:k^{\prime}}^{i}$ the $i^{th}$ agent’s state history from instant $k$ to $k^{\prime}$, i.e., $X_{k:k^{\prime}}^{i}=(X_{k}^{i},\cdots,X_{k^{\prime}}^{i})$. Let the set of non-negative integers and real numbers be denoted by $\mathbb{Z}^{+}$ and $\mathbb{R}^{+}$, respectively. The transpose of matrix $A$ is denoted by $A^{\prime}$ and trace of a square matrix $M$ by $Tr\{M\}$. For some vector $z$ and positive semi-definite matrix $S$, let $\|z\|^{2}_{S}=z^{\prime}Sz$. Unless stated otherwise, let $\|\cdot\|$ denote the 2-norm.

Consider an $N$-player game on infinite time horizon. Each agent’s dynamics evolves according to a linear discrete-time controlled stochastic process as

where $X_{k}^{i}\in\mathbb{R}^{n}$ and $U_{k}^{i}\in\mathbb{R}^{m}$ are the state process and the control input, respectively, for the $i^{th}$ agent. $W_{k}^{i}\in\mathbb{R}^{n}$ is an i.i.d. Gaussian process with zero mean and finite covariance $\Sigma_{w}$. The initial state $X_{0}^{i}$ has mean $\nu_{\phi_{i},0}$ and covariance $\Sigma_{x}$, and is assumed to be statistically independent of $W_{k}^{i}$, $\forall k\in\mathbb{Z}^{+}$. All covariance matrices are assumed to be positive definite. $A(\phi_{i})$ and $B(\phi_{i})$ are constant matrices with appropriate dimensions. $\phi_{i}$ denotes the type of the $i^{th}$ agent drawn from a finite set $\Phi:=\{\phi_{1},\cdots,\phi_{m}\}$ and is chosen according to the empirical distribution

where $\mathbb{I}_{\{\cdot\}}$ is the indicator function. It is further assumed that $\lim\limits_{N\rightarrow\infty}{F_{N}(\phi)}=F(\phi)$ weakly, for some probability distribution $F(\phi)$ over the support of $\Phi$, with corresponding probability mass functions $P_{N}(\phi)$ and $P(\phi)$, respectively.

To complete the problem formulation, we define the information structure on each of the blocks in Fig. 1. Such information structures are standard and appear in applications like industrial and process control [1] and wireless sensor networks [6].

First, we define a transmission time as an instant when information is sent over the channel. Let the history of transmission times up to the current instant $k$ ($k>0$) be denoted by the set $\mathcal{K}(k):=\{l\mid l\leq k,\gamma_{l}=1\}$, where $\gamma_{l}$ denotes the transmission instant as formalized in the next paragraph. By convention, we take $\mathcal{K}(0)=\{0\}$.

The information states, information spaces, and the input-output maps defining the scheduler, encoder, decoder and controller are as defined in Table I. The scheduler has access to the history of plant states and the decoded outputs, based on which, it decides the transmission times of plant state through the channel. The decision whether to transmit or not is taken based on an (innovations based) threshold scheduling policy (of the form $\delta_{k}^{\prime}S\delta_{k}\geq\alpha$, where $\delta_{k}$ is the error between plant and decoder output at instant $k$, to be defined later, $S>0$ is a user-defined constant positive definite matrix and $\alpha>0$ is the threshold parameter). Note that here, by innovations-based process, we mean, given a process $\{z_{k}\}$, the innovation process $\{\tilde{z}_{k}\}$ contains new information not carried in the sequences $z_{k-1},z_{k-2},\cdots$ [20, Section 5.3]. We define $\gamma_{k}:=\xi_{k}^{i}\left(I^{i,sc}_{k}\right)$, where $\gamma_{k}=1$ signifies that $k$ is a transmission instant and $\gamma_{k}=0$ signifies no transmissions ($\varphi$).

Next, the encoder transmits encoded state information ($c_{k}^{i}$) at transmission times over the channel. The signal $\bar{c}_{k}^{i}$ in Figure 1 is given as:

The encoder is assumed to have full knowledge of the system as in Table I, which leads to better control performance compared to structures where only partial information is available [4] (cf. [4] for a detailed study on other encoder information structures and how they may be realized). Such a situation emerges when the encoder and the scheduler are collocated with the plant, and hence, can observe both its state as well as the control actions applied to it [4]. In addition, we assume that the encoder is predictive, i.e., it transmits over the channel, functions of the true state minus the decoder output at the same instant. Such encoders are used in practice while encoding sequential data [8].

It is imperative to note here that the above assumptions on the encoder and the scheduler information structures, as we prove in the next section, will entail no dual-effect of control, and hence, lead to a simple controller design. The dual effect of control refers to the dual role of the controller in the evolution of the system dynamics and to probe the scheduler and new measurements to reduce its uncertainty on the system state [10]. Finally, we note, in the table, $c_{\mathcal{K}(k)}^{i}:=\{c_{l}^{i}\mid l\leq k,\gamma_{l}=1\}$ and $d_{\mathcal{K}(k)}^{i}:=\{d_{l}^{i}\mid l\leq k,\gamma_{l}=1\}$.

The encoded signal is sent over an AWGN channel, which is analog, memoryless and is modeled as:

where $v_{l}^{i}$ is an i.i.d. zero mean Gaussian process with finite positive-definite covariance $\Sigma_{v}$ and represents the channel noise. The input and output alphabets of the channel lie in $\mathbb{R}^{n}$. The signal $\bar{d}_{k}^{i}$ in Figure 1 is given as:

Finally, the controller calculates control actions by minimizing an infinite-horizon average cost function

where $Q(\phi_{i})\geq 0$, $R(\phi_{i})>0$ and the parameter $\phi_{i}\in\Phi$ determines the tuple $(A(\phi),B(\phi),Q(\phi),R(\phi))$ for each agent. Further, $\pi^{-i}:=(\pi^{1},\cdots,\pi^{i-1},\pi^{i+1},\cdots\pi^{N})$, where $\pi^{i}_{k}$ is as defined in Table I. The control law for agent $i$ over the sequence of deterministic control policies $\pi^{i}:=(\pi_{0}^{i},~{}\pi_{1}^{i},\cdots,)\in\bar{\mathcal{M}}_{i}$ and $\bar{\mathcal{M}}_{i}$ is the space of admissible decentralized control laws. The coupling between agents enters via the consensus term $\frac{1}{N}\sum_{j=1}^{N}{X_{k}^{j}}$ in the objective. Further, the cost incorporates a soft constraint on the control actions alongside penalizing state deviations from the consensus term, which each agent aims to track. Finally, the expectation in (4) is taken with respect to noise statistics and the initial state distribution.

Now, that the problem description is complete, the aim is to design a decentralized control policy for each agent in (1) minimizing its local objective (4), which is done using the MFG framework in the next section.

In this section, we solve the problem for the $N$-player system (1) with objective (4) by considering the limiting case as $N\rightarrow\infty$. In this setting, the consensus term in (4) can be approximated by a known deterministic sequence (also termed the mean-field trajectory) following the Nash Certainty Equivalence Principle [12]. This reduces the problem to a tracking control problem and a consistency condition as shown later. First, we obtain the solution (to a fully observed tracking problem constructed from a partially observed one) for this infinite agent (mean-field) system. This solution, called the MFE, consists of computing an equilibrium control policy and the equilibrium MF trajectory. Finally, we demonstrate its $\epsilon$–Nash property.