Semantic Communication in Multi-team Dynamic Games: A Mean Field Perspective

Games among teams have been an attractive topic of interest in the recent past. These systems consist of cooperating decision-makers (within a team) who aim to collectively optimize the shared team objective, while being affected by the actions of other competing teams. Applications of the same include multi-player perimeter defense games [1], electricity markets [2], oligopolies [3], and networked robotics [4], to quote a few.

In this paper, we address the problem of finding equilibrium policies for multiple dynamic teams competing to access a shared wireless channel. A prototype of such a system is shown in Fig. 1. Each team is comprised of a dynamically evolving plant controlled by a sensor-controller pair. The controller relies on state information from the sensor, which communicates over the shared wireless channel. Costs arise from (1) the controller transmitting data over the network, and (2) the sensor implementing control policies for plant stabilization. An effective control policy requires consistent state measurements, which increase communication costs. Conversely, reducing the number of communications lowers these costs, but degrades control performance, leading to a classic communication-control trade-off for each team. Consequently, the sensor and controller act as two active decision-makers within a team, optimizing a common objective to find a team-optimal policy pair.

Furthermore, since the channel is shared, the communication cost is coupled with the transmission activity of other users. We model this coupling effect as the product of sensor action and the average number of communication instances of the other teams. Therefore, if the channel is highly congested, each sensor will seek to reduce its communication, whereas if the channel is underutilized, each sensor will increase its transmissions to improve control performance. This dynamic interaction suggests that a (Nash) equilibrium channel utilization may emerge for each team. However, with a large number of teams, calculating exact Nash policies is challenging because each team needs to be aware of other teams’ policies. To address this, we employ the mean-field game (MFG) framework to compute approximate decentralized equilibrium policies for each team, which depend only on the local information. This approach involves analyzing the limiting team game problem (with infinite teams), which we refer to as the mean-field team (MFT) system. In the MFT system, individual team deviations become insignificant due to the infinite number of teams, and hence, it can be characterized by a representative team interacting with a mass distribution. This allows us to determine the optimal policy for a generic team, referred to as the mean-field team equilibrium (MFTE) policy, which aligns with the population’s behavior. We provide conditions for the existence and uniqueness of an approximate MFTE and demonstrate when the derived equilibrium policies constitute approximate Nash policies for the finite-team system.

Consider the networked multi-agent system shown in Fig. 2, comprised of $N$ teams. Each team $i$ is modeled by a triple $(P_{i},S_{i},C_{i})$, i.e., a dynamically evolving plant ($P_{i}$), a sensor ($S_{i}$), and a controller ($C_{i}$), where $S_{i}$ and $C_{i}$ can be viewed as the two active agents of team $i$. The plant $P_{i}$ evolves according to a stochastic linear difference equation as

where $X^{i}_{k}\in\mathbb{R}^{n}$ denotes the state, and $U^{i}_{k}\in\mathbb{R}^{m}$ denotes the control input, both for the $i^{th}$ team. The system noise $W^{i}_{k}$ is assumed to be Gaussian with zero mean and finite covariance $K_{W^{i}}$. Further, $A(\omega_{i})$ and $B(\omega_{i})$ are matrices with appropriate dimensions which depend on the type of the agent $\omega_{i}\in\Omega:=\{\omega_{1},\cdots,\omega_{p}\}$, chosen according to empirical probability mass function (pmf) $\mathbb{P}_{N}(\omega=\omega_{i})$, for all $\omega\in\Omega$. The initial state $X_{0}^{i}$ has a symmetric density with mean $\nu(\omega_{i})$ and positive definite covariance $\Sigma_{0}(\omega_{i})$ for each $i$. We assume that $X_{0}^{i}$ is independent of noise $W^{i}_{k}$ for all $k,i$.

Next, for each team $i$, full-state information ($X^{i}_{k}$) of the plant is relayed by $S_{i}$ to $C_{i}$ over a shared two-hop wireless channel via a central base station (as shown in Fig. 2) which coordinates the flow of information. The hop from the sensor to the BS is called the uplink and the one from the BS to the controller is called the downlink. The sensory information is relayed to the controller subject to a one-step delay, i.e., if $X^{i}_{k-1}$ and $y^{i}_{k}$ denote the uplink input and downlink output, respectively, then we have that

for $k\in[T]$ and $y_{0}=\varphi$. The quantity $\varphi$ denotes an empty symbol and $\gamma^{i}_{k}$ denotes the transmission requests of $S_{i}$ as

We further assume that the downlink has enough bandwidth to accommodate all the sensor requests.¹¹1The case where the downlink suffers from a capacity constraint with an ideal uplink is dealt with in the works [37, 44], where the authors use a weighted age of information-based metric to devise optimal downlink transmission strategies for the base station. However, the case where both the uplink and the downlink suffer from non-idealities is a significantly challenging problem since one would need to jointly design each team’s policies and the scheduling policy of the base station. We will leave this as an open and promising research direction. Thus, whenever a transmission request is generated, the sensor is connected to the controller by the base station. However, for each packet communicated by the sensor to the controller, team $i$ incurs a cost ($J^{S,N}_{i}$) which is proportional to the average number of users communicating on the uplink. More precisely, the associated time-average communication cost over a finite horizon of $T>0$ is given as

where we refer to the term $\frac{1}{N}\sum_{j=1}^{N}\gamma^{j}_{k}$ as the average channel utilization, which models the mean traffic flow through the wireless channel. Further, $\xi^{i}:=(\xi^{i}_{0},\xi^{i}_{1},\xi^{i}_{2},\cdots,\xi^{i}_{T-1})$ denotes the transmission policy of sensor $S_{i}$ and $\xi^{-i}:=(\xi^{1},\cdots,\xi^{i-1},\xi^{i+1},\cdots,\xi^{N})$ denotes the transmission policies of sensors other than $S_{i}$. In the absence of any communication, $C_{i}$ creates the best estimate of the state which resets to the latest information in the event of a communication.

Let us define the information set of the $i^{th}$ sensor as

with $\mathcal{I}^{S,i}_{0}=\{X^{i}_{0},y^{i}_{0}\}$. Then, we can define the space of admissible centralized²²2Note here we use the term centralized to emphasize that the information set of sensor $i$ contains the transmission instants of other agents as well. sensor policies for $S_{i}$ as $\Xi^{cen,i}=\{\xi^{i}\mid\xi^{i}_{k}~{}\text{is adapted to }\sigma(\mathcal{I}^{S,i}_{t},~{}t=0,1,\cdots,k)\}$, where $\sigma(\cdot)$ is the $\sigma$–algebra generated by its arguments. Finally, the expectation in (8) is taken with respect to the stochasticity induced by the (possibly) randomized policies, the system noise, and the initial state distribution.

Next, the controller $C_{i}$ observes the incoming message from $S_{i}$ as in (4). Between communication instants, $C_{i}$ computes the best estimate of the state $Z^{i}_{k}$ (calculated later) based on its information history given by

with $I^{C,i}_{0}=\{y^{i}_{0}\}$. For each control input applied by $C_{i}$, the $i^{th}$ team incurs a time-average control cost given as

where $Q(\omega_{i})\geq 0$, $R(\omega_{i})>0$, and $\pi^{i}:=(\pi^{i}_{0},~{}\pi^{i}_{1},\cdots,\pi^{i}_{T-1})$ denotes an admissible policy of the control agent of team $i$. More precisely, given $\mathcal{I}^{C,i}_{k}$, we define the space of admissible control policies for team $i$ as $\mathcal{M}^{i}=\{\pi^{i}\mid\pi^{i}~{}\text{is adapted to }\sigma(\mathcal{I}^{C,i}_{s},~{}s=0,1,\cdots,k)\}$. Finally, the expectation in (11) is taken with respect to the noise statistics and the initial state distribution.

As a result of the cost functions in (8) and (11), each team is faced with the following stochastic team-game problem.

Henceforth, we refer to $(\pi^{i},\xi^{i})$ as the $i^{th}$ team’s policy. We note here that each team’s policy depends on the strategies of the other teams through their sensor policies, which leads to a noncooperative game setup. Further, the works [26, 27] constitute a special case of our setup (with $N=1$). Thus, the objective of this work is to compute Nash equilibrium policies for the $N$ teams. That is, we are looking for a $2N$–tuple $(\pi^{*,i}\in{\cal M}^{i},\xi^{*,i}\in\Xi^{cen,i},i\in[N])$ such that

for all $i\in[N]$. We note that for team $i$, the above objective depends on the sensing policies $\xi^{*,-i}$ of the other teams. Due to the presence of a large number of such teams within a typical networked system, it becomes unrealistic for each team to have access to (and store) this information of the sensing policies of other teams in the population. Thus, we would like to compute team equilibrium policies based on only the local information accessible to each team. Toward this end, we will use the mean-field game framework [35, 52, 53] to characterize approximate decentralized Nash equilibrium solutions in the next three sections.

In this section, we model the $N$–team game using the mean-field team game (MFTG) framework. In this regard, we first consider the limiting system (with $N=\infty$ teams). Under this setting, the average channel utilization ratio in (8) (and hence in Problem 1) can be approximated by a pre-computable deterministic mean-field trajectory by using the Nash certainty equivalence principle [33]. This reduces Problem 1 to a team optimal control problem for a representative team, the solution of which is consistent with the other teams in the population. Thus, we first compute a team optimal policy for the representative team. Consequently, we will observe that the finite cardinality of the sensing action set (which is given as $\{0,1\}$) leads to non-contractivity of the mean field operator (as also noted in [54]). To alleviate this issue, we construct a Boltzmann-type sensor policy, prove the existence of a unique approximate mean-field team equilibrium (MFTE). Finally, we will show that the solution obtained constitutes an approximate Nash equilibrium for the finite $N$–team game problem, with “approximation” defined precisely.

We start this section by solving Problem 2. In this regard, let us start by determining the optimal controller policy. Using completion of squares, we can equivalently write the cost function in (15) as

where $\hat{R}_{k}(\omega):=R(\omega)+B^{\top}(\omega)P_{k+1}(\omega)B(\omega)$ and $P\succeq 0$ is the unique solution to the Riccati equation

with $P_{T}(\omega)=0$. Thus, from (IV), the optimal controller can be obtained as

where $Z_{k}\!:=\!\mathbb{E}[X_{k}\!\mid\!\mathcal{I}^{C,\omega}_{k}]$ is the conditional mean of the state given the information available to the controller. Then, we have the following result.

The proof of the above proposition readily follows by taking expectation in (12) conditioned on the controller information structure and is thus not included here.

Let us define $e_{k}:=X_{k}-Z_{k}$. Then, by substituting the optimal controller (17) in (IV), we arrive at

where $J^{*}=\mathbb{E}[\|X_{0}(\omega)\|^{2}_{P_{0}(\omega)}]+\frac{1}{T}\sum_{k=0}^{T-1}tr(P_{k}(\omega)K_{W}(\omega))$,

$\Gamma_{k}:=A^{\top}(\omega)P_{k+1}^{\top}(\omega)B(\omega)\hat{R}^{-1}_{k}(\omega)B^{\top}(\omega)P_{k+1}(\omega)A(\omega)$, and $e_{k}$ satisfies the following stochastic difference equation:

Thus, the sensor objective is to solve for the optimal policy minimizing (19) subject to (20). To obtain it, let us first define the optimal cost-to-go ($V_{\cdot}^{\omega}(\mathcal{I}^{S,\omega}_{\cdot},g)$) of the sensor as follows:

for all $k\in[T]$ and with the convention that $g_{-1}=0$. Then, we invoke the following definition [26] of the value of information (VoI).

We now have the following proposition which characterizes the optimal sensor policy ($\zeta^{*}$).

Let us now denote the optimal policy pair for the representative team of type $\omega$ as $(\pi^{*},\zeta^{*})$, which of course depends on the mean field $g$. Then, we proceed toward computing a MFTE using this policy. In this regard, we first make a crucial observation. The optimal sensor policy $\zeta^{*}$ computed in Proposition 2 is discontinuous at the point where $\operatorname{VoI}_{k}^{\omega}=0$. This makes it prohibitive to employ contraction-type arguments to establish the existence of a unique MFTE [33, 54]. Thus, our objective in what follows is to construct an approximation of the policy $\!\zeta^{*}\!$, by restricting it to a special class of policies, then leading to a unique (approximate) MFTE within that class.

Let $\alpha>0$, and define a new policy $\hat{\zeta}:=\zeta^{\alpha}$ from which an action $\hat{\gamma}_{k}$ is chosen as follows:

We note that the above policy is indeed an approximation of $\zeta^{*}$ since $\hat{\zeta}\rightarrow\zeta^{*}$ as $\alpha\rightarrow\infty$ under the convention that $\hat{\gamma}_{k}=1$ whenever $\operatorname{VoI}_{k}^{\omega}=0$. Further, it is also measurable with respect to $\mathcal{I}^{S,\omega}_{k}$. Let the policy $\hat{\zeta}$ belong to the policy class $\Xi^{\alpha}$. Then, we prove the existence of a unique approximate MFE in the sense above, which is defined in precise terms as below.

Also, let us define the mean-field operator $\mathcal{T}$ as

We now first state and prove the following result on the existence of an approximate MFTE.

In addition, we can guarantee the contraction of the MF operator $\mathcal{T}$ for sufficiently low values of $\alpha$, which allows us to prove a stronger result on the uniqueness of the approximate MFTE as follows.

The above results show the existence of an approximate MFTE which is unique if the parameter $\alpha$ is small enough. Further, we would also like to note from the proof of Theorem 1 that the contraction condition in (26) is only a sufficient one and one may be able to obtain a more precise characterization of the ‘smallness’ of $\alpha$. Furthermore, in the numerical simulation section, we will compute an approximate MFTE even when (26) is not satisfied.

Now that we have characterized the approximate MFTE for the mean-field game, we next proceed towards proving that this MFTE satisfies the approximate Nash property (as formalized later in Section VI) for the original finite-team game problem.

In this and the subsequent section, we return to the finite-team setup of Section II and show that the $\alpha$–approximate MFTE as computed in the previous section constitutes an ($\epsilon_{a},\epsilon_{b}$)–Nash equilibrium for the finite-team game problem. The term $\epsilon_{a}$ arises as a result of the approximation parameter $\alpha$ in the sensor policy and the term $\epsilon_{b}$ term arises due to the infinite team approximation.

Suppose now that team $i$ deviates toward using a policy different from the $\alpha$–approximate MFTE policy $\hat{\zeta}^{*}$. We call the former policy $\xi^{i}\in\Xi^{cen,i}$ and an action chosen from it at instant $k$ as $\tilde{\gamma}_{k}$. Also, let us define the quantity $\gamma^{N,av}_{k}:=\frac{1}{N}\sum_{j=1}^{N}\gamma^{j}_{k}$, which denotes the empirical average of the actions chosen by all the sensors. We start by proving the following lemma which computes the mean deviation between the empirical average $\gamma^{N,av}$ and the approximate equilibrium mean-field trajectory $\hat{g}^{*}$.

The above lemma demonstrates that the deviation of the empirical average from the approximate mean-field trajectory vanishes toward zero as the number of teams increases, and hence the latter ‘reasonably’ approximates the former if the number of teams is large.

In this subsection, we prove the approximate-Nash property of the MF solution for the finite-team game problem, which is made precise below.

We also note that the infimum on the RHS of the above inequality is taken over the set of centralized equilibrium policies as defined before, hence bringing in an element of conservatism.

Now, we define the following cost functions:

We next state and prove another main result of the paper in the following theorem, which essentially characterizes how ‘well’ the MFTE policy performs.