Informational Design of Dynamic Multi-Agent System

We follow a growing line of research on creating incentives for interacting agents to behave in a desired way. The most straightforward way is based on mechanism design approaches that properly provide reward incentives (e.g., contingent payments, penalty, supply of resources) by directly modifying the game itself to change the induced preferences of the agents over actions. Mechanism design approaches have been fruitfully studied in both static [15] as well as dynamic environment [16, zhang2021incentive, 17]. For example, auctions [18, 19] specify the way in which the agents can place their bid and clarify how the agents pay for the items; in matching markets [20, 21], matching rules matches agents in one side of a market to agents of another side that directly affect the payoff of each matched individuals. In reinforcement learning literature, reward engineering [22, 23, 24] is similar to mechanism design that directly crafts the reward functions of the agents that post specifications of the learning goal.

Our work lies in another direction: the information design. Information design studies how to influence the outcomes of the decision makings by choosing signal (also referred to as signal structure, information structure, Blackwell experiment, or data-generating process) whose realizations are observed by the agents [25]. In a seminal paper [7], Kamenica and Gentzkow has introduced Bayesian persuasion in which there is an informed sender and an uninformed receiver. The sender is endowed to commit to choosing any probability distribution (i.e., the information structure) of the signals as a function of the state of the world which is payoff-relevant to and unobserved by the receiver. The Bayesian persuasion can be interpreted as a communication device that is used by the sender to inform the receiver through the signals that contain knowledge about the state of the world. Hence, the sender controls what the agent gets to know about the payoff-relevant state. With the knowledge about the information structure, the receiver forms a posterior belief about the unobserved state based on the received signal. Hence, the information design of Bayesian persuasion is also referred to as an exercise in belief manipulation. Other works alongside with the Bayesian persuasion include [26, 27, 28, 29]. In [5], Mathevet et al. extends the single-agent Bayesian persuasion of [7] to a multi-agent game and formulate the information design of influencing agents’ behaviors through inducing distributions over agents’ beliefs. In [6], Bergemann and Morris have also considered information design in games. They have formulated the Myersonian approach for the information design in an incomplete-information environment. The essential of the Myersonian information design is the notion of Bayes correlated equilibrium, which characterizes the all possible Bayesian Nash equilibrium outcomes that could be induced by all available information structures. The Myersonian approach avoids the modeling of belief hierarchies [30] and constructs the information design problem as a linear programming. Information design has been applied in a variety of areas to study and improve real-world decision making protocols, including stress test in finance [31, 32], law enforcement and security [33, 34], censorship [35], routing system [36], finance and insurance [37, 38, 39]. Kamenica [25] has provided a recent survey of the literature of Bayesian persuasion and information design.

This work fundamentally differs from existing works on the information design. First, we consider a different environment. Specifically, we consider the setting when there are multiple sources of signals and each agent chooses one realized signal as an additional (payoff-relevant) information at each time. Among these sources of signals, there is an information designer who controls one of these sources and aims to induce equilibrium outcomes of the incomplete-information Markov game by strategically crafting information structures. Second, other than only taking actions, each agent in our model makes a coupled decision of selecting a realized signal and taking an action. Hence, the characterization of the solution concepts in our work is different from the equilibrium analysis in other works. Third, we also provide an approach with an explicit formulation to relaxing the optimal information design problem.

In this section, we first describe some fundamental concepts of a canonical Markov game model and then define our new model of a game called augmented Markov game by extending the canonical model.

This work considers games of finite agents, denoted by $\mathcal{N}\equiv[n]$, $0<n<\infty$. A normal-form (or strategic-form) is a basic representation of a static game:

Each agent $i\in\mathcal{N}$ simultaneously chooses an action $a_{i}\in\mathcal{A}$ and receives a reward $r_{i}(a_{i},\bm{a}_{-i})$ when other agents choose actions $\bm{a}_{-i}\in\mathcal{A}^{n-1}$.

With reference to Fig. 1, a canonical Markov game generalizes a normal-form game to dynamic settings as well as Markov decision processing to multi-agent interactions. We consider a finite-agent infinite-horizon Markov game. The game is played in discrete time indexed by $t=0,1,\dots$. Each agent $i$’s identification is captured by the parameter known as type denoted by $\theta_{i}$. A canonical Markov game can be defined by a tuple $\widehat{M}[\bm{\theta}]\equiv<\mathcal{N},\mathcal{G},\mathcal{A},d_{g},\mathcal{T}_{g},\{\hat{R}_{i}(\cdot|\theta_{i})\}_{i\in\mathcal{N}}>$. Here, $\mathcal{G}$ is a finite set of states. $\mathcal{A}$ is a finite set of actions available to each agent at each period. $d_{g}\in\Delta(\mathcal{G})$ is an initial distribution of the state. $\mathcal{T}_{g}:\mathcal{G}\times\mathcal{A}^{n}\mapsto\Delta(\mathcal{G})$ is the transition function of the state, such that $\mathcal{T}_{g}(\cdot|g_{t},\bm{a}_{t})\in\Delta(\mathcal{G})$ specifies the probability distribution of next-period state when the current state is $g_{t}$ and the current-period joint action is $\bm{a}_{t}\equiv(a_{i,t})_{i\in\mathcal{N}}$. Each agent $i$’s goal is characterized by his reward function $\hat{R}_{i}(\cdot|\theta_{i}):\mathcal{G}\times\mathcal{A}^{n}\mapsto\mathbb{R}$ is the reward function of agent $i$ parameterized by $\theta_{i}\in\Theta$ that realizes a one-stage reward $r_{i}(g_{t},\bm{a}_{t})$ for agent $i$ when the state is $g_{t}$ and the joint action is $\bm{a}_{t}$. A canonical Markov game is complete-information.

A solution to $\widehat{M}$ is a sequence of policy profile $\{\bm{\hat{\pi}}_{t}\}_{t\geq 0}$ in which $\bm{\hat{\pi}}_{t}:\mathcal{G}\times\mathcal{A}^{n}\mapsto\Delta(\mathcal{A}^{n})$, such that $\bm{\hat{\pi}}_{t}(\cdot|g_{t})$ specifies the probability of the joint action of the agents given the current-period state $g_{t}$. A policy profile is stationary if the agents’ decisions of actions depend only on the current-period payoff-relevant information and is independent of the calendar time; i.e., we denote the policy profile as $\bm{\hat{\pi}}:\mathcal{G}\times\mathcal{A}^{n}\mapsto\Delta(\mathcal{A}^{n})$. When the policy profile is a pure strategy if $\bm{\hat{\pi}}:\mathcal{G}\times\mathcal{A}^{n}\mapsto\mathcal{A}^{n}$ is a deterministic mapping. In a canonical Markov game, $\bm{\hat{\pi}}$ can be either independent (i.e., $\bm{\hat{\pi}}(\bm{a}_{t}|s_{t})=\prod_{i\in\mathcal{N}}\hat{\pi}_{i}(a_{i,t}|s_{t})$) or correlated (i.e., a joint function).

According to Ionescu Tulcea theorem (see, e.g., [41]), the initial distribution $d_{g}$ on $g_{0}$, the transition function $\mathcal{T}_{g}$, and the policy profile $\bm{\hat{\pi}}$ together define a unique probability measure $P^{\bm{\hat{\pi}}}$ on $(\mathcal{G}\times\mathcal{A}^{n})^{\infty}$. We denote the expectation with respect to $P^{\bm{\hat{\pi}}}$ as $\mathbb{E}_{\bm{\hat{\pi}}}[\cdot]$. The optimality criterion for each agent’s decision making at each period is to maximize his expected payoff. Each agent discounts his future payoffs by a discount factor $0<\gamma<1$. Thus, agent $i$’s period-$t$ infinite-horizon interim expected payoff is defined as: for $g_{t}\in\mathcal{G}$, $\bm{a}_{t}\in\mathcal{A}^{n}$, $\theta_{i}\in\Theta$, $i\in\mathcal{N}$, $t\geq 0$,

Let $\bm{\hat{h}}\equiv(\bm{a}^{(\infty)},g^{(\infty)})\in\bm{\hat{H}}\equiv\mathcal{A}^{n\times\infty}\times\mathcal{G}^{\infty}$ denote any infinite sequence of action-state pairs. Define $\mathtt{ExR}_{i}(\bm{\hat{h}})\equiv\sum_{t=0}^{\infty}\gamma^{t}\hat{R}_{i}(\bm{a}_{t},g_{t}|\theta_{i})$, for any $\bm{\hat{h}}\in\bm{\hat{H}}$, $\theta_{i}$, $i\in\mathcal{N}$. Let $\bm{\hat{h}}(t)$ denote the first $t$ components of $\bm{\hat{h}}$, for all $t\geq 0$.

Lemma 1 shows that for a fixed discount rate $0<\gamma<1$, the canonical Markov game is continuous at infinity. The continuity at infinity is essential for the existence of MPE of infinite-horizon stochastic game. The following proposition shows the existence of MPE due to [maskin2001markov].

With reference to Fig. 2, we extend the canonical Markov game model in this section by introducing an additional payoff-relevant information for the agents in addition to the state.

Signals. We call the additional information as signal. Let $\Omega$ be a finite set of signals. We consider that at each period $t$ each agent $i$ observes a batch of $m$ (finite) signals (signal batch, batch), for $m>1$, denoted by $W_{i,t}\equiv\{\omega^{j}_{i,t}\}_{j=1}^{m}\subseteq\Omega^{m}$, for all $i\in\mathcal{N}$, $t\geq 0$, in which $\omega^{j}_{i,t}\in\Omega$ denotes a typical signal indexed by $j\in[m]$ sent to agent $i$ at $t$. Upon observing $W_{i,t}$, each agent $i$ selects one signal $\omega_{i,t}$ from the batch $W_{i,t}$ and $\omega_{i,t}$ becomes payoff-relevant to him in addition to the state $g_{t}$.

Dynamics of Information. Let $\omega^{k}_{i,t}$ denote one signal contained in the batch $W_{i,t}$. Given $\omega^{k}_{i,t}$, the signal batch can be represented as $W_{i,t}=\{\omega^{k}_{i,t},W^{-k}_{i,t}\}$, where $W^{-k}_{i,t}\equiv\{\omega^{j}_{i,t}\}_{j=1,j\neq k}^{m}$, for all $i\in\mathcal{N}$, $t\geq 0$. We assume that the probability of $\omega^{k}_{i,t}$ at any $t$ is specified by $\mathcal{P}^{k}_{i,t}\in\Delta(\Omega)$. Let $\mathcal{P}^{-k}_{i,t}\in\Delta(\Omega^{m-1})$ denote the probability of $W^{-k}_{i,t}$. Here, both $\mathcal{P}^{k}_{i,t}$ and $\mathcal{P}^{-k}_{i,t}$, for all $i\in\mathcal{N}$ may depend on the current state $g_{t}$, the joint type $\bm{\theta}$, the calendar time, or the past realizations of states or actions. Although there is an additional information in the game, the transition of the state is controlled by the current $g_{t}$ and the realized actions $\bm{a}_{t}$ and, however, is independent of the selected signal $\omega_{i,t}$, for all $i\in\mathcal{N}$, $t\geq 0$. As in the canonical game $\widehat{M}$, the transition of the state is Markov: given any $g_{t}\in\mathcal{G},\bm{a}_{t}\in\mathcal{A}^{n}$, the transition function $\mathcal{T}_{g}(g_{t+1}|g_{t},\bm{a}_{t})$ specifies the probability of a next state $g_{t+1}\in\mathcal{G}$, for all $t\geq 0$.

Augmented Markov Game. We refer to the Markov game with signal as the augmented Markov game, denoted by $M$: $M\equiv<\mathcal{N},\mathcal{G},\mathcal{A},\Omega,d_{g},\mathcal{T}_{g},$ $\{\mathcal{P}^{k}_{i,t},\mathcal{P}^{-k}_{i}\}_{i\in\mathcal{N},t\geq 0},$ $\{R_{i}(\cdot|\theta_{i})\}_{i\in\mathcal{N}}>$, where $R_{i}(\cdot|\theta_{i}):\mathcal{A}^{n}\times\mathcal{G}\times\Omega\mapsto\mathbb{R}$ is the reward function ($\theta_{i}$-parameterized) of agent $i$, which takes into consideration agent $i$’s selected signal. A special setting for $M$ is that each agent $i$ makes two sequentially-coupled decisions. Specifically, agent $i$ first selects a signal $\omega_{i,t}$ from the batch $W_{i,t}$, given the state $g_{t}$; then, he takes an action $a_{i,t}$, given $g_{t}$ and his selected signal $\omega_{i,t}$. Hence, at a given period $t$, if the state is $g_{t}$, agent $i$ observes a set of signals $W_{i,t}$, the joint action of all other agents is $\bm{a}_{-i,t}$, and if agent $i$ firstly selects a signal $\omega_{i,t}$ from $W_{i,t}$ and secondly takes an action $a_{i,t}$, then the single-period reward is $R_{i}(\{a_{i,t},\bm{a}_{-i,t}\},g_{t},\omega_{i,t}|\theta_{i})$. We assume that the game model $M$ is complete-information.

In this section, we define a stationary equilibrium concept of the game $M[\bm{\alpha}|\bm{\theta}]$. With a slight abuse of notation, we suppress the notations of $\mathcal{P}^{-k}_{i}$, $W^{-k}_{i,t}$, and $\bm{W}^{-k}_{t}$ and only show $\mathcal{P}^{k}_{i,t}$, $\omega^{k}_{i,t}$ (of $W_{i,t}$), and $\bm{\omega}^{k}_{t}$ (of $\bm{W}_{t}$), for all $i\in\mathcal{N}$, $t\geq 0$, unless otherwise stated. Since we focus on stationary environment, we suppress the time indexes from the notations, unless otherwise stated.

Similar to the canonical game $\widehat{M}$, Ionescu Tulcea theorem (see, e.g., [41]) implies that the initial distribution $d_{g}$ of the state, the transition function $\mathcal{T}_{g}$, the distribution $\bm{\mathcal{P}}^{-k}$, the signaling rule $\bm{\alpha}$, and the strategy profile $<\bm{\beta},\bm{\pi}>$ together define a unique probability measure $P^{\bm{\alpha},\bm{\beta}}_{\bm{\pi}}$ on $(\mathcal{G}\times\Omega^{m\times n}\times\mathcal{A}^{n})^{\infty}$. The expectation with respect to $P^{\bm{\alpha},\bm{\beta}}_{\bm{\pi}}$ is denoted by $\mathbb{E}^{\bm{\beta},\bm{\alpha}}_{\bm{\pi}}\big{[}\cdot\big{]}$ or $\mathbb{E}^{\bm{\beta},\bm{\alpha}}_{\bm{\pi}}\big{[}\cdot\big{|}\cdot\big{]}$. With a slight abuse of notation, let $\mathcal{T}^{\bm{\alpha},\bm{\beta},\bm{\pi}}_{g^{\prime}g}(\bm{\omega};\bm{\omega}^{k})$ denote the transition probability from state $g$ to state $g^{\prime}$, given that the signal batch is $\bm{\omega}^{k}$ and the agents select $\bm{\omega}=\bm{\beta}(g,\bm{\theta}|\bm{\omega}^{k})$: for any $\bm{\omega}^{k}\in\Omega^{n}$ with $\bm{\alpha}(\bm{\omega}^{k}|g,\bm{\theta})>0$,

Let $\mathcal{T}^{\bm{\alpha},\bm{\beta},\bm{\pi}}_{g^{\prime},g}=\sum_{\bm{\omega}^{k}}\mathcal{T}^{\bm{\beta},\bm{\pi}}_{g^{\prime},g}(\bm{\omega};$ $\bm{\omega}^{k})\bm{\alpha}(\omega^{k}|g,\bm{\theta})$. Given $\bm{\alpha}$, $\bm{\beta}$, $\bm{\pi}$, define the state-signal value function $V_{i}^{\bm{\pi},\bm{\beta},\bm{\alpha}}$ of agent $i$, representing agent $i$’s expected reward, originating at some $g,\bm{\omega}^{k}\in\mathcal{G}\times\Omega^{n}$ with $\bm{\alpha}(\bm{\omega}^{k}|g,\bm{\theta})>0$, agents select $\bm{\omega}=\bm{\beta}(g,\bm{\theta}|\bm{\omega}^{k})$,

where $\bm{\omega}^{\prime}=\bm{\beta}(g,\bm{\theta}|\bm{\omega}^{k^{\prime}})$.

Define the state value function $J^{\bm{\pi},\bm{\beta},\bm{\alpha}}_{i}$ of agent $i$ that describes his expected reward, originating at any state $g\in\mathcal{G}$:

Define the state-signal-action value function $Q^{\bm{\pi},\bm{\beta},\bm{\alpha}}_{i}$ that represents agent $i$’s expected reward if $(\bm{\omega},\bm{a})\in\Omega^{n}\times\mathcal{A}^{n}$ are played in $(g,\bm{\omega}^{k})\in\mathcal{G}\times\Omega^{n}$:

where $\bm{\omega}^{\prime\prime}=\bm{\beta}(g^{\prime\prime},\bm{\theta}|\bm{\omega}^{k^{\prime\prime}})$.

We define an equilibrium concept known as sequential Markov perfect equilibrium (SMPE) as follows.

An SMPE extends the stationary Markov perfect equilibrium (see, e.g., [42]) to our augmented Markov game $M[\bm{\alpha}|\bm{\theta}]$ The sequential-perfect rationality describes the coupled sequential best responses of each agent’s selection and action given a state and the available signal batch. In words, a strategy profile $<\bm{\beta}^{*},\bm{\pi}^{*}>$ is sequential-perfectly rational if (i) given that agents choose actions according to the equilibrium policy profile $\bm{\pi}^{*}$, there is no state $g\in\mathcal{G}$ such that once it is reached, the agents strictly prefer to deviate from $\bm{\beta}^{*}$; and (ii) there is no information set $(g,\{\bm{\omega}^{*},\bm{W}^{-k}\})\in\mathcal{G}\times\Omega^{m\times n}$ where $\bm{\omega}^{*}$ is selected by $\bm{\beta}^{*}$ such that once it is reached, the agents strictly prefer to deviate from $\bm{\pi}^{*}$.

The concept of SMPE in Definition 1.2, however, permits arbitrarily complex and possibly nonstationary deviations from (stationary) equilibrium profile $<\bm{\beta}^{*},\bm{\pi}^{*}>$. The following lemma states that it entails no loss of generality to consider any one-shot deviations from $<\bm{\beta}^{*},\bm{\pi}^{*}>$.

A one-shot deviation is a behavior of each agent $i$’s deviating from the equilibrium profile $<\bm{\beta}^{*},\bm{\pi}^{*}>$ by selecting a signal using any $\beta^{\prime}_{i,0}$ and taking an action using any $\pi^{\prime}_{i,0}$ at the initial period of any subgame of $M[\bm{\alpha}|\bm{\theta}]$, then reverting back to his equilibrium profile for the rest of the game. The one-shot deviation property in Lemma 2 allows the principal to restrict attention to the equilibrium characterization by considering the robustness of the information design to the one-shot deviation without lose of generality.

As in a canonical Markov game, each agent $i$’s decision of choosing an action $a$ takes into account other agents’ decisions of choosing $\bm{a}_{-i}$ because its immediate reward of taking $a_{i}$ directly depends on $\bm{a}_{-i}$. In an augmented Markov game $M[\bm{\alpha}|\bm{\theta}]$, agent $i$’s choices of $\beta_{i}$ and $\pi_{i}$ are coupled because $\omega_{i}$ specified by $\beta_{i}$ has a direct causal effect on $a_{i}$ through $\pi_{i}$. Thus, each agent’s immediate reward indirectly depends on other agents’ selected signals through their actions. Hence, agents’ selection of signals is also a part of the strategic interactions in a $M[\bm{\alpha}|\bm{\theta}]$. Since $\bm{\mathcal{P}}^{-k}$ is fixed, the principal’s choice of $\bm{\alpha}$ controls the dynamics of $\bm{W}_{t}$ given the strategy profiles. Therefore, it is possible for the principal to influence the equilibrium behaviors of agents in $M[\bm{\alpha}|\bm{\theta}]$ through proper designs of $\bm{\alpha}$.

The principal’s information design takes an objective-first approach to design the information structure (given $\Omega$) of the signals sent to agents, toward desired objectives $\bm{\kappa}$, in a strategic setting through the design of $\bm{\alpha}$, where self-interested agents act rationally by choosing $\bm{\beta}$ and $\bm{\pi}$. Although any realization of the signal depends on the current state, the choice of $\bm{\alpha}$ is independent of the realizations of the states. The key restriction on the principal’s $\bm{\alpha}$ is that the agents are elicited to perform equilibrium actions that coincide with the principal’s goal, which is referred to as admissibility.

The admissibility imposes a constraint on the signaling rule $\bm{\alpha}$ and the induced policy profile $\bm{\pi}$ such that the goal $\bm{\kappa}$ is achieved in the sense of (10). We define a strong version of admissibility as follows.

The strong version constrains $\bm{\alpha}$ and $\bm{\pi}$ additionally by imposing individual-level constraint on the signaling rule. It is straightforward to verify that the strong admissibility implies admissibility but not vice versa.

The success criterion of information design for the game $M[\bm{\alpha}|\bm{\theta}]$ in equilibrium is captured by the notion of implementability.

The SMPE implementability requires that (i) the signaling rule $\bm{\alpha}$ designed by the principal induces a SMPE of $M[\bm{\alpha}|\bm{\theta}]$ and (ii) the principal’s goal is achieved (i.e., the equilibrium policy profile is admissible or strong admissible).

Given any $\bm{\mathcal{P}}^{-k}$, the distribution of the action $\bm{a}$ conditioning on any state $g$ is jointly determined by the agents’ $\bm{\beta}$ and the principal’s $\bm{\alpha}$. Hence, given $\bm{\mathcal{P}}^{-k}$, the signal $\bm{\omega}^{k}$ sent by the principal by using $\bm{\alpha}$ ultimately influences each agent’ expected reward. However, this information is transmitted indirectly through the agents’ selection rules when the selected signal $\bm{\omega}=\bm{\beta}(g,\bm{\theta}|\{\bm{\omega}^{k},\bm{W}^{-k}\})$ is not equal to $\bm{\omega}^{-k}$. We refer to the design of $\bm{\alpha}$ that induces such selection rules in SMPE of $M[\bm{\alpha}|\bm{\theta}]$ as an indirect information design (IID). We call the game with such $\bm{\alpha}$ as indirect augmented Markov game, denoted by $M^{-D}[\bm{\alpha}]$.

As a designer, the principal takes into consideration how each agent strategically behaves according to the game rules and reactions to his opponents’ behaviors as well as the responses from the environment. In any $M^{-D}[\bm{\alpha}]$, the principal’s design of $\bm{\alpha}$ must predict the agents’ equilibrium selection rule profile $\bm{\beta}$ and their corresponding equilibrium policy profile $\bm{\pi}$ that might be induced by $\bm{\alpha}$. In contrast to $M^{-D}[\bm{\alpha}]$, the principal may elect to a direct information design in which the principal makes her signals payoff-relevant to each agent’s decision of choosing an action by incentivizing each agent to select her signal at each state. We refer to the game with such signaling rule as direct augmented Markov game, denoted by $M^{D}[\bm{\alpha}]$. The implementability of the direct information design requires a restriction noted as obedience in addition to the admissibility.

We will refer to a SMPE with obedient selection rule profile as (dominant-strategy or Bayesian) obedient SMPE (O-SMPE).

In a $M^{D}[\bm{\alpha}|\bm{\theta}]$, the principal wants $\bm{\omega}^{k}_{t}$ to directly enter the immediate rewards of the agents at each period $t$. The OIL guarantees that (i) agents would want to select the signal $\bm{\omega}^{k}$ sent by the principal than choose any other signals from $\bm{W}^{-k}$, and (ii) agents take actions specified by the admissible policy other than other available actions. Hereafter, we denote obedient selection rule by $\bm{\beta}^{O}$ and $\beta^{O}_{i}$, for all $i\in\mathcal{N}$.

A successful information design depends on the principal’s having accurate beliefs in regard to the agents’ decision processes. This includes all the possible indirect selection behaviors of the agents, i.e., all possible $\bm{\beta}\neq\bm{\beta}^{O}$. The point of direct information design is that it allows the principal to ignore analyzing all of agents’ indirect selections behaviors and focus on the obedient $\bm{\beta}^{O}$.

In this section, we characterize the OIL and formulate the principal’s information design problem given a goal $\bm{\kappa}$.

The following proposition is an analog of Bellman’s Theorem [43].

From Proposition 2.1, we can re-define $V^{\bm{\alpha},\bm{\beta},\bm{\pi}}_{i}$, $J^{\bm{\alpha},\bm{\beta},\bm{\pi}}_{i}$, and $Q^{\bm{\alpha},\bm{\beta},\bm{\pi}}_{i}$ given in (3)-(5), respectively, as the unique pair of value functions that satisfy the Bellman recursions (14), (15), and (16).