The Sample-Communication Complexity Trade-off in Federated Q-Learning

Sudeep Salgia
CMU
Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA. Yuejie Chi¹¹footnotemark: 1
CMU

(April 2024)

We consider the problem of federated Q-learning, where $M$ agents aim to collaboratively learn the optimal Q-function of an unknown infinite-horizon Markov decision process with finite state and action spaces. We investigate the trade-off between sample and communication complexities for the widely used class of intermittent communication algorithms. We first establish the converse result, where it is shown that a federated Q-learning algorithm that offers any speedup with respect to the number of agents in the per-agent sample complexity needs to incur a communication cost of at least an order of $\frac{1}{1-\gamma}$ up to logarithmic factors, where $\gamma$ is the discount factor. We also propose a new algorithm, called Fed-DVR-Q , which is the first federated Q-learning algorithm to simultaneously achieve order-optimal sample and communication complexities. Thus, together these results provide a complete characterization of the sample-communication complexity trade-off in federated Q-learning.

1 Introduction

Reinforcement Learning (RL) (Sutton2018RLBook) refers to a paradigm of sequential decision making where an agent aims to learn an optimal policy, i.e., a policy that maximizes the long-term total reward, through repeated interactions with an unknown environment. RL finds applications across a diverse array of fields including, but not limited to, autonomous driving, games, recommendation systems, robotics and Internet of Things (IoT) (Kober2013RoboticsSurvey; Yurtsever2020SelfDriving; Huang2016Go; Lim2020IoT).

The primary hurdle in RL applications is often the high-dimensional nature of the decision space that necessitates the learning agent to have to access to an enormous amount of data in order to have any hope of learning the optimal policy. Moreover, the sequential collection of such an enormous amount of data through a single agent is extremely time-consuming and often infeasible in practice (mnih2016a3c). Consequently, practical implementations of RL involve deploying multiple agents to collect data in parallel. This decentralized approach to data collection has fueled the design and development of distributed or federated RL algorithms that can collaboratively learn the optimal policy without actually transferring the collected data to a centralized server, while achieving a linear speedup in terms of the number of agents. Such a federated approach to RL, which does not require the transfer of local data, is gaining interest due to lower bandwidth requirements and lower security and privacy risks. In this work, we focus on federated variants of the vastly popular Q-learning algorithm (Watkins1992QL), where the agents collaborate to directly learn the optimal Q-function without forming an estimate of the underlying unknown environment.

A particularly important aspect of designing federated RL algorithms, including federated Q-learning algorithms, is to address the natural tension between sample and communication complexities. At one end of the spectrum lies the naïve approach of running a centralized algorithm with an optimal sample complexity after transferring and combining all the collected data at a central server. Such an approach trivially achieves the optimal sample complexity while suffering from a very high and prohibitive communication complexity. On the other hand, several recently proposed algorithms (Khodadadian2022FederatedQL; Woo2023FedSynQ; Woo2024FedOffline; zheng2024federated) operate in more practical regimes, offering significantly lower communication complexities when compared to the naïve approach at the cost of sub-optimal sample complexities. These results suggest the existence of an underlying trade-off between sample and communication complexities of federated RL algorithms. The primary goal of this work is to better understand this trade-off in the context of federated Q-learning by investigating these following fundamental questions.

•

Fundamental limit of communication: What is the minimum amount of communication required by a federated Q-learning algorithm to achieve any statistical benefit of collaboration?
•

Optimal algorithm design: How does one design a federated Q-learning algorithm that simultaneously offers optimal sample and communication complexity guarantees, i.e., operates on the optimal frontier of the sample-communication complexity trade-off?

1.1 Main results

In this work, we consider a setup where $M$ distributed agents — each with access to a local generative model (Kearns1998FiniteSampleQL) — collaborate to learn the optimal Q-function of an infinite-horizon Markov decision process (MDP) (Puterman2014MDPBook), which is defined over a finite state space $\mathcal{S}$ and a finite action space $\mathcal{A}$ , and has a discount factor of $\gamma\in(0,1)$ . To probe the communication complexity, we consider a common setup in federated learning called the intermittent communication setting (Woodworth2021IntermittentComm), where the agents intermittently share information among themselves with the help of a central server. We provide a complete characterization of the trade-off between sample and communication complexities under the aforementioned setting by providing answers to both the questions. Summarized below, the main result of this work is twofold.

•

Fundamental lower bounds on the communication complexity of federated Q-learning. We establish lower bounds on the communication complexity of federated Q-learning, both in terms of the number of communication rounds and the overall number of bits that need to be transmitted in order to achieve any speedup in the convergence rate with respect to the number of agents. Specifically, we show that in order for an intermittent communication algorithm to obtain any benefit of collaboration, i.e., any order of speedup with respect to the number of agents, the number of communication rounds must be least $\Omega\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ and the number of bits sent by each agent to the server must be least $\Omega\left(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)\log^{2}N}\right)$ , where $N$ denotes the number of samples taken by the algorithm for each state-action pair.
•

Achieving the optimal sample-communication complexity trade-off of federated Q-learning. We propose a new federated Q-learning algorithm called Federated Doubly Variance Reduced Q-Learning (dubbed Fed-DVR-Q ), that simultaneously achieves order-optimal sample complexity and communication complexity as dictated by the lower bound. We show that Fed-DVR-Q learns an $\varepsilon$ -accurate optimal Q-function in the $\ell_{\infty}$ sense with $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{M\varepsilon^{2}(1-\gamma)^{3}}\right)$ i.i.d. samples from the generative model at each agent while incurring a total communication cost of $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}\right)$ bits per agent across $\tilde{\mathcal{O}}\left(\frac{1}{1-\gamma}\right)$ rounds of communication. Thus, Fed-DVR-Q not only improves upon both the sample and communication complexities of existing algorithms, but also is the first algorithm to achieve order-optimal sample and communication complexities (See Table 1 for a comparison).

Algorithm/Reference	Number of Agents	Sample Complexity	Communication Complexity
Q-learning (Li2023QLMinimax)	$1$	$\dfrac{\|\mathcal{S}\|\|\mathcal{A}\|}{(1-\gamma)^{4}\varepsilon^{2}}$	N/A
Variance Reduced Q-learning (Wainwright2019VarianceReduced)	$1$	$\dfrac{\|\mathcal{S}\|\|\mathcal{A}\|}{(1-\gamma)^{3}\varepsilon^{2}}$	N/A
Fed-SynQ (Woo2023FedSynQ)	$M$	$\dfrac{\|\mathcal{S}\|\|\mathcal{A}\|}{M(1-\gamma)^{5}\varepsilon^{2}}$	$\dfrac{M}{1-\gamma}$
Fed-DVR-Q (This work)	$M$	$\dfrac{\|\mathcal{S}\|\|\mathcal{A}\|}{M(1-\gamma)^{3}\varepsilon^{2}}$	$\dfrac{1}{1-\gamma}$
Lower bound ((Azar2013MinimaxPAC), This work)	$M$	$\dfrac{\|\mathcal{S}\|\|\mathcal{A}\|}{M(1-\gamma)^{3}\varepsilon^{2}}$	$\dfrac{1}{1-\gamma}$

Table 1: Comparison of sample and communication complexities (per-agent) of various single-agent and federated Q-learning algorithms for learning an

\varepsilon

-optimal Q-function under the synchronous setting. We hide logarithmic factors and burn-in costs for all results for simplicity of presentation. Here,

\mathcal{S}

and

\mathcal{A}

represent state and action spaces respectively and

\gamma

denotes the discount factor. We report the communication complexity only in terms of the number of rounds as existing algorithms assume transmission of real numbers and hence do not report bit-level costs. For the lower bound, Azar2013MinimaxPAC and this work establish the lower bounds for the sample and communication complexities, respectively.

1.2 Related work

Single-agent Q-learning.

Q-learning has been extensively studied in the single-agent setting in terms of its asymptotic convergence (Jaakkola1993StochasticIterativeDP; Tsitsiklis1994AsynchronousQL; Szepesvari1997AsymptoticQL; Borkar2000ODE-RL) and its finite-time sample complexity in synchronous (EvenDar2004LearningRates; Kearns1998FiniteSampleQL; Beck2012ConstantStepSize; sidford2018near; Wainwright2019ConeContractive; Wainwright2019VarianceReduced; Chen2020FiniteSampleAnalysis; Li2023QLMinimax) and asynchronous settings (Chen2021Lyapunov; Li2023QLMinimax; Li2021SampleComplexityAsynchronousQL; Qu2020FiniteAsychronousQL; xia2024instance) in terms of convergence in the $\ell_{\infty}$ sense. On the other hand, regret analysis of Q-learning has been carried out in both online settings (jin2018q; bai2019provably; menard2021ucb; li2021breaking) and offline settings (shi2022pessimistic; yan2022efficacy), to name a few.

Federated and distributed RL.

There has also been a considerable effort towards developing distributed and federated RL algorithms. The distributed variants of the classical temporal difference (TD) learning algorithm have been investigated in a series of studies (Chen2021OffPolicyTDC; Doan2019TD0LinearFunctionMARL; Doan2021DistributedTD0; Sun2020FiniteDecentralizedTDLinear; Wai2020ConvergenceConsensus; liu2023distributed; Wang2020DecentralizedTDTracking; Zeng2021FiniteDecentralizedSA). The impact of environmental heterogeneity in federated RL was studied in Wang2023FederatedTDHeterogeneity for TD learning, and in Jin2022FederatedRLHeterogeneity when the local environments are known. A distributed version of the actor-critic algorithm was studied by Shen2023AsynchronousAdvantageActorCritic where the authors established convergence of their algorithm and demonstrated a linear speedup in the number of agents in their sample complexity bound. Chen2022DecentralizedActorCritic proposed a new distributed actor-critic algorithm which improved the dependence of sample complexity on the error $\varepsilon$ with a communication cost of $\tilde{\mathcal{O}}(\varepsilon^{-1})$ . Chen2021CommEfficientPolicyGrad proposed a communication-efficient distributed policy gradient algorithm with convergence analysis and established a communication complexity of $\mathcal{O}(1/(M\varepsilon))$ . Xie2023FedKL adopts a distributed policy optimization perspective, which is different from the Q-learning paradigm considered in this work. Moreover, the algorithm in Xie2023FedKL obtains a linear communication cost, which is worse than that obtained in our work. Similarly, Zhang2024FiniteTimeOnPolicy focuses on on-policy learning and incurs a communication cost that depends polynomially on the required error $\varepsilon$ . Several additional studies (Yang2023FederatedNPG; Zeng2021DecentralizedNPG; Lan2024AsynchronousPG) have also developed and analyzed other distributed/federated variants of the classical natural policy gradient method (Kakade2001PolicyGradient). Assran2019GossipActorCritic; Espeholt2018Impala; Mnih2016AsynchronousDeepRL developed distributed algorithms to train deep RL networks more efficiently.

Federated Q-learning.

Federated Q-learning has been explored relatively recently with theoretical sample and communication complexity guarantees. Khodadadian2022FederatedQL proposed and analyzed a federated Q-learning algorithm in the asynchronous setting, however, its sample complexity guarantee exhibits pessimistic dependencies with respect to salient problem-dependent parameters. Woo2023FedSynQ provided improved analyses for federated Q-learning under both synchronous and asynchronous settings, and introduced importance averaging to tame the heterogeneity of local behavior policies in the asynchronous setting to further improve the sample complexity, showing that a collaborative coverage of the entire state-action space suffices for federated Q-learning. Moving to the offline setting, Woo2024FedOffline proposed a federated Q-learning algorithm for offline RL in the finite-horizon setting and established sample and communication complexities that only require a collaborative coverage of the state-action pairs visited by the optimal policy. zheng2024federated, on the other hand, established a linear speedup for federated Q-learning in the online setting from the regret minimization perspective.

Accuracy-communication trade-off in federated learning.

The trade-off between communication complexity and accuracy (or equivalently, sample complexity) has been studied in various federated and distributed learning problems, including stochastic approximation algorithms for convex optimization. Duchi2014DistributedEstimation; Braverman2016CommunicationLowerBounds established the celebrated inverse linear relationship between the error and the communication cost for the problem of distributed mean estimation. Similar trade-offs for distributed stochastic optimization, multi-armed bandits and linear bandits have been studied and established across numerous studies, e.g., (Woodworth2018GraphOracle; Woodworth2021IntermittentComm; Tsitsiklis1987; Shi2021FMAB; Salgia2023LinearBandits).

2 Background and Problem Formulation

In this section, we provide a brief background of MDPs, outline the performance measures for federated Q-learning algorithms and describe the class of intermittent communication algorithms considered in this work.

2.1 Markov decision processes

We focus on an infinite-horizon MDP, denoted by $\mathcal{M}$ , over a state space $\mathcal{S}$ and an action space $\mathcal{A}$ with a discount factor $\gamma\in(0,1)$ . Both the state and action spaces are assumed to be finite sets. In an MDP, the state $s$ evolves dynamically under the influence of actions based on a probability transition kernel, $P:(\mathcal{S}\times\mathcal{A})\times\mathcal{S}\mapsto[0,1]$ . The entry $P(s^{\prime}|s,a)$ denotes the probability of moving to state $s^{\prime}$ when an action $a$ is taken in state $s$ . An MDP is also associated with a deterministic reward function $r:\mathcal{S}\times\mathcal{A}\mapsto[0,1]$ , where $r(s,a)$ denotes the immediate reward obtained for taking action $a$ in state $s$ . Thus, the transition kernel $P$ along with the reward function $r$ completely characterize an MDP.

A policy $\pi:\mathcal{S}\mapsto\Delta(\mathcal{A})$ is a rule for selecting actions across different states, where $\Delta(\mathcal{A})$ denotes the simplex over $\mathcal{A}$ and $\pi(a|s)$ denotes the probability of choosing action $a$ in state $s$ . Each policy $\pi$ is associated with a state value function and a state-action value function, or the Q-function, denoted by $V^{\pi}$ and $Q^{\pi}$ respectively. Specifically, $V^{\pi}$ and $Q^{\pi}$ measure the expected discounted cumulative reward attained by policy $\pi$ starting from certain state $s$ and state-action pair $(s,a)$ respectively. Mathematically, $V^{\pi}$ and $Q^{\pi}$ are given as

\displaystyle V^{\pi}(s):=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\bigg{|}\ s_{0}=s\right];\quad Q^{\pi}(s,a):=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\bigg{|}\ s_{0}=s,a_{0}=a\right],

(1)

where $a_{t}\sim\pi(\cdot|s_{t})$ , $s_{t+1}\sim P(\ \cdot\ |s_{t},a_{t})$ for all $t\geq 0$ , and the expectation is taken w.r.t. the randomness in the trajectory $\{s_{t},a_{t}\}_{t=1}^{\infty}$ . Since the rewards lie in $[0,1]$ , it follows immediately that both the value function and the Q-function lie in the range $[0,\frac{1}{1-\gamma}]$ .

An optimal policy $\pi^{\star}$ is a policy that maximizes the value function uniformly over all the states and it has been shown that such an optimal policy $\pi^{\star}$ always exists (Puterman2014MDPBook). The optimal value and Q-functions are those corresponding to that of an optimal policy $\pi^{\star}$ , denoted as $V^{\star}:=V^{\pi^{\star}}$ and $Q^{\star}:=Q^{\pi^{\star}}$ respectively. The optimal Q-function, $Q^{\star}$ , is also the unique fixed point of the Bellman optimality operator $\mathcal{T}:\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S}\times\mathcal{A}$ , given by

\displaystyle\mathcal{T}Q^{\star}=Q^{\star},\qquad\mbox{where}\quad(\mathcal{T}Q)(s,a)=r(s,a)+\gamma\cdot\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}\left[\max_{a^{\prime}\in\mathcal{A}}Q(s^{\prime},a^{\prime})\right].

(2)

The popular Q-learning algorithm (Watkins1992QL) aims to learn the optimal policy by first learning $Q^{\star}$ as the solution to the above fixed-point equation — via stochastic approximation when $\mathcal{T}$ is only accessed through samples — and then obtaining a deterministic optimal policy via greedy action selection, i.e., $\pi^{\star}(s)=\operatorname*{arg\,max}_{a}Q^{\star}(s,a)$ .

2.2 Performance measures in federated Q-learning

We consider a federated learning setup consisting of $M$ agents, where all the agents face a common, unknown MDP, i.e., the transition kernel and the reward function are the same across agents. In addition, we consider the synchronous setting (Wainwright2019ConeContractive), where each agent has access to an independent generative model or simulator from which they can draw independent samples from the unknown underlying distribution $P(\cdot|s,a)$ for all state-action pair $(s,a)$ (Kearns1998FiniteSampleQL) simultaneously. Let $Z\in\mathcal{S}^{|\mathcal{S}||\mathcal{A}|}$ be a corresponding random vector whose $(s,a)$ -th coordinate is drawn from the distribution $P(\cdot|s,a)$ , independently of all other coordinates. We define the random operator $\hat{\mathcal{T}}_{Z}:(\mathcal{S}\times\mathcal{A})\mapsto(\mathcal{S}\times\mathcal{A})$ as

\displaystyle(\hat{\mathcal{T}}_{Z}Q)(s,a)=r(s,a)+\gamma V(Z(s,a)),

(3)

where $V(s^{\prime})=\max_{a^{\prime}\in\mathcal{A}}Q(s^{\prime},a^{\prime})$ . The operator $\hat{\mathcal{T}}_{Z}$ can be interpreted as the sample Bellman operator, where we have the relation $\mathcal{T}Q=\mathbb{E}_{Z}\big{[}\hat{\mathcal{T}}_{Z}Q\big{]}$ for all Q-functions. For a given value of $\varepsilon\in(0,\frac{1}{1-\gamma})$ , the objective of agents is to collaboratively learn an $\varepsilon$ -optimal estimate (in the $\ell_{\infty}$ sense) of the optimal Q-function of the unknown MDP.

We measure the performance of a federated Q-learning algorithm $\mathscr{A}$ using two metrics — sample complexity and communication complexity. For a given MDP $\mathcal{M}$ , let $\widehat{Q}_{\mathcal{M}}(\mathscr{A},N,M)$ denote the estimate of $Q^{\star}_{\mathcal{M}}$ , the optimal Q-function of the MDP $\mathcal{M}$ , returned by an algorithm $\mathscr{A}$ , when given access to $N$ i.i.d. samples from the generative model for each $(s,a)$ pair at all the $M$ agents. The error rate of the algorithm $\mathscr{A}$ , denoted by $\textsf{ER}(\mathscr{A};N,M)$ , is defined as

\displaystyle\textsf{ER}(\mathscr{A};N,M):=\sup_{\mathcal{M}}\mathbb{E}\left[\big{\|}\widehat{Q}_{\mathcal{M}}(\mathscr{A},N,M)-Q^{\star}_{\mathcal{M}}\big{\|}_{\infty}\right],

(4)

where the expectation is taken over the samples and any randomness in the algorithm. Given a value of $\varepsilon\in(0,\frac{1}{1-\gamma})$ , the sample complexity of $\mathscr{A}$ , denoted by $\textsf{SC}(\mathscr{A};\varepsilon,M)$ is given by

\displaystyle\textsf{SC}(\mathscr{A};\varepsilon,M):=|\mathcal{S}||\mathcal{A}|\cdot\min\left\{N\in\mathbb{N}:\textsf{ER}(\mathscr{A};N,M)\leq\varepsilon\right\}.

(5)

Similarly, we can also define a high-probability version for any $\delta\in(0,1)$ as follows:

\displaystyle\textsf{SC}(\mathscr{A};\varepsilon,M,\delta):=|\mathcal{S}||\mathcal{A}|\cdot\min\left\{N\in\mathbb{N}:\Pr\left(\sup_{\mathcal{M}}\|\widehat{Q}_{\mathcal{M}}(\mathscr{A},N,M)-Q^{\star}_{\mathcal{M}}\|_{\infty}\leq\varepsilon\right)\geq 1-\delta\right\}.

We measure the communication complexity of any federated learning algorithm both in terms of the frequency of information exchange and the total number of bits uploaded by the agents. For each agent $m$ , let $C_{\textsf{round}}^{m}(\mathscr{A};N)$ and $C_{\textsf{bit}}^{m}(\mathscr{A};N)$ respectively denote the number of times agent $m$ sends a message, and, the total number of bits uploaded by agent $m$ to the server when an algorithm $\mathscr{A}$ is run with $N$ i.i.d. samples from the generative model for each $(s,a)$ pair at all the $M$ agent. The communication complexity of $\mathscr{A}$ , when measured in terms of the frequency of communication and the total number of bits exchanged, is given by

\displaystyle\textsf{CC}_{\textsf{round}}(\mathscr{A};N):=\frac{1}{M}\sum_{m=1}^{M}C_{\textsf{round}}^{m}(\mathscr{A};N);\quad\textsf{CC}_{\textsf{bit}}(\mathscr{A};N):=\frac{1}{M}\sum_{m=1}^{M}C_{\textsf{bit}}^{m}(\mathscr{A};N),

(6)

respectively. Similarly, for a given value of $\varepsilon\in(0,\frac{1}{1-\gamma})$ , we can also define $\textsf{CC}_{\textsf{round}}(\mathscr{A};\varepsilon)$ and $\textsf{CC}_{\textsf{bit}}(\mathscr{A};\varepsilon)$ when $\mathscr{A}$ is run to guarantee an error of at most $\varepsilon$ , as well as the high-probability version for any $\delta\in(0,1)$ as $\textsf{CC}_{\textsf{round}}(\mathscr{A};\varepsilon,\delta)$ and $\textsf{CC}_{\textsf{bit}}(\mathscr{A};\varepsilon,\delta)$ .

2.3 Intermittent-communication algorithm protocols

We consider a popular class of federated learning algorithms with intermittent communication. The intermittent communication setting provides a natural framework to extend single-agent Q-learning algorithms to the distributed setting. As the name suggests, under this setting, the agents intermittently communicate with each other or a central server, sharing their updated beliefs about $Q^{\star}$ . Between two communication rounds, each agent updates their belief about $Q^{\star}$ using stochastic approximation iterations based on the locally available data, similar to a single-agent setup. Such intermittent communication algorithms have been extensively studied and used to establish lower bounds on communication complexity of distributed stochastic convex optimization (Woodworth2018GraphOracle; Woodworth2021IntermittentComm).

1: Input :

T,R,\{\eta_{t}\}_{t=1}^{T},\mathcal{C}=\{t_{r}\}_{r=1}^{R},B

2: Set

Q_{0}^{m}=0

for all agents

m

3: for

t=1,2,\dots,T

4: for

m=1,2,\dots,M

5: Compute

Q_{t-\frac{1}{2}}^{m}

according to Eqn. (7)

6: Compute

Q_{t}^{m}

according to Eqn. (8)

7: end for

8: end for

9: return

Q_{T}

Algorithm 1 A generic federated Q-learning algorithm

\mathscr{A}

A generic federated Q-learning algorithm with intermittent communication is outlined in Algorithm 1. It is characterized by the following five parameters: (i) the total number of updates $T$ ; (ii) the number of communication rounds $R$ ; (iii) a step size schedule $\{\eta_{t}\}_{t=1}^{T}$ ; (iv) a communication schedule $\mathcal{C}=\{t_{r}\}_{r=1}^{R}$ ; (v) batch size $B$ . During the $t$ -th iteration, each agent $m$ computes $\{\widehat{\mathcal{T}}_{Z_{b}}(Q_{t-1}^{m})\}_{b=1}^{B}$ , a minibatch of sample Bellman operators at the current estimate $Q_{t-1}^{m}$ , using $B$ samples from the generative model for each $(s,a)$ pair, and obtains an intermediate local estimate using the Q-learning update rule as follows:

\displaystyle Q_{t-\frac{1}{2}}^{m}=(1-\eta_{t})Q_{t-1}^{m}+\frac{\eta_{t}}{B}\sum_{b=1}^{B}\widehat{\mathcal{T}}_{Z_{b}}(Q_{t-1}^{m}).

(7)

Here, $\eta_{t}\in(0,1]$ is the step size chosen corresponding to the $t$ -th time step. The intermediate estimates are averaged based on a communication schedule $\mathcal{C}=\{t_{r}\}_{r=1}^{R}$ consisting of $R$ rounds, i.e.,

\displaystyle Q_{t}^{m}=\begin{cases}\frac{1}{M}\sum_{j=1}^{M}Q_{t-\frac{1}{2}}^{j}&\text{ if }t\in\mathcal{C},\\ Q_{t-\frac{1}{2}}^{m}&\text{otherwise.}\end{cases}

(8)

In the above equation, the averaging step can also be replaced with any distributed mean estimation routine that includes compression to control the bit level costs. Without loss of generality, we assume that $Q^{m}_{0}=0$ for all agent and $t_{R}=T$ , i.e., the last iterates are always averaged. It is straightforward to note that the number of samples taken per agent by an intermittent communication algorithm is $BT$ , i.e, $N=BT$ and the communication complexity $\textsf{CC}_{\textsf{round}}(\mathscr{A};N)=R$ .

3 Communication Complexity Lower Bound

In this section, we investigate the first of the two questions regarding the lower bound on communication complexity. The following theorem establishes a lower bound on the communication complexity of a federated Q-learning algorithm with intermittent communication as described in Algorithm 1.

Theorem 1.

Assume that $\gamma\in[5/6,1)$ and the state and action spaces satisfy $|\mathcal{S}|\geq 4$ and $|\mathcal{A}|\geq 2$ . Let $\mathscr{A}$ be a federated Q-learning algorithm with intermittent communication (as described in Algorithm 1) that is run for $T\geq\max\{16,\frac{1}{1-\gamma}\}$ steps with a step size schedule of either $\eta_{t}:=\frac{1}{1+c_{\eta}(1-\gamma)t}$ or $\eta_{t}:=\eta$ for all $1\leq t\leq T$ . If

\displaystyle R=\textsf{CC}_{\textsf{round}}(\mathscr{A};N)\leq\frac{c_{0}}{(1-\gamma)\log^{2}N};\text{ or }\textsf{CC}_{\textsf{bit}}(\mathscr{A};N)\leq\frac{c_{1}|\mathcal{S}||\mathcal{A}|}{(1-\gamma)\log^{2}N}

for some universal constants $c_{0},c_{1}>0$ , then for all choices of communication schedule, batch size $B$ , $c_{\eta}>0$ and $\eta\in(0,1)$ , the error of $\mathscr{A}$ satisfies

\displaystyle\textsf{ER}(\mathscr{A};N,M)\geq\frac{C_{\gamma}}{\sqrt{N}\log^{3}N}

for all $M\geq 2$ and $N=BT$ . Here $C_{\gamma}>0$ is a constant that depends only on $\gamma$ .

The above theorem states that for any federated Q-learning algorithm with intermittent communication to obtain any benefit of collaboration, i.e., for the error rate $\textsf{ER}(\mathscr{A};N,M)$ to decrease w.r.t. the number of agents, it must have at least $\Omega\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ rounds of communication and transmit $\Omega\left(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)\log^{2}N}\right)$ bits of information per agent, both of which scale inverse proportionally to the effective horizon $\frac{1}{1-\gamma}$ of the MDP. The above lower bound on the communication complexity also immediately applies to federated Q-learning algorithms that offer order-optimal sample complexity, and thereby a linear speedup with respect to the number of agents. Therefore, this characterizes the converse relation for the sample-communication tradeoff in federated Q-learning.

The above lower bound on the communication complexity of federated Q-learning is a consequence of the bias-variance trade-off that governs the convergence of the Q-learning algorithm. While a careful choice of step sizes alone is sufficient to balance this trade-off in the centralized setting, the choice of communication schedule also plays an important role in balancing this trade-off in the federated setting. The local steps between two communication rounds induce a positive estimation bias that depends on the standard deviation of the iterates and is a well-documented issue of “over-estimation” in Q-learning (Hasselt2010DoubleQL). Since such a bias is driven by local updates, it does not reflect any benefit of collaboration. During a communication round, the averaging of iterates across agents allows the algorithm an opportunity to counter this bias by reducing the effective variance of the updates through averaging. In our analysis, we show that if the communication is infrequent, the local bias becomes the dominant term and averaging of iterates is insufficient to counter the impact of the positive bias induced by the local steps. As a result, we do not observe any statistical gains when the communication is infrequent. Our argument is inspired by the analysis of Q-learning in Li2023QLMinimax and is based on analyzing the convergence of an intermittent communication algorithm on a specifically chosen “hard” MDP instance. The detailed proof is deferred to Appendix A.

Remark 1 (Communication complexity of policy evaluation).

Several recent studies (liu2023distributed; tian2024one) established that a single round of communication is sufficient to achieve linear speedup of TD learning for policy evaluation, which do not contradict with our results focusing on Q-learning for policy learning. The latter is more involved due to the nonlinearity of the Bellman optimality operator. Specifically, if the operator whose fixed point is to be found is linear in the decision variable (e.g., the value function in TD learning) then the fixed point update only induces a variance term corresponding to the noise. However, if the operator is non-linear, then in addition to the variance term, we also obtain a bias term in the fixed point update. While the variance term can be controlled with one-shot averaging, more frequent communication is necessary to ensure that the bias term is small enough.

Remark 2 (Extension to asynchronous Q-learning).

We would like to point out that our lower bound extends to the asynchronous setting (Li2023QLMinimax) as the assumption of i.i.d. noise corresponding to a generative model is a special case of Markovian noise observed in the asynchronous setting.

4 The Fed-DVR-Q Algorithm

Having characterized the lower bound on the communication complexity of federated Q-learning, we explore our second question of interest — designing a federated Q-learning algorithm that achieves this lower bound while simultaneously offering an optimal order of sample complexity (up to logarithmic factors).

We propose a new federated Q-learning algorithm, Fed-DVR-Q , that achieves not only a communication complexity of $\textsf{CC}_{\textsf{round}}=\tilde{\mathcal{O}}\left(\frac{1}{1-\gamma}\right)$ and $\textsf{CC}_{\textsf{bit}}=\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}\right)$ but also order-optimal sample complexity (up to logarithmic factors), thereby providing a tight characterization of the achievability frontier that matches with the converse result derived in the previous section.

4.1 Algorithm description

{wrapfigure}

R0.6 1: Input : Error bound $\varepsilon>0$ , failure probability $\delta>0$ 2: $k\leftarrow 1,Q^{(0)}\leftarrow\mathbf{0}$ 3: // Set parameters as described in Section 4.1.3 4: for $k=1,2,\dots,K$ do 5: $Q^{(k)}\leftarrow\textsc{RefineEstimate}(Q^{(k-1)},B,I,{L}_{k},D_{k},J)$ 6: $k\leftarrow k+1$ 7: end for 8: return $Q^{(K)}$ Algorithm 2 Fed-DVR-Q

Fed-DVR-Q proceeds in epochs. During an epoch $k\geq 1$ , the agents collaboratively update $Q^{(k-1)}$ , the estimate of $Q^{\star}$ obtained at the end of the previous epoch, to a new estimate $Q^{(k)}$ , with the aid of the sub-routine called RefineEstimate. The sub-routine RefineEstimate is designed to ensure that the suboptimality gap, $\|Q^{(k)}-Q^{\star}\|_{\infty}$ , reduces by a factor of $2$ at the end of every epoch. Thus, at the end of $K=\mathcal{O}(\log(1/\varepsilon))$ epochs, Fed-DVR-Q obtains an $\varepsilon$ -optimal estimate of $Q^{\star}$ , which is then set to be the output of the algorithm. Please refer to Algorithm 2 for a pseudocode.

4.1.1 The RefineEstimate sub-routine

RefineEstimate, starting from $\overline{Q}$ , an initial estimate of $Q^{\star}$ , uses variance-reduced Q-learning updates (Wainwright2019VarianceReduced; sidford2018near) to obtain an improved estimate of $Q^{\star}$ . It is characterized by four parameters — the initial estimate $\overline{Q}$ , the number of local iterations $I$ , the re-centering sample size $L$ and the batch size $B$ , which can be appropriately tuned to control the quality of the returned estimate. Additionally, it also takes input two parameters $D$ and $J$ required by the compressor $\mathscr{C}(\cdot;D,J)$ , whose description is deferred to Section 4.1.2.

The first step in RefineEstimate is to collaboratively approximate $\mathcal{T}\overline{Q}$ for the variance-reduced updates. To this effect, each agent $m$ builds an approximation of $\mathcal{T}\overline{Q}$ as follows:

\displaystyle\widetilde{\mathcal{T}}^{(m)}_{L}(\overline{Q}):=\frac{1}{\lceil{L}/M\rceil}\sum_{l=1}^{\lceil{L}/M\rceil}\mathcal{T}_{Z_{l}^{(m)}}(\overline{Q}),

(9)

where $\{Z_{1}^{(m)},Z_{2}^{(m)},\ldots,Z_{\lceil{L}/M\rceil}^{(m)}\}$ are $\lceil{L}/M\rceil$ i.i.d. samples with $Z_{1}^{(m)}\sim Z$ . Each agent then sends a compressed version, $\mathscr{C}\left(\widetilde{\mathcal{T}}^{(m)}_{{L}}(\overline{Q})-\overline{Q};D,J\right)$ , of the difference $\widetilde{\mathcal{T}}^{(m)}_{{L}}(\overline{Q})-\overline{Q}$ to the server, which collects all the estimates from the agents and constructs the estimate

\displaystyle\widetilde{\mathcal{T}}_{L}(\overline{Q})=\overline{Q}+\frac{1}{M}\sum_{m=1}^{M}\mathscr{C}\left(\widetilde{\mathcal{T}}^{(m)}_{{L}}(\overline{Q})-\overline{Q};D,J\right)

(10)

and sends it back to the agents for the variance-reduced updates. Equipped with the estimate $\widetilde{\mathcal{T}}_{L}(\overline{Q})$ , RefineEstimate constructs a sequence $\{Q_{i}\}_{i=1}^{I}$ using the following iterative update scheme initialized with $Q_{0}=\overline{Q}$ . During the $i$ -th iteration, each agent $m$ carries out the following update:

\displaystyle Q_{i-\frac{1}{2}}^{m}=(1-\eta)Q_{i-1}+\eta\left[\widehat{\mathcal{T}}_{i}^{(m)}Q_{i-1}-\widehat{\mathcal{T}}_{i}^{(m)}\overline{Q}+\widetilde{\mathcal{T}}_{L}(\overline{Q})\right].

(11)

In the above equation, $\eta\in(0,1)$ is the step size and $\widehat{\mathcal{T}}_{i}^{(m)}Q:=\frac{1}{B}\sum_{z\in\mathcal{Z}_{i}^{(m)}}\mathcal{T}_{z}Q$ , where $\mathcal{Z}_{i}^{(m)}$ is the minibatch of $B$ i.i.d. random variables drawn according to $Z$ , independently at each agent $m$ for all iterations $i$ . Each agent then sends a compressed version of the update, i.e., $\mathscr{C}\left(Q_{i-\frac{1}{2}}^{m}-Q_{i-1};D,J\right)$ , to the server, which uses them to compute the next iterate

\displaystyle Q_{i}=Q_{i-1}+\frac{1}{M}\sum_{m=1}^{M}\mathscr{C}\left(Q_{i-\frac{1}{2}}^{m}-Q_{i-1};D,J\right),

(12)

and broadcast it to the agents. After $I$ such updates, the obtained iterate $Q_{I}$ is returned by the sub-routine. A pseudocode of RefineEstimateis given in Algorithm 3.

1: Input: Initial estimate

\overline{Q}

, batch size

B

, number of iterations

I

, re-centering sample size

L

, quantization bound

D

, message size

J

2: // Build an approximation for

\mathcal{T}\overline{Q}

which is to be used for variance reduced updates

3: for

m=1,2,\dots,M

4: Draw

\lceil L/M\rceil

i.i.d. samples from the generative model for each

(s,a)

pair and evaluate

\widetilde{\mathcal{T}}_{L}^{(m)}(\overline{Q})

according to Eqn. (9)

5: Send

\mathscr{C}(\widetilde{\mathcal{T}}_{L}^{(m)}(\overline{Q})-\overline{Q};D,J)

to the server

6: Receive

\frac{1}{M}\sum_{m=1}^{M}\mathscr{C}(\widetilde{\mathcal{T}}_{L}^{(m)}(\overline{Q})-\overline{Q};D,J)

from the server and compute

\widetilde{\mathcal{T}}_{L}(\overline{Q})

according to Eqn. (10)

7: end for

Q_{0}\leftarrow\overline{Q}

9: // Variance reduced updates with minibatching

10: for

i=1,2,\dots,I

11: for

m=1,2,\dots,M

12: Draw

B

i.i.d. samples from the from the generative model for each

(s,a)

pair

13: Compute

Q_{i-\frac{1}{2}}^{m}

according to Eqn. (11)

14: Send

\mathscr{C}(Q_{i-\frac{1}{2}}^{m}-Q_{i-1};D,J)

to the server

15: Receive

\frac{1}{M}\sum_{m=1}^{M}\mathscr{C}(Q_{i}^{m}-Q_{i-1};D,J)

from the server and compute

Q_{i}

according to Eqn. (12)

16: end for

17: end for

18: return

Q_{I}

Algorithm 3 RefineEstimate

(\overline{Q},B,I,L,D,J)

4.1.2 The compression operator

The compressor, $\mathscr{C}(\cdot;D,J)$ , used in the proposed algorithm Fed-DVR-Q is based on the popular stochastic quantization scheme. In addition to the input vector $Q$ to be quantized, the compressor or quantizer $\mathscr{C}$ takes two input parameters $D$ and $J$ : (i) $D$ corresponds to an upper bound on the $\ell_{\infty}$ norm of $Q$ , i.e., $\|Q\|_{\infty}\leq D$ ; (ii) $J$ corresponds to the resolution of the compressor, i.e., number of bits used by the compressor to represent each coordinate of the output vector.

The compressor first splits the interval $[0,D]$ into $2^{J}-1$ intervals of equal length where $0=d_{1}<d_{2}\ldots<d_{2^{J}}=D$ correspond to end points of the intervals. Each coordinate of $Q$ is then separately quantized as follows. The value of the $n$ -th coordinate, $\mathscr{C}(Q)[n]$ , is set to be $d_{j_{n}-1}$ with probability $\frac{d_{j_{n}}-Q[n]}{d_{j_{n}}-d_{j_{n}-1}}$ and to $d_{j_{n}}$ with the remaining probability, where $j_{n}:=\min\{j:d_{j}<Q[i]\leq d_{j+1}\}$ . It is straightforward to note that each coordinate of $\mathscr{C}(Q)$ can be represented using $J$ bits.

4.1.3 Setting the parameters

The desired convergence of the iterates $\{Q^{(k)}\}$ is obtained by carefully choosing the parameters of the sub-routine RefineEstimate and the compression operator $\mathscr{C}$ . Given a target accuracy $\varepsilon\in(0,1]$ and $\delta\in(0,1)$ , the total number of epochs is set to

\displaystyle K=\left\lceil\frac{1}{2}\log_{2}\left(\frac{1}{1-\gamma}\right)\right\rceil+\left\lceil\frac{1}{2}\log_{2}\left(\frac{1}{(1-\gamma)\varepsilon^{2}}\right)\right\rceil.

(13)

For all epochs $k\geq 1$ , we set the number of iterations $I$ , the batch size $B$ , and the number of bits $J$ of the compressor $\mathscr{C}$ to be


$\displaystyle I$	$\displaystyle:=\left\lceil\frac{2}{\eta(1-\gamma)}\right\rceil,$	(14a)
$\displaystyle B$	$\displaystyle:=\left\lceil\frac{2}{M}\left(\frac{12\gamma}{1-\gamma}\right)^{2}\log\left(\frac{8KI\|\mathcal{S}\|\|\mathcal{A}\|}{\delta}\right)\right\rceil,$	(14b)
$\displaystyle J$	$\displaystyle:=\left\lceil\log_{2}\left(\frac{70}{\eta(1-\gamma)}\sqrt{\frac{4}{M}\log\Big{(}\frac{8KI\|\mathcal{S}\|\|\mathcal{A}\|}{\delta}\Big{)}}\right)\right\rceil$	(14c)

respectively. The re-centering sample sizes $L_{k}$ and bounds $D_{k}$ of the compressor $\mathscr{C}$ are set to be the following functions of epoch index $k$ respectively:

\displaystyle L_{k}:=\frac{39200}{(1-\gamma)^{2}}\log\left(\frac{8KI|\mathcal{S}||\mathcal{A}|}{\delta}\right)\cdot\begin{cases}4^{k}&\text{ if }k\leq K_{0}\\ 4^{k-K_{0}}&\text{ if }k>K_{0}\end{cases};\quad D_{k}:=16\cdot\frac{2^{-k}}{1-\gamma},

(15)

where $K_{0}=\lceil\frac{1}{2}\log_{2}(\frac{1}{1-\gamma})\rceil$ . The piecewise definition of $L_{k}$ is crucial to obtain the optimal dependence with respect to $\frac{1}{1-\gamma}$ , similar to the two-step procedure outlined in Wainwright2019VarianceReduced.

4.2 Performance guarantees

The following theorem characterizes the sample and communication complexities of Fed-DVR-Q .

Theorem 2.

Consider any $\delta\in(0,1)$ and $\varepsilon\in(0,1]$ . The sample and communication complexities of the Fed-DVR-Q algorithm, when run with the choice of parameters described in Section 4.1.3 and a learning rate $\eta\in(0,1)$ , satisfy the following relations for some universal constant $C_{1}>0$ :

	$\displaystyle\textsf{SC}(\textsf{Fed-DVR-Q };\varepsilon,M,\delta)$	$\displaystyle\leq\frac{C_{1}}{\eta M(1-\gamma)^{3}\varepsilon^{2}}\log_{2}\left(\frac{1}{(1-\gamma)\varepsilon}\right)\log\left(\frac{8KI\|\mathcal{S}\|\|\mathcal{A}\|}{\delta}\right),$
	$\displaystyle\textsf{CC}_{\textsf{round}}(\textsf{Fed-DVR-Q };\varepsilon,\delta)$	$\displaystyle\leq\frac{16}{\eta(1-\gamma)}\log_{2}\left(\frac{1}{(1-\gamma)\varepsilon}\right),$
	$\displaystyle\textsf{CC}_{\textsf{bit}}(\textsf{Fed-DVR-Q };\varepsilon,\delta)$	$\displaystyle\leq\frac{32\|\mathcal{S}\|\mathcal{A}\|}{\eta(1-\gamma)}\log_{2}\left(\frac{1}{(1-\gamma)\varepsilon}\right)\log_{2}\left(\frac{70}{\eta(1-\gamma)}\sqrt{\frac{4}{M}\log\left(\frac{8KI\|\mathcal{S}\|\|\mathcal{A}\|}{\delta}\right)}\right).$

A proof of Theorem 2 can be found in Appendix LABEL:appendix:dvr_analysis. A few implications of the theorem are in order.

Optimal sample-communication complexity trade-off.

As shown by the above theorem, Fed-DVR-Q offers a linear speedup in the sample complexity with respect to the number of agents while simultaneously achieving the same order of communication complexity as dictated by the lower bound derived in Theorem 1, both in terms of frequency and bit level complexity. Moreover, Fed-DVR-Q is the first federated Q-learning algorithm that achieves a sample complexity with optimal dependence on all the salient parameters, i.e., $|\mathcal{S}|,|\mathcal{A}|$ and $\frac{1}{1-\gamma}$ , in addition to linear speedup w.r.t. to the number of agents, thereby bridging the existing gap between upper and lower bounds on the sample complexity for federated Q-learning. Thus, Theorem 1 and 2 together provide a characterization of optimal operating point of the sample-communication complexity trade-off in federated Q-learning.

Role of minibatching.

The commonly adopted approach in intermittent communication algorithm is to use a local update scheme that takes multiple small (i.e., $B=\mathcal{O}(1)$ ), noisy updates between communication rounds, as evident from the algorithm design in Khodadadian2022FederatedQL; Woo2023FedSynQ and even numerous FL algorithms for stochastic optimization (Mcmahan2017FedAvg; Haddadpour2019LUPASGD; Khaled2020LocalSGDHeterogenous). In Fed-DVR-Q , we replace the local update scheme of taking multiple small, noisy updates by a single, large update with smaller variance, obtained by averaging the noisy updates over a minibatch of samples. The use of updates with smaller variance in variance-reduced Q-learning yields the algorithm its name. While both the approaches result in similar sample complexity guarantees, the local update scheme requires more frequent averaging across clients to ensure that the bias of the estimate, also commonly referred to as “client drift”, is not too large. On the other hand, the minibatching approach does not encounter the problem of bias accumulation from local updates and hence can afford more infrequent averaging, allowing Fed-DVR-Q to achieve optimal order of communication complexity.

Compression.

Fed-DVR-Q is the first federated Q-learning algorithm to analyze and establish communication complexity at the bit level. While all existing studies on federated Q-learning focus only on the frequency of communication and assume transmission of real numbers with infinite bit precision, our analysis provides a more holistic view point of communication complexity at the bit level, which is of great practical significance. While some recent other studies (Wang2023FederatedTDHeterogeneity) also considered quantization in federated RL, their objective is to understand the impact of message size on the convergence with no constraint on the frequency of communication, unlike the holistic viewpoint adopted in this work.

5 Conclusion

We presented a complete characterization of the sample-communication trade-off for federated Q-learning algorithms with intermittent communication. We showed that no federated Q-learning algorithm with intermittent communication can achieve any speedup with respect to the number of agents if its number of communication rounds are sublinear in $\frac{1}{1-\gamma}$ . We also proposed a new federated Q-learning algorithm called Fed-DVR-Q that uses variance reduction along with minibatching to achieve optimal-order sample and communication complexities. In particular, we showed that Fed-DVR-Q has a per-agent sample complexity of $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{M(1-\gamma)^{3}\varepsilon^{2}}\right)$ , which is order-optimal in all salient problem parameters, and a communication complexity of $\tilde{\mathcal{O}}\left(\frac{1}{1-\gamma}\right)$ rounds and $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}\right)$ bits.

The results in this work raise several interesting questions that are worth exploring. While we focus on the tabular setting in this work, it is of great interest to investigate to the trade-off in other settings where we use function approximation to model the $Q^{\star}$ and $V^{\star}$ functions. Moreover, it is interesting to explore the trade-off in the finite-horizon setting, where there is no discount factor. Furthermore, it is also worthwhile to explore if the communication complexity can be further reduced by going beyond the class of intermittent communication algorithms.

Appendix A Proof of Theorem 1

In this section, we prove the main result of the paper, the lower bound on the communication complexity of federated Q-learning algorithms. At a high level, the proof consists of the following three steps.

Introducing the “hard” MDP instance.

The proof builds upon analyzing the behavior of a generic algorithm $\mathscr{A}$ outlined in Algorithm 1 over a particular instance of MDP. The particular choice of MDP is inspired by, and borrowed from, other lower bound proofs in the single-agent setting (Li2023QLMinimax) and helps highlight core issues that lie at the heart of the sample-communication complexity trade-off. Following Li2023QLMinimax, the construction is first over a small state-action space that allows us to focus on a simpler problem before generalizing it to larger state-action spaces.

Establishing the performance of intermittent communication algorithms.

In the second step, we analyze the error of the iterates generated by an intermittent communication algorithm $\mathscr{A}$ . The analysis is inspired by the single-agent analysis in Li2023QLMinimax, which highlights the underlying bias-variance trade-off. Through careful analysis of the algorithm dynamics in the federated setting, we uncover the impact of communication on the bias-variance trade-off and the resulting error of the iterates to obtain the lower bound on the communication complexity.

Generalization to larger MDPs.

As the last step, we generalize our construction of the “hard” instance to more general state-action space and extend our insights to obtain the statement of the theorem.

A.1 Introducing the “hard” instance

We first introduce an MDP instance $\mathcal{M}_{h}$ that we will use throughout the proof to establish the result. Note that this MDP is identical to the one considered in Li2023QLMinimax to establish the lower bounds on the performance of single-agent Q-learning algorithm. It consists of four states $\mathcal{S}=\{0,1,2,3\}$ . Let $\mathcal{A}_{s}$ denote the action set associated with the state $s$ . The probability transition kernel and the reward function of $\mathcal{M}_{h}$ is given as follows:


$\displaystyle\mathcal{A}_{0}=\{1\}$	$\displaystyle\quad\quad P(0\|0,1)=1$		$\displaystyle\quad r(0,1)=0,$	(16a)
$\displaystyle\mathcal{A}_{1}=\{1,2\}$	$\displaystyle\quad\quad P(1\|1,1)=p$	$\displaystyle\quad P(0\|1,1)=1-p$	$\displaystyle\quad r(1,1)=1,$	(16b)
	$\displaystyle\quad\quad P(1\|1,2)=p$	$\displaystyle\quad P(0\|1,2)=1-p$	$\displaystyle\quad r(1,2)=1,$	(16c)
$\displaystyle\mathcal{A}_{2}=\{1\}$	$\displaystyle\quad\quad P(2\|2,1)=p$	$\displaystyle\quad P(0\|2,1)=1-p$	$\displaystyle\quad r(2,1)=1,$	(16d)
$\displaystyle\mathcal{A}_{3}=\{1\}$	$\displaystyle\quad\quad P(3\|3,1)=1$		$\displaystyle\quad r(3,1)=1,$	(16e)

where the parameter $p=\dfrac{4\gamma-1}{3\gamma}$ . We have the following results about the optimal $Q$ and $V$ functions of this hard MDP instance.

Lemma 1 ((Li2023QLMinimax, Lemma 3)).

Consider the MDP $\mathcal{M}_{h}$ constructed in Eqn. (16). We have,

	$\displaystyle V^{\star}(0)$	$\displaystyle=Q^{\star}(0,1)=0$
	$\displaystyle V^{\star}(1)$	$\displaystyle=Q^{\star}(1,1)=Q^{\star}(1,2)=V^{\star}(2)=Q^{\star}(2,1)=\frac{1}{1-\gamma p}=\frac{3}{4(1-\gamma)}$
	$\displaystyle V^{\star}(3)$	$\displaystyle=Q^{\star}(3,1)=\frac{1}{1-\gamma}.$

Throughout the next section of the proof, we focus on this MDP with four states and two actions. In Appendix A.4, we generalize the proof to larger state-action spaces.

A.2 Notation and preliminary results

For convenience, we first define some notation that will be used throughout the proof.

Useful relations of the learning rates.

We consider two kinds of step size sequences that are commonly used in Q-learning — the constant step size schedule, i.e., $\eta_{t}=\eta$ for all $t\in\{1,2,\dots,T\}$ and the rescaled linear step size schedule, i.e., $\eta_{t}=\frac{1}{1+c_{\eta}(1-\gamma)t}$ , where $c_{\eta}>0$ is a universal constant that is independent of the problem parameters.

We define the following quantities:

\displaystyle\eta_{k}^{(t)}

\displaystyle=\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i}(1-\gamma p))~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{ for all }0\leq k\leq t,

(17)

where we take $\eta_{0}=1$ and use the convention throughout the proof that if a product operation does not have a valid index, we take the value of that product to be $1$ . For any integer $0\leq\tau<t$ , we have the following relation, which will be proved at the end of this subsection for completeness:

\displaystyle\prod_{k=\tau+1}^{t}(1-\eta_{k}(1-\gamma p))+(1-\gamma p)\sum_{k=\tau+1}^{t}\eta_{k}^{(t)}=1.

(18)

Similarly, we also define,

\displaystyle\widetilde{\eta}_{k}^{(t)}

\displaystyle=\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{ for all }0\leq k\leq t,

(19)

which satisfies the relation

\displaystyle\prod_{k=\tau+1}^{t}(1-\eta_{k})+\sum_{k=\tau+1}^{t}\widetilde{\eta}_{k}^{(t)}=1.

(20)

for any integer $0\leq\tau<t$ . The claim follows immediately by plugging $p=0$ in (18). Note that for constant step size, the sequence $\widetilde{\eta}_{k}^{(t)}$ is clearly increasing. For the rescaled linear step size, we have,

\displaystyle\frac{\widetilde{\eta}_{k-1}^{(t)}}{\widetilde{\eta}_{k}^{(t)}}=\frac{\eta_{k}}{\eta_{k-1}(1-\eta_{k})}=1-\frac{(1-c_{\eta}(1-\gamma))\eta_{k}}{1-c_{\eta}(1-\gamma)\eta_{k}}\leq 1

(21)

whenever $c_{\eta}\leq\frac{1}{1-\gamma}$ . Thus, $\widetilde{\eta}_{k}^{(t)}$ is an increasing sequence as long as $c_{\eta}\leq\frac{1}{1-\gamma}$ . Similarly, $\eta_{k}^{(t)}$ is also clearly increasing for the constant step size schedule. For the rescaled linear step size schedule, we have,

\displaystyle\frac{\eta_{k-1}^{(t)}}{\eta_{k}^{(t)}}=\frac{\eta_{k}}{\eta_{k-1}(1-\eta_{k}(1-\gamma p))}\leq\frac{\eta_{k}}{\eta_{k-1}(1-\eta_{k})}\leq 1,

whenever $c_{\eta}\leq\frac{1}{1-\gamma}$ . The last bound follows from Eqn. (21).

Proof of (18).

We can show the claim using backward induction. For the base case, note that,

	$\displaystyle(1-\gamma p)\eta_{t}^{(t)}+(1-\gamma p)\eta_{t-1}^{(t)}$	$\displaystyle=(1-\gamma p)\eta_{t}+(1-\gamma p)\eta_{t-1}(1-(1-\gamma p)\eta_{t})$
		$\displaystyle=1-(1-\eta_{t}(1-\gamma p))(1-\eta_{t-1}(1-\gamma p))=1-\prod_{k=t-1}^{t}(1-\eta_{k}(1-\gamma p)),$

as required. Assume (18) is true for some $\tau$ . We have,

	$\displaystyle(1-\gamma p)\sum_{k=\tau}^{t}\eta_{k}^{(t)}$	$\displaystyle=(1-\gamma p)\eta_{\tau}^{t}+(1-\gamma p)\sum_{k=\tau+1}^{t}\eta_{k}^{(t)}$
		$\displaystyle=(1-\gamma p)\eta_{\tau}\prod_{k=\tau+1}^{t}(1-\eta_{k}(1-\gamma p))+1-\prod_{k=\tau+1}^{t}(1-\eta_{k}(1-\gamma p))$
		$\displaystyle=1-\prod_{k=\tau}^{t}(1-\eta_{k}(1-\gamma p)),$

thus completing the induction step.

Sample transition matrix.

Recall $Z\in\mathcal{S}^{|\mathcal{S}||\mathcal{A}|}$ is a random vector whose $(s,a)$ -th coordinate is drawn from the distribution $P(\cdot|s,a)$ . We use $\widehat{P}_{t}^{m}$ to denote the sample transition at time $t$ and agent $m$ obtained by averaging $B$ i.i.d. samples from the generative model. Specifically let $\{Z_{t,b}^{m}\}_{b=1}^{B}$ denote a collection of $B$ i.i.d. copies of $Z$ collected at time $t$ at agent $m$ . Then, for all $s,a,s^{\prime}$ ,

\displaystyle\widehat{P}_{t}^{m}(s^{\prime}|s,a)=\frac{1}{B}\sum_{b=1}^{B}P_{t,b}^{m}(s^{\prime}|s,a),

(22)

where $P_{t,b}^{m}(s^{\prime}|s,a)=\mathbbm{1}\{Z_{t,b}^{m}(s,a)=s^{\prime}\}$ for $s^{\prime}\in\mathcal{S}$ .

Preliminary relations of the iterates.

We state some preliminary relations regarding the evolution of the Q-function and the value function across different agents that will be helpful for the analysis later.

We begin with the state $0$ , where we have $Q_{t}^{m}(0,1)=V_{t}^{m}(0)=0$ for all agents $m\in[M]$ and $t\in[T]$ . This follows almost immediately from the fact that state $0$ is an absorbing state with zero reward. Note that $Q_{0}^{m}(0,1)=V_{0}^{m}(0)=0$ holds for all clients $m\in[M]$ . Assuming that $Q_{t-1}^{m}(0,1)=V_{t-1}^{m}(0)=0$ for all clients for some time instant $t-1$ , by induction, we have,

\displaystyle Q_{t-1/2}^{m}(0,1)=(1-\eta_{t})Q_{t-1}^{m}(0,1)+\eta_{t}(\gamma V_{t-1}^{m}(0))=0.

Consequently, $Q_{t}^{m}(0,1)=0$ and $V_{t}^{m}(0)=0$ , for all agents $m$ , irrespective of whether there is averaging.

For state $3$ , the iterates satisfy the following relation:

	$\displaystyle Q_{t-1/2}^{m}(3,1)$	$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(3,1)+\eta_{t}(1+\gamma V_{t-1}^{m}(3))$
		$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(3,1)+\eta_{t}(1+\gamma Q_{t-1}^{m}(3,1))$
		$\displaystyle=(1-\eta_{t}(1-\gamma))Q_{t-1}^{m}(3,1)+\eta_{t},$

where the second step follows by noting $V_{t}^{m}(3)=Q_{t}^{m}(3,1)$ . Once again, one can note that averaging step does not affect the update rule implying that the following holds for all $m\in[M]$ and $t\in[T]$ :

\displaystyle V_{t}^{m}(3)=Q_{t}^{m}(3,1)=\sum_{k=1}^{t}\eta_{k}\left(\prod_{i=k+1}^{t}(1-\eta_{i}(1-\gamma))\right)=\frac{1}{1-\gamma}\left[1-\prod_{i=1}^{t}(1-\eta_{i}(1-\gamma))\right],

(23)

where the last step follows from Eqn. (18) with $p=1$ .

Similarly, for state $1$ and $2$ , we have,

$\displaystyle Q_{t-1/2}^{m}(1,1)$	$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(1,1)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,1)V_{t-1}^{m}(1)),$	(24)
$\displaystyle Q_{t-1/2}^{m}(1,2)$	$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(1,2)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,2)V_{t-1}^{m}(1)),$	(25)
$\displaystyle Q_{t-1/2}^{m}(2,1)$	$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(2,1)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(2\|2,1)V_{t-1}^{m}(2)).$	(26)

Since the averaging makes a difference in the update rule, we further analyze the update as required in later proofs.

A.3 Main analysis

We first focus on establishing a bound on the number of communication rounds, i.e., $\textsf{CC}_{\textsf{round}}(\mathscr{A})$ (where we drop the dependency with other parameters for notational simplicity), and then use this lower bound to establish the bound on the bit level communication complexity $\textsf{CC}_{\textsf{bit}}(\mathscr{A})$ .

To establish the lower bound on $\textsf{CC}_{\textsf{round}}(\mathscr{A})$ for any intermittent communication algorithm $\mathscr{A}$ , we analyze the convergence behavior of $\mathscr{A}$ on the MDP $\mathcal{M}_{h}$ . We assume that the averaging step in line $6$ of Algorithm 1 is carried out exactly. Since the use of compression only makes the problem harder, it is sufficient for us to consider the case where there is no loss of information in the averaging step for establishing a lower bound. Lastly, throughout the proof, without loss of generality we assume that

\displaystyle\log N\leq\frac{1}{1-\gamma},

(27)

otherwise, the lower bound in Theorem 1 reduces to the trivial lower bound.

We divide the proof into following three parts based on the choice of learning rates and batch sizes:

1.

Small learning rates: For constant learning rates, $0\leq\eta<\frac{1}{(1-\gamma)T}$ and for rescaled linear learning rates, the constant $c_{\eta}$ satisfies $c_{\eta}\geq\log T$ .
2.

Large learning rates with small $\eta_{T}/(BM)$ : For constant learning rates, $\eta\geq\frac{1}{(1-\gamma)T}$ and for rescaled linear learning rates, the constant $c_{\eta}$ satisfies $0\leq c_{\eta}\leq\log T\leq\frac{1}{1-\gamma}$ (c.f. (27)). Additionally, the ratio $\frac{\eta_{T}}{BM}$ satisfies $\frac{\eta_{T}}{BM}\leq\frac{1-\gamma}{100}$ .
3.

Large learning rates with large $\eta_{T}/(BM)$ : We have the same condition on the learning rates as above. However, in this case the ratio $\frac{\eta_{T}}{BM}$ satisfies $\frac{\eta_{T}}{BM}>\frac{1-\gamma}{100}$ .

We consider each of the cases separately in the following three subsections.

A.3.1 Small learning rates

In this subsection, we prove the lower bound for small learning rates, which follow from similar arguments in Li2023QLMinimax.

For this case, we focus on the dynamics of state $2$ . We claim that the same relation established in Li2023QLMinimax continues to hold, which will be established momentarily:

\displaystyle\mathbb{E}[V_{T}^{m}(2)]=\left(\frac{1}{M}\sum_{j=1}^{M}\mathbb{E}[V_{T}^{j}(2)]\right)=\sum_{k=1}^{T}\eta_{k}^{(t)}=\frac{1-\eta_{0}^{(T)}}{1-\gamma p}.

(28)

Consequently, for all $m\in[M]$ , we have

\displaystyle V^{\star}(2)-\mathbb{E}[V_{T}^{m}(2)]=\frac{\eta_{0}^{(T)}}{1-\gamma p}.

(29)

To obtain lower bound on $V^{\star}(2)-\mathbb{E}[V_{T}^{m}(2)]$ , we need to obtain a lower bound on $\eta_{0}^{(T)}$ , which from (Li2023QLMinimax, Eqn. (120)) we have

\displaystyle\log(\eta_{0}^{(T)})\geq-1.5\sum_{t=1}^{T}\eta(1-\gamma p)\geq-2\sum_{t=1}^{T}\frac{1}{t\log T}\geq-2\qquad\Longrightarrow\qquad\eta_{0}^{(T)}\geq e^{-2}

when $T\geq 16$ for both choices of learning rates. On plugging this bound in (29), we obtain,

\displaystyle\mathbb{E}[\|Q_{T}^{m}-Q^{\star}\|_{\infty}]\geq\mathbb{E}[|Q^{\star}(2)-Q_{T}^{m}(2)|]\geq V^{\star}(2)-\mathbb{E}[V_{T}^{m}(2)]\geq\frac{3}{4e^{2}(1-\gamma)\sqrt{N}}

(30)

holds for all $m\in[M]$ , $N\geq 1$ and $M\geq 2$ . Thus, it can be noted that the error rate $\textsf{ER}(\mathscr{A};N,M)$ is bounded away from a constant value irrespective of the number of agents and the number of communication rounds. Thus, even with $\textsf{CC}_{\textsf{round}}=\Omega(T)$ , we will not observe any collaborative gain if the step size is too small.

Proof of (28).

Recall that from (26), we have,

\displaystyle Q_{t-1/2}^{m}(2,1)

\displaystyle=(1-\eta_{t})V_{t-1}^{m}(2)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(2|2,1)V_{t-1}^{m}(2)).

Here, $Q_{t-1}^{m}(2,1)=V_{t-1}^{m}(2)$ as the second state has only a single action.

•

When $t$ is not an averaging instant, we have,

\displaystyle V_{t}^{m}(2)=Q_{t}^{m}(2,1)=(1-\eta_{t})V_{t-1}^{m}(2)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(2|2,1)V_{t-1}^{m}(2)).

(31)

On taking expectation on both sides of the equation, we obtain,

$\displaystyle\mathbb{E}[V_{t}^{m}(2)]$	$\displaystyle=(1-\eta_{t})\mathbb{E}[V_{t-1}^{m}(2)]+\eta_{t}(1+\gamma\mathbb{E}[\widehat{P}_{t}^{m}(2\|2,1)V_{t-1}^{m}(2)])$
	$\displaystyle=(1-\eta_{t})\mathbb{E}[V_{t-1}^{m}(2)]+\eta_{t}\left(1+\gamma\mathbb{E}[\widehat{P}_{t}^{m}(2\|2,1)]\mathbb{E}[V_{t-1}^{m}(2)]\right)$
	$\displaystyle=(1-\eta_{t})\mathbb{E}[V_{t-1}^{m}(2)]+\eta_{t}\left(1+\gamma p\mathbb{E}[V_{t-1}^{m}(2)]\right)$
	$\displaystyle=(1-\eta_{t}(1-\gamma p))\mathbb{E}[V_{t-1}^{m}(2)]+\eta_{t}.$	(32)

In the second step, we used the fact that $\widehat{P}_{t}^{m}(2|2,1)$ is independent of $V_{t-1}^{m}(2)$ .

•

Similarly, if $t$ is an averaging instant, we have,

	$\displaystyle V_{t}^{m}(2)$	$\displaystyle=Q_{t}^{m}(2,1)=\frac{1}{M}\sum_{j=1}^{M}Q_{t-1/2}^{j}(2,1)$
		$\displaystyle=(1-\eta_{t})\frac{1}{M}\sum_{j=1}^{M}V_{t-1}^{j}(2)+\frac{1}{M}\sum_{j=1}^{M}\eta_{t}(1+\gamma\widehat{P}_{t}^{j}(2\|2,1)V_{t-1}^{j}(2)).$		(33)

Once again, upon taking expectation we obtain,

$\displaystyle\mathbb{E}[V_{t}^{m}(2)]$	$\displaystyle=(1-\eta_{t})\frac{1}{M}\sum_{j=1}^{M}\mathbb{E}[V_{t-1}^{j}(2)]+\frac{1}{M}\sum_{j=1}^{M}\eta_{t}(1+\gamma\mathbb{E}[\widehat{P}_{t}^{j}(2\|2,1)V_{t-1}^{j}(2)])$
	$\displaystyle=(1-\eta_{t})\frac{1}{M}\sum_{j=1}^{M}\mathbb{E}[V_{t-1}^{j}(2)]+\frac{1}{M}\sum_{j=1}^{M}\eta_{t}(1+\gamma p\mathbb{E}[V_{t-1}^{j}(2)])$
	$\displaystyle=(1-\eta_{t}(1-\gamma p))\left(\frac{1}{M}\sum_{j=1}^{M}\mathbb{E}[V_{t-1}^{j}(2)]\right)+\eta_{t}.$	(34)

Eqns. (32) and (34) together imply that for all $t\in[T]$ ,

\displaystyle\left(\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}[V_{t}^{m}(2)]\right)=(1-\eta_{t}(1-\gamma p))\left(\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}[V_{t-1}^{m}(2)]\right)+\eta_{t}.

(35)

On unrolling the above recursion with $V_{0}^{m}=0$ for all $m\in[M]$ , we obtain the desired claim (28).

A.3.2 Large learning rates with small $\frac{\eta_{T}}{BM}$

In this subsection, we prove the lower bound for case of large learning rates when the ratio $\frac{\eta_{T}}{BM}$ is small. For the analysis in this part, we focus on the dynamics of state $1$ . Unless otherwise specified, throughout the section we implicitly assume that the state is $1$ .

We further define a key parameter that will play a key role in the analysis:

\displaystyle\tau:=\min\{k\in\mathbb{N}:\forall\ t\geq k,\eta_{t}\leq\eta_{k}\leq 3\eta_{t}\}.

(36)

It can be noted that for constant step size sequence $\tau=1$ and for rescaled linear stepsize $\tau=T/3$ .

Step 1: introducing an auxiliary sequence.

We define an auxiliary sequence $\widehat{Q}^{m}_{t}(a)$ for $a\in\{1,2\}$ and all $t=1,2,\dots,T$ to aid our analysis, where we drop the dependency with state $s=1$ for simplicity. The evolution of the sequence $\widehat{Q}_{t}^{m}$ is defined in Algorithm 4, where $\widehat{V}_{t}^{m}=\max_{a\in\{1,2\}}\widehat{Q}_{t}^{m}(a)$ . In other words, the iterates $\{\widehat{Q}_{t}^{m}\}$ evolve exactly as the iterates of Algorithm 1 except for the fact that sequence $\{\widehat{Q}_{t}^{m}\}$ is initialized at the optimal $Q$ -function of the MDP. We would like to point out that we assume that the underlying stochasticity is also identical in the sense that the evolution of both $Q_{t}^{m}$ and $\widehat{Q}_{t}^{m}$ is governed by the same $\widehat{P}_{t}^{m}$ matrices. The following lemma controls the error between the iterates $Q_{t}^{m}$ and $\widehat{Q}_{t}^{m}$ , allowing us to focus only on $\widehat{Q}_{t}^{m}$ .

1: Input :

T,R,\{\eta_{t}\}_{t=1}^{T},\mathcal{C}=\{t_{r}\}_{r=1}^{R},B

2: Set

\widehat{Q}_{0}^{m}(a)\leftarrow Q^{\star}(1,a)

for

a\in\{1,2\}

and all agents

m

// Different initialization

3: for

t=1,2,\dots,T

4: for

m=1,2,\dots,M

5: Compute

\widehat{Q}_{t-\frac{1}{2}}^{m}

according to Eqn. (7)

6: Compute

\widehat{Q}_{t}^{m}

according to Eqn. (8)

7: end for

8: end for

Algorithm 4 Evolution of

\widehat{Q}

Lemma 2.

The following relation holds for all agents $m\in[M]$ , all $t\in[T]$ and $a\in\{1,2\}$ :

\displaystyle Q_{t}^{m}(1,a)-\widehat{Q}_{t}^{m}(a)\geq-\frac{1}{1-\gamma}\prod_{i=1}^{t}(1-\eta_{i}(1-\gamma)).

By Lemma 2, bounding the error of the sequence $\widehat{Q}_{t}^{m}$ allows us to obtain a bound on the error of $Q_{t}^{m}$ . To that effect, we define the following terms for any $t\leq T$ and all $m\in[M]$ :

	$\displaystyle\Delta_{t}^{m}(a)$	$\displaystyle:=\widehat{Q}_{t}^{m}(a)-Q^{\star}(1,a);\quad a=1,2;$
	$\displaystyle\Delta_{t,\max}^{m}$	$\displaystyle=\max_{a\in\{1,2\}}\Delta_{t}^{m}(a).$

In addition, we use $\overline{\Delta}_{t}=\frac{1}{M}\sum_{m=1}^{M}\Delta_{t}^{m}$ to denote the error of the averaged iterate¹¹1We use this different notation in appendix as opposed to the half-time indices used in the main text to improve readability of the proof., and similarly,

\displaystyle\overline{\Delta}_{t,\max}:=\max_{a\in\{1,2\}}\overline{\Delta}_{t}(a).

(37)

We first derive a basic recursion regarding $\Delta_{t}^{m}(a)$ . From the iterative update rule in Algorithm 4, we have,

	$\displaystyle\Delta_{t}^{m}(a)$	$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m}-Q^{\star}(1,a))$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m}-pV^{\star}(1))$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(p(\widehat{V}_{t-1}^{m}-V^{\star}(1))+(\widehat{P}_{t}^{m}(1\|1,a)-p)\widehat{V}_{t-1})$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(p\Delta_{t-1,\max}^{m}+(\widehat{P}_{t}^{m}(1\|1,a)-p)\widehat{V}_{t-1}^{m}).$

Here in the last line, we used the following relation:

\displaystyle\Delta_{t,\max}^{m}

\displaystyle=\max_{a\in\{1,2\}}(\widehat{Q}_{t}^{m}(a)-Q^{\star}(1,a))=\max_{a\in\{1,2\}}\widehat{Q}_{t}^{m}(a)-V^{\star}(1)=\widehat{V}_{t-1}^{m}-V^{\star}(1),

as $Q^{\star}(1,1)=Q^{\star}(1,2)=V^{\star}(1)$ .

Recursively unrolling the above expression, and using the expression (19), we obtain the following relation: for any $t^{\prime}<t$ when there is no averaging during the interval $(t^{\prime},t)$

\displaystyle\Delta_{t}^{m}(a)

\displaystyle=\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k})\right)\Delta_{t^{\prime}}^{m}(a)+\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma(p\Delta_{k-1,\max}^{m}+(\widehat{P}_{k}^{m}(1|1,a)-p)\widehat{V}_{k-1}^{m}).

(38)

For any $t^{\prime},t$ with $t^{\prime}<t$ , we define the notation

$\displaystyle\varphi_{t^{\prime},t}$	$\displaystyle:=\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}),$	(39)
$\displaystyle\xi_{t^{\prime},t}^{m}(a)$	$\displaystyle:=\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma(\widehat{P}_{k}^{m}(1\|1,a)-p)\widehat{V}_{k-1}^{m},\quad a=1,2;$	(40)
$\displaystyle\xi_{t^{\prime},t,\max}^{m}$	$\displaystyle:=\max_{a\in\{1,2\}}\xi_{t^{\prime},t}^{m}(a).$	(41)

Note that by definition, $\mathbb{E}[\xi_{t^{\prime},t}^{m}(a)]=0$ for $a\in\{1,2\}$ and all $m$ , $t^{\prime}$ and $t$ . Plugging them into the previous expression leads to the simplified expression

\displaystyle\Delta_{t}^{m}(a)

\displaystyle=\varphi_{t^{\prime},t}\Delta_{t^{\prime}}^{m}(a)+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]+\xi_{t^{\prime},t}^{m}(a).

We specifically choose $t^{\prime}$ and $t$ to be the consecutive averaging instants to analyze the behaviour of $\Delta_{t}^{m}$ across two averaging instants. Consequently, we can rewrite the above equation as

\displaystyle\Delta_{t}^{m}(a)

\displaystyle=\varphi_{t^{\prime},t}\overline{\Delta}_{t^{\prime}}(a)+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]+\xi_{t^{\prime},t}^{m}(a).

(42)

Furthermore, after averaging, we obtain,

\displaystyle\overline{\Delta}_{t}(a)

\displaystyle=\varphi_{t^{\prime},t}\overline{\Delta}_{t^{\prime}}(a)+\frac{1}{M}\sum_{m=1}^{M}\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]+\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(a).

(43)

Step 2: deriving a recursive bound for $\mathbb{E}[\overline{\Delta}_{t,\max}]$ .

Bounding (42), we obtain,


$\displaystyle\Delta_{t,\max}^{m}$	$\displaystyle\geq\varphi_{t^{\prime},t}\overline{\Delta}_{t^{\prime},\max}+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]+\xi_{t^{\prime},t,\max}^{m}-\varphi_{t^{\prime},t}\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|,$	(44a)
$\displaystyle\Delta_{t,\max}^{m}$	$\displaystyle\leq\varphi_{t^{\prime},t}\overline{\Delta}_{t^{\prime},\max}+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]+\xi_{t^{\prime},t,\max}^{m},$	(44b)

where in the first step we used the fact that

\displaystyle\max\{a_{1}+b_{1},a_{2}+b_{2}\}\geq\min\{a_{1},a_{2}\}+\max\{b_{1},b_{2}\}=\max\{a_{1},a_{2}\}+\max\{b_{1},b_{2}\}-|a_{1}-a_{2}|.

(45)

On taking expectation, we obtain,


$\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]$	$\displaystyle\geq\varphi_{t^{\prime},t}\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\mathbb{E}[\Delta_{k-1,\max}^{m}]\right]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]-\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|],$	(46a)
$\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]$	$\displaystyle\leq\varphi_{t^{\prime},t}\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\mathbb{E}[\Delta_{k-1,\max}^{m}]\right]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}].$	(46b)

Similarly, using (43) and (45) we can write,


$\displaystyle\overline{\Delta}_{t,\max}$	$\displaystyle\geq\varphi_{t^{\prime},t}\overline{\Delta}_{t^{\prime},\max}+\frac{1}{M}\sum_{m=1}^{M}\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\Delta_{k-1,\max}^{m}\right]-\varphi_{t^{\prime},t}\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}$	(47a)
$\displaystyle\implies\mathbb{E}[\overline{\Delta}_{t,\max}]$	$\displaystyle\geq\varphi_{t^{\prime},t}\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\frac{1}{M}\sum_{m=1}^{M}\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\mathbb{E}[\Delta_{k-1,\max}^{m}]\right]-\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|]$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\mathbb{E}\left[\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}\right].$	(47b)

On combining (46b) and (47b), we obtain,

	$\displaystyle\mathbb{E}[\overline{\Delta}_{t,\max}]$	$\displaystyle\geq\frac{1}{M}\sum_{m=1}^{M}\left[\mathbb{E}[\Delta_{t,\max}^{m}]-\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\right]-\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|]$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\mathbb{E}\left[\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}\right].$		(48)

In order to simplify (48), we make use of the following lemmas.

Lemma 3.

Let $t^{\prime}<t$ be two consecutive averaging instants. Then for all $m\in[M]$ ,

	$\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]-\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle\geq\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\right)\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\left[\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}-1\right]_{+}$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|],$

where $[x]_{+}=\max\{x,0\}$ .

Lemma 4.

For all consecutive averaging instants $t^{\prime},t$ satisfying $t-\max\{t^{\prime},\tau\}\geq 1/\eta_{\tau}$ and all $m\in[M]$ , we have,

	$\displaystyle\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle\geq\frac{1}{240\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)}\cdot\frac{\nu}{\nu+1},$
	$\displaystyle\mathbb{E}\left[\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}\right]$	$\displaystyle\geq\frac{1}{240\log\left(\frac{180BM}{\eta_{T}(1-\gamma)}\right)}\cdot\frac{\nu}{\nu+\sqrt{M}},$

where $\nu:=\sqrt{\dfrac{20\eta_{T}}{B(1-\gamma)}}$ .

Lemma 5.

For all $t\in\{t_{r}\}_{r=1}^{R}$ , we have

\displaystyle\mathbb{E}[|\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)|]\leq\sqrt{\frac{8\eta_{T}}{3BM(1-\gamma)}}.

Thus, on combining the results from Lemmas 3, 4, and 5 and plugging them into (48), we obtain the following relation for $t,t^{\prime}\geq\tau$ :

$\displaystyle\mathbb{E}[\overline{\Delta}_{t,\max}]$	$\displaystyle\geq\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\right)\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\left[\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}-1\right]_{+}-2\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|]$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\mathbb{E}\left[\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}\right]$
	$\displaystyle\geq(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\left(\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{5760\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(1-\gamma p)}\right)\cdot\frac{\nu}{\nu+1}\cdot\mathbbm{1}\left\{t-t^{\prime}\geq\frac{8}{\eta_{\tau}}\right\}$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}-2(1-\eta_{T})^{t-t^{\prime}}\sqrt{\frac{8\eta_{T}}{3BM(1-\gamma)}}+\frac{1}{240\log\left(\frac{180BM}{\eta_{T}(1-\gamma)}\right)}\cdot\frac{\nu}{\nu+\sqrt{M}}\cdot\mathbbm{1}\left\{t-t^{\prime}\geq\frac{8}{\eta_{\tau}}\right\},$	(49)

where we used the relation $\varphi_{t^{\prime},t}\leq(1-\eta_{T})^{t-t^{\prime}}$ , as well as the value of $\nu$ as defined in Lemma 4 along with the fact

\displaystyle\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}-1\geq\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{24(1-\gamma p)}

(50)

for all $t,t^{\prime}\geq\tau$ such that $t-t^{\prime}\geq 8/\eta_{\tau}$ .

Proof of (50).

We have,

$\displaystyle\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}-1$	$\displaystyle=\sum_{k=t^{\prime}+1}^{t}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i}(1-\gamma p))\right)-1$
	$\displaystyle\geq\sum_{k=t^{\prime}+1}^{t}\left(\eta_{t}\prod_{i=k+1}^{t}(1-\eta_{\tau}(1-\gamma p))\right)-1$
	$\displaystyle\geq\eta_{t}\sum_{k=t^{\prime}+1}^{t}(1-\eta_{\tau}(1-\gamma p))^{t-k}-1$
	$\displaystyle\geq\eta_{t}\cdot\left(\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{\eta_{\tau}(1-\gamma p)}\right)-1$
	$\displaystyle\geq\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{3(1-\gamma p)}-1.$	(51)

To show (50), it is sufficient to show that $\displaystyle\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{3(1-\gamma p)}\geq\frac{8}{7}$ for $t-t^{\prime}\geq 8/\eta_{\tau}$ . Thus, for $t-t^{\prime}\geq 8/\eta_{\tau}$ we have,

	$\displaystyle\frac{1-(1-\eta_{\tau}(1-\gamma p))^{t-t^{\prime}}}{3(1-\gamma p)}$	$\displaystyle\geq\frac{1-\exp(-\eta_{\tau}(1-\gamma p)\cdot(t-t^{\prime}))}{3(1-\gamma p)}$
		$\displaystyle\geq\frac{1-\exp(-8(1-\gamma p))}{3(1-\gamma p)}.$		(52)

Since $\gamma\geq 5/6$ , $1-\gamma p\leq 2/9$ . For $x\leq 2/9$ , the function $f(x)=\frac{1-e^{-8x}}{3x}\geq 8/7$ , proving the claim.

Step 3: lower bounding $\mathbb{E}[\overline{\Delta}_{T,\max}]$ .

We are now interested in evaluating $\mathbb{E}[\overline{\Delta}_{T,\max}]$ based on the recursion (49). To this effect, we introduce some notation to simplify the presentation. Let

\displaystyle R_{\tau}

\displaystyle:=\min\{r:t_{r}\geq\tau\}.

(53)

For $r=R_{\tau},\dots,R$ , we define the following terms:

	$\displaystyle x_{r}$	$\displaystyle:=\mathbb{E}[\overline{\Delta}_{t_{r},\max}],$
	$\displaystyle\alpha_{r}$	$\displaystyle:=(1-\eta_{\tau}(1-\gamma p))^{t_{r}-t_{r-1}},$
	$\displaystyle\beta_{r}$	$\displaystyle:=(1-\eta_{T})^{t_{r}-t_{r-1}},$
	$\displaystyle\mathcal{I}_{r}$	$\displaystyle:=\{r\geq r^{\prime}>R_{\tau}:t_{r^{\prime}}-t_{r^{\prime}-1}\geq 8/\eta_{\tau}\},$
	$\displaystyle C_{1}$	$\displaystyle:=\frac{1}{5760\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(1-\gamma p)}\cdot\frac{\nu}{\nu+1},$
	$\displaystyle C_{2}$	$\displaystyle:=\sqrt{\frac{32\eta_{T}}{3BM(1-\gamma)}},$
	$\displaystyle C_{3}$	$\displaystyle:=\frac{1}{240\log\left(\frac{180BM}{\eta_{T}(1-\gamma)}\right)}\cdot\frac{\nu}{\nu+\sqrt{M}}.$

With these notations in place, the recursion in (49) can be rewritten as

\displaystyle x_{r}\geq\alpha_{r}x_{r-1}-\beta_{r}C_{2}+C_{3}\mathbbm{1}\{r\in\mathcal{I}_{r}\}+(1-\alpha_{r})C_{1}\mathbbm{1}\{r\in\mathcal{I}_{r}\},

(54)

for all $r\geq R_{\tau}$ . We claim that $x_{r}$ satisfies the following relation for all $r\geq R_{\tau}+1$ (whose proof is deferred to the end of this step):

	$\displaystyle x_{r}$	$\displaystyle\geq\left(\prod_{i=R_{\tau}+1}^{r}\alpha_{i}\right)x_{R_{\tau}}-\sum_{k=R_{\tau}+1}^{r}\beta_{k}\left(\prod_{i=k+1}^{r}\alpha_{i}\right)C_{2}+\sum_{k=R_{\tau}+1}^{r}\left(\prod_{i=k+1}^{r}\alpha_{i}\right)\mathbbm{1}\{k\in\mathcal{I}_{k}\}C_{3}$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+C_{1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right),$		(55)

where we recall that if there is no valid index for a product, its value is taken to be $1$ .

Invoking (55) for $r=R$ and using the relation $x_{R_{\tau}-1}\geq 0$ , we obtain,

$\displaystyle x_{R}$	$\displaystyle\geq-\sum_{k=R_{\tau}}^{R}\beta_{k}\left(\prod_{i=k+1}^{R}\alpha_{i}\right)C_{2}+\sum_{k=R_{\tau}}^{R}\left(\prod_{i=k+1}^{R}\alpha_{i}\right)C_{3}\mathbbm{1}\{k\in\mathcal{I}_{k}\}+C_{1}\left(\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)$
	$\displaystyle\geq-RC_{2}+C_{1}\left(\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)$
	$\displaystyle\geq-R\cdot\sqrt{\frac{32\eta_{T}}{3BM(1-\gamma)}}+\left(\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)\cdot\frac{1}{5760\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(1-\gamma p)}\cdot\frac{\nu}{\nu+1},$	(56)

where we used the fact $\beta_{k}\left(\prod_{i=k+1}^{R}\alpha_{i}\right)\leq 1$ and that $C_{3}\geq 0$ . Consider the expression

\displaystyle\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}

\displaystyle=\prod_{i\notin\mathcal{I}_{R}}(1-\eta_{\tau}(1-\gamma p))^{t_{i}-t_{i-1}}\geq 1-\eta_{\tau}(1-\gamma p)\cdot\underbrace{\sum_{i\notin\mathcal{I}_{R}}(t_{i}-t_{i-1})}_{=:T_{1}}.

(57)

Consequently,

\displaystyle\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)=1-(1-\eta_{\tau}(1-\gamma p))^{T-\tau-T_{1}}\geq 1-\exp\left(-\eta_{\tau}(1-\gamma p)\left(T-\tau-T_{1}\right)\right).

(58)

Note that $T_{1}$ satisfies the following bound

\displaystyle T_{1}:=\sum_{i\notin\mathcal{I}_{R}}(t_{i}-t_{i-1})\leq(R-|\mathcal{I}_{R}|)\cdot\frac{8}{\eta_{\tau}}\leq\frac{8R}{\eta_{\tau}}.

(59)

We split the remainder of the analysis based on the step size schedule.

•

For the constant step size schedule, i.e., $\eta_{t}=\eta\geq\frac{1}{(1-\gamma)T}$ , we have, $R_{\tau}=0$ , with $\tau=0$ and $t_{0}=0$ (as all agents start at the same point). If $R\leq\frac{1}{96000(1-\gamma)\log\left(\frac{180B}{\eta(1-\gamma)}\right)}$ , then, (57), (58) and (59) yield the following relations:

	$\displaystyle T_{1}$	$\displaystyle\leq\frac{8R}{\eta}\leq\frac{T}{12000\log(180N)},$
	$\displaystyle\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}$	$\displaystyle\geq 1-\eta(1-\gamma p)\cdot T_{1}\geq 1-\frac{32R(1-\gamma)}{3}\geq 1-\frac{1}{9000\log(180N)},$
	$\displaystyle\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)$	$\displaystyle\geq 1-\exp\left(-\eta(1-\gamma p)\left(T-T_{1}\right)\right)\geq 1-\exp\left(-\frac{4}{3}\left(1-\frac{1}{9000\log(180N)}\right)\right).$

On plugging the above relations into (56), we obtain

\displaystyle x_{R}

\displaystyle\geq\frac{\sqrt{40}}{96000\log\left(\frac{180B}{\eta(1-\gamma)}\right)(1-\gamma)}\cdot\left(\frac{\nu}{\nu+1}-\frac{\nu}{5\sqrt{M}}\right)

(60)

where recall that $\nu:=\sqrt{\dfrac{20\eta}{3B(1-\gamma)}}$ . Consider the function $f(x)=\frac{x}{x+1}-\frac{x}{5\sqrt{M}}$ . We claim that for $x\in[0,\sqrt{M}]$ and all $M\geq 2$ ,

\displaystyle f(x)\geq\frac{7}{20}\min\{x,1\}.

(61)

The proof of the above claim is deferred to the end of the section. In light of the above claim, we have,

	$\displaystyle x_{R}$	$\displaystyle\geq\frac{\sqrt{40}}{96000\log\left(\frac{180B}{\eta(1-\gamma)}\right)(1-\gamma)}\cdot\frac{7}{20}\cdot\min\left\{1,\sqrt{\frac{20\eta}{3B(1-\gamma)}}\right\}$
		$\displaystyle\geq\frac{\sqrt{40}}{96000\log\left(180N\right)}\cdot\frac{7}{20}\cdot\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{20}{3(1-\gamma)^{4}N}}\right\},$		(62)

where we used the fact that $M\geq 2$ , $\frac{\sqrt{x}}{\log(1/x)}$ is an increasing function and the relation $\dfrac{\nu}{M}=\dfrac{20\eta}{3BM(1-\gamma)}\leq\dfrac{1}{15}\leq 1$ .

•

Next, we consider the rescaled linear step size schedule, where $\tau=T/3$ (cf. (36)). To begin, we assume $t_{R_{\tau}}\leq\max\{\frac{3T}{4},T-\frac{1}{6\eta_{\tau}(1-\gamma p)}\}$ . It is straightforward to note that

\displaystyle\max\left\{\frac{3T}{4},T-\frac{1}{6\eta_{\tau}(1-\gamma p)}\right\}=\begin{cases}\frac{3T}{4}&\text{ if }c_{\eta}\geq 3\\ T-\frac{1}{6\eta_{\tau}(1-\gamma p)}&\text{ if }c_{\eta}<3.\end{cases}

If $R\leq\frac{1}{384000(1-\gamma)\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)\cdot(5+c_{\eta})}$ then, (57), (58) and (59) yield the following relations:

\displaystyle T_{1}

\displaystyle\leq\frac{8R}{\eta_{\tau}},\qquad\prod_{i\notin\mathcal{I}_{R}}\alpha_{i}\geq 1-\eta_{\tau}(1-\gamma p)\cdot T_{1}\geq 1-\frac{32R(1-\gamma)}{3}\geq 1-\frac{1}{36000}.

For $c_{\eta}\geq 3$ , we have,

	$\displaystyle\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)$	$\displaystyle\geq 1-\exp\left(-\eta_{\tau}(1-\gamma p)\left(T-t_{R_{\tau}}-T_{1}\right)\right)$
		$\displaystyle\geq 1-\exp\left(-\frac{(1-\gamma)T}{(3+c_{\eta}(1-\gamma)T)}+\frac{32R(1-\gamma)}{3}\right)$
		$\displaystyle\geq\frac{1}{2(3+c_{\eta})},$

where we used $T\geq\frac{1}{1-\gamma}$ in the second step. Similarly, for $c_{\eta}<3$ , we have,

	$\displaystyle\left(1-\prod_{i\in\mathcal{I}_{R}}\alpha_{i}\right)$	$\displaystyle\geq 1-\exp\left(-\eta_{\tau}(1-\gamma p)\left(T-t_{R_{\tau}}-T_{1}\right)\right)$
		$\displaystyle\geq 1-\exp\left(-\frac{1}{6}+\frac{32R(1-\gamma)}{3}\right)$
		$\displaystyle\geq\frac{1}{10}.$

On plugging the above relations into (56), we obtain

$\displaystyle x_{R}$	$\displaystyle\geq\frac{18\sqrt{1.6}}{384000\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(1-\gamma)(5+c_{\eta})}\cdot\left(\frac{\nu}{\nu+1}-\frac{\nu}{18{\sqrt{M}}}\right)$
	$\displaystyle\geq\frac{18\sqrt{1.6}}{384000\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(1-\gamma)(5+c_{\eta})}\cdot\frac{7}{20}\cdot\min\left\{1,\sqrt{\frac{20\eta_{T}}{3B(1-\gamma)}}\right\}$
	$\displaystyle\geq\frac{18\sqrt{1.6}}{384000\log\left(\frac{180B}{\eta_{T}(1-\gamma)}\right)(5+c_{\eta})}\cdot\frac{7}{20}\cdot\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{20\eta_{T}}{3B(1-\gamma)^{3}}}\right\}$
	$\displaystyle\geq\frac{18\sqrt{1.6}}{384000\log\left(180N(1+\log N)\right)(5+\log N)}\cdot\frac{7}{20}\cdot\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{20}{3B(1+\log N)(1-\gamma)^{4}N}}\right\},$	(63)

where we again used the facts that $M\geq 2$ , $c_{\eta}\leq\log N$ , $\frac{\sqrt{x}}{\log(1/x)}$ is an increasing function and the relation $\dfrac{\nu}{M}=\dfrac{20\eta_{T}}{3BM(1-\gamma)}\leq 1$ .

•

Last but not least, let us consider the rescaled linear step size schedule case when $t_{R_{\tau}}>\max\{\frac{3T}{4},T-\frac{1}{6\eta_{\tau}(1-\gamma p)}\}$ . The condition implies that the time between the communication rounds $R_{\tau}-1$ and $R_{\tau}$ is at least $T_{0}:=\max\{\frac{5T}{12},\frac{2T}{3}-\frac{1}{6\eta_{\tau}(1-\gamma p)}\}$ . Thus, (49) yields that

\displaystyle\mathbb{E}[\overline{\Delta}_{t_{R_{\tau}}}]

\displaystyle\geq\left(\frac{1-(1-\eta_{\tau}(1-\gamma p))^{T_{0}}}{5760\log\left(\frac{180}{B\eta_{T}(1-\gamma)}\right)(1-\gamma p)}\right)\cdot\frac{\nu}{\nu+1}-2(1-\eta_{T})^{T_{0}}\sqrt{\frac{8\eta_{T}}{3BM(1-\gamma)}}.

(64)

Using the above relation along with (55), we can conclude that

	$\displaystyle x_{R}$	$\displaystyle\geq(1-\eta_{\tau}(1-\gamma p))^{T-t_{R_{\tau}}}\left(\frac{1-(1-\eta_{\tau}(1-\gamma p))^{T_{0}}}{5760\log\left(\frac{180}{B\eta_{T}(1-\gamma)}\right)(1-\gamma p)}\right)\cdot\frac{\nu}{\nu+1}$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-2(1-\eta_{T})^{T_{0}}\cdot(1-\eta_{\tau}(1-\gamma p))^{T-t_{R_{\tau}}}\sqrt{\frac{8\eta_{T}}{3BM(1-\gamma)}}-RC_{2}.$		(65)

In the above relation, we used the trivial bounds $C_{1},C_{3}\geq 0$ and a crude bound on the term corresponding to $C_{2}$ , similar to (56). Let us first consider the case of $c_{\eta}\geq 3$ . We have,

	$\displaystyle 1-(1-\eta_{\tau}(1-\gamma p))^{T_{0}}$	$\displaystyle\geq 1-\exp\left(-\eta_{\tau}(1-\gamma p)5T/12\right)\geq 1-\exp\left(-\frac{5(1-\gamma)T}{3(3+c_{\eta}(1-\gamma)T)}\right)\geq\frac{1}{3+c_{\eta}},$
	$\displaystyle(1-\eta_{\tau}(1-\gamma p))^{T-t_{R_{\tau}}}$	$\displaystyle\geq 1-\eta_{\tau}(1-\gamma p)\frac{T}{4}\geq 1-\frac{(1-\gamma)T}{(3+c_{\eta}(1-\gamma)T)}\geq 1-\frac{1}{c_{\eta}}\geq\frac{2}{3}.$

Similarly, for $c_{\eta}<3$ , we have,

	$\displaystyle 1-(1-\eta_{\tau}(1-\gamma p))^{T_{0}}$	$\displaystyle\geq 1-\exp\left(-\eta_{\tau}(1-\gamma p)\frac{2T}{3}+\frac{1}{6}\right)$
		$\displaystyle\geq 1-\exp\left(-\frac{8(1-\gamma)T}{3(3+c_{\eta}(1-\gamma)T)}+\frac{1}{6}\right)\geq 1-e^{-5/18},$
	$\displaystyle(1-\eta_{\tau}(1-\gamma p))^{T-t_{R_{\tau}}}$	$\displaystyle\geq 1-\frac{\eta_{\tau}(1-\gamma p)}{6\eta_{\tau}(1-\gamma p)}\geq\frac{5}{6}.$

The above relations implies that $(1-\eta_{\tau}(1-\gamma p))^{T-t_{R_{\tau}}}(1-(1-\eta_{\tau}(1-\gamma p))^{T_{0}})\geq c$ for some constant $c$ , which only depends on $c_{\eta}$ . On plugging this into (65), we obtain a relation that is identical to that in (56) up to leading constants. Thus, by using a similar sequence of argument as used to obtain (63), we arrive at the same conclusion as for the case of $t_{R_{\tau}}\leq\max\{\frac{3T}{4},T-\frac{1}{6\eta_{\tau}(1-\gamma p)}\}$ .

Step 4: finishing up the proof.

Thus, (62), (63) along with the above conclusion together imply that there exists a numerical constant $c_{0}>0$ such that

\displaystyle\mathbb{E}[|\widehat{V}_{T}^{m}(1)-V^{\star}(1)|]\geq\mathbb{E}[\overline{\Delta}_{T,\max}]\geq\frac{c_{0}}{\log^{3}N}\cdot\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{1}{(1-\gamma)^{4}N}}\right\}.

(66)

The above equation along with Lemma 2 implies

\displaystyle\mathbb{E}[|V_{T}^{m}-V^{\star}(1)|]\geq\frac{c_{0}}{\log^{3}N}\cdot\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{1}{(1-\gamma)^{4}N}}\right\}-\frac{1}{1-\gamma}\prod_{i=1}^{T}(1-\eta_{i}(1-\gamma)).

(67)

On the other hand, from (23) we know that

\displaystyle\mathbb{E}[|V_{T}^{m}(3)-V^{\star}(3)|]\geq\frac{1}{1-\gamma}\prod_{i=1}^{T}(1-\eta_{i}(1-\gamma)).

(68)

Hence,

$\displaystyle\mathbb{E}[\\|Q_{T}^{m}-Q^{\star}\\|_{\infty}]$	$\displaystyle\geq\mathbb{E}\left[\max\left\{\|V_{T}^{m}(3)-V^{\star}(3)\|,\|V_{T}^{m}(1)-V^{\star}(1)\|\right\}\right]$
	$\displaystyle\geq\max\left\{\mathbb{E}\left[\|V_{T}^{m}(3)-V^{\star}(3)\|\right],\mathbb{E}\left[\|V_{T}^{m}(1)-V^{\star}(1)\|\right]\right\}$
	$\displaystyle\geq\max\left\{\frac{1}{1-\gamma}\prod_{i=1}^{T}(1-\eta_{i}(1-\gamma)),\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{1}{(1-\gamma)^{4}N}}\right\}-\frac{1}{1-\gamma}\prod_{i=1}^{T}(1-\eta_{i}(1-\gamma))\right\}$
	$\displaystyle\geq\frac{1}{2}\min\left\{\frac{1}{1-\gamma},\sqrt{\frac{1}{(1-\gamma)^{4}N}}\right\},$	(69)

where the third step follows from (67) and (68) and the fourth step uses $\max\{a,b\}\geq(a+b)/2$ .

Thus, from (30) and (69) we can conclude that whenever $\textsf{CC}_{\textsf{round}}=\mathcal{O}\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ , $\textsf{ER}(\mathscr{A};N,M)=\Omega\left(\frac{1}{\log^{3}N\sqrt{N}}\right)$ for all values of $M\geq 2$ . In other words, for any algorithm to achieve any collaborative gain, its communication complexity should satisfy $\textsf{CC}_{\textsf{round}}=\Omega\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ , as required.

Proof of (55).

We now return to establish (55) using induction. For the base case, (54) yields

\displaystyle x_{R_{\tau}+1}

\displaystyle\geq\alpha_{R_{\tau}+1}x_{R_{\tau}}-\beta_{R_{\tau}+1}C_{2}+C_{3}\mathbbm{1}\{R_{\tau}+1\in\mathcal{I}_{R_{\tau}+1}\}+(1-\alpha_{R_{\tau}+1})C_{1}\mathbbm{1}\{{R_{\tau}+1}\in\mathcal{I}_{R_{\tau}+1}\}.

(70)

Note that this is identical to the expression in (55) for $r=R_{\tau}+1$ as

\displaystyle\left(\prod_{i\notin\mathcal{I}_{R_{\tau}+1}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{R_{\tau}+1}}\alpha_{i}\right)=(1-\alpha_{R_{\tau}+1})\mathbbm{1}\{{R_{\tau}+1}\in\mathcal{I}_{R_{\tau}+1}\}

based on the adopted convention for products with no valid indices. For the induction step, assume (55) holds for some $r\geq R_{\tau}+1$ . On combining (54) and (55), we obtain,

$\displaystyle x_{r+1}$	$\displaystyle\geq\alpha_{r+1}x_{r}-\beta_{r+1}C_{2}+C_{3}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}+(1-\alpha_{r+1})C_{1}\mathbbm{1}\{r+1\in\mathcal{I}_{r+1}\}$
	$\displaystyle\geq\alpha_{r+1}\left(\prod_{i=R_{\tau}+1}^{r}\alpha_{i}\right)x_{R_{\tau}}-\alpha_{r+1}\sum_{k=R_{\tau}+1}^{r}\beta_{k}\left(\prod_{i=k+1}^{r}\alpha_{i}\right)C_{2}+\alpha_{r+1}\sum_{k=R_{\tau}+1}^{r}\left(\prod_{i=k+1}^{r}\alpha_{i}\right)C_{3}\mathbbm{1}\{k\in\mathcal{I}_{k}\}$
	$\displaystyle+\alpha_{r+1}C_{1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)-\beta_{r+1}C_{2}+C_{3}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}+(1-\alpha_{r+1})C_{1}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}$
	$\displaystyle\geq\left(\prod_{i=R_{\tau}+1}^{r+1}\alpha_{i}\right)x_{R_{\tau}}-\sum_{k=R_{\tau}+1}^{r+1}\beta_{k}\left(\prod_{i=k+1}^{r+1}\alpha_{i}\right)C_{2}+\sum_{k=R_{\tau}+1}^{r+1}\left(\prod_{i=k+1}^{r+1}\alpha_{i}\right)C_{3}\mathbbm{1}\{k\in\mathcal{I}_{k}\}$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}+\alpha_{r+1}C_{1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)+(1-\alpha_{r+1})C_{1}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}.$	(71)

If $(r+1)\notin\mathcal{I}_{r+1}$ , then $\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)=\left(1-\prod_{i\in\mathcal{I}_{r+1}}\alpha_{i}\right)$ and $\alpha_{r+1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)=\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)$ . Consequently,

\displaystyle\alpha_{r+1}C_{1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)+(1-\alpha_{r+1})C_{1}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}=C_{1}\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r+1}}\alpha_{i}\right).

(72)

On the other hand, if $(r+1)\in\mathcal{I}_{r+1}$ , then $\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)=\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)$ . Consequently, we have,

	$\displaystyle\alpha_{r+1}C_{1}\left(\prod_{i\notin\mathcal{I}_{r}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)+(1-\alpha_{r+1})C_{1}\mathbbm{1}\{(r+1)\in\mathcal{I}_{r+1}\}$
	$\displaystyle=\alpha_{r+1}C_{1}\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)+(1-\alpha_{r+1})C_{1}$
	$\displaystyle\geq C_{1}\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)\left[\alpha_{r+1}\left(1-\prod_{i\in\mathcal{I}_{r}}\alpha_{i}\right)+(1-\alpha_{r+1})\right]$
	$\displaystyle\geq C_{1}\left(\prod_{i\notin\mathcal{I}_{r+1}}\alpha_{i}\right)\left(1-\prod_{i\in\mathcal{I}_{r+1}}\alpha_{i}\right).$		(73)

Combining (71), (72) and (73) proves the claim.

Proof of (61).

To establish this result, we separately consider the cases $x\leq 1$ and $x\geq 1$ .

•

When $x\leq 1$ , we have

$\displaystyle f(x)=\frac{x}{x+1}-\frac{1}{5\sqrt{M}}\geq x\cdot\left(\frac{1}{2}-\frac{x}{5\sqrt{M}}\right)\geq\frac{7x}{20},$ (74)

where in the last step, we used the relation $M\geq 2$ .
•

Let us now consider the case $x\geq 1$ . The second derivative of $f$ is given by $f^{\prime\prime}(x)=-\frac{1}{2(x+1)^{3}}$ . Clearly, for all $x\geq 1$ , $f^{\prime\prime}<0$ implying that $f$ is a concave function. It is well-known that a continuous, bounded, concave function achieves its minimum values over a compact interval at the end points of the interval (Bauer’s minimum principle). For all $M\geq 2$ , we have,

$\displaystyle f(1)=\frac{1}{2}-\frac{1}{5\sqrt{M}}\geq\frac{7}{20};\quad f(\sqrt{M})=\frac{\sqrt{M}}{\sqrt{M}+1}-\frac{1}{5}\geq\frac{7}{20}.$

Consequently, we can conclude that for all $x\in[1,\sqrt{M}]$ ,

$\displaystyle f(x)\geq\frac{7}{20}.$ (75)

Combining (74) and (75) proves the claim.

A.3.3 Large learning rates with large $\frac{\eta_{T}}{BM}$

In order to bound the error in this scenario, note that $\frac{\eta_{T}}{BM}$ controls the variance of the stochastic updates in the fixed point iteration. Thus, when $\frac{\eta_{T}}{BM}$ is large, the variance of the iterates is large, resulting in a large error. To demonstrate this effect, we focus on the dynamics of state $2$ . This part of the proof is similar to the large learning rate case of Li2023QLMinimax. For all $t\in[T]$ , define:

\displaystyle\overline{V}_{t}(2):=\frac{1}{M}\sum_{m=1}^{M}V_{t}^{m}(2).

(76)

Thus, from (35), we know that $\mathbb{E}[\overline{V}_{t}(2)]$ obeys the following recursion:

\displaystyle\mathbb{E}[\overline{V}_{t}(2)]=(1-\eta_{t}(1-\gamma p))\mathbb{E}[\overline{V}_{t-1}(2)]+\eta_{t}.

Upon unrolling the recursion, we obtain,

\displaystyle\mathbb{E}[\overline{V}_{T}(2)]=\left(\prod_{k=t+1}^{T}(1-\eta_{k}(1-\gamma p))\right)\mathbb{E}[\overline{V}_{t}(2)]+\sum_{k=t+1}^{T}\eta_{k}^{(T)}.

Thus, the above relation along with (18) and the value of $V^{\star}(2)$ yields us,

\displaystyle V^{\star}(2)-\mathbb{E}[\overline{V}_{T}(2)]=\prod_{k=t+1}^{T}(1-\eta_{k}(1-\gamma p))\left(\frac{1}{1-\gamma p}-\mathbb{E}[\overline{V}_{t}(2)]\right).

(77)

Similar to Li2023QLMinimax, we define

\displaystyle\tau^{\prime}:=\min\left\{0\leq t^{\prime}\leq T-2\ \bigg{|}\ \mathbb{E}[(\overline{V}_{t})^{2}]\geq\frac{1}{4(1-\gamma)^{2}}\text{ for all }t^{\prime}+1\leq t\leq T\right\}.

If such a $\tau^{\prime}$ does not exist, it implies that either $\mathbb{E}[(\overline{V}_{T})^{2}]<\frac{1}{4(1-\gamma)^{2}}$ or $\mathbb{E}[(\overline{V}_{T-1})^{2}]<\frac{1}{4(1-\gamma)^{2}}$ . If the former is true, then,

\displaystyle V^{\star}(2)-\mathbb{E}[\overline{V}_{T}(2)]=\frac{3}{4(1-\gamma)}-\sqrt{\mathbb{E}[(\overline{V}_{T})^{2}]}>\frac{1}{4(1-\gamma)}.

(78)

Similarly, if $\mathbb{E}[(\overline{V}_{T-1})^{2}]<\frac{1}{4(1-\gamma)^{2}}$ , it implies $\mathbb{E}[\overline{V}_{T-1}]<\frac{1}{2(1-\gamma)}$ . Using (35), we have,

\displaystyle\mathbb{E}[\overline{V}_{T}(2)]=(1-\eta_{T}(1-\gamma p))\mathbb{E}[\overline{V}_{T-1}(2)]+\eta_{T}\leq\mathbb{E}[\overline{V}_{T-1}(2)]+1<\frac{1}{2(1-\gamma)}+\frac{1}{6(1-\gamma)}=\frac{2}{3(1-\gamma)}.

Consequently,

\displaystyle V^{\star}(2)-\mathbb{E}[\overline{V}_{T}(2)]>\frac{3}{4(1-\gamma)}-\frac{2}{3(1-\gamma)}>\frac{1}{12(1-\gamma)}.

(79)

For the case when $\tau^{\prime}$ exists, we divide the proof into two cases.

•

We first consider the case when the learning rates satisfy:

\displaystyle\prod_{k=\tau^{\prime}+1}^{T}(1-\eta_{k}(1-\gamma p))\geq\frac{1}{2}.

(80)

The analysis for this case is identical to that considered in Li2023QLMinimax. We explicitly write the steps for completeness. Specifically,

$\displaystyle V^{\star}(2)-\mathbb{E}[\overline{V}_{T}(2)]$	$\displaystyle=\left(\prod_{k=\tau^{\prime}+1}^{T}(1-\eta_{k}(1-\gamma p))\right)\left(\frac{1}{1-\gamma p}-\mathbb{E}[\overline{V}_{\tau^{\prime}}(2)]\right)$
	$\displaystyle\geq\frac{1}{2}\cdot\left(\frac{3}{4(1-\gamma)}-\sqrt{\mathbb{E}[(\overline{V}_{\tau^{\prime}}(2))^{2}]}\right)$
	$\displaystyle\geq\frac{1}{2}\cdot\left(\frac{3}{4(1-\gamma)}-\frac{1}{2(1-\gamma)}\right)\geq\frac{1}{8(1-\gamma)},$	(81)

where the first line follows from (77), the second line from the condition on step sizes and the third line from the definition of $\tau^{\prime}$ .

•

We now consider the other case where,

\displaystyle 0\leq\prod_{k=\tau^{\prime}+1}^{T}(1-\eta_{k}(1-\gamma p))<\frac{1}{2}.

(82)

Using (Li2023QLMinimax, Eqn.(134)), for any $t^{\prime}<t$ and all agents $m$ , we have the relation

\displaystyle V_{t}^{m}(2)=\frac{1}{1-\gamma p}-\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\left(\frac{1}{1-\gamma p}-V_{t^{\prime}}^{m}(2)\right)+\sum_{k=t^{\prime}+1}\eta_{k}^{(t)}\gamma(\hat{P}_{k}^{m}(2|2)-p)V_{k-1}^{m}(2).

The above equation is directly obtained by unrolling the recursion in (26) along with noting that $Q_{t}(2,1)=V_{t}(2)$ for all $t$ . Consequently, we have,

\displaystyle\overline{V}_{T}(2)=\frac{1}{1-\gamma p}-\prod_{k=t^{\prime}+1}^{T}(1-\eta_{k}(1-\gamma p))\left(\frac{1}{1-\gamma p}-\overline{V}_{t^{\prime}}(2)\right)+\frac{1}{M}\sum_{m=1}^{M}\sum_{k=t^{\prime}+1}^{T}\eta_{k}^{(T)}\gamma(\hat{P}_{k}^{m}(2|2)-p)V_{k-1}^{m}(2).

(83)

Let $\{\mathscr{F}_{t}\}_{t=0}^{T}$ be a filtration such that $\mathscr{F}_{t}$ is the $\sigma$ -algebra corresponding to $\{\{\hat{P}^{m}_{s}(2|2)\}_{m=1}^{M}\}_{s=1}^{t}$ . It is straightforward to note that $\left\{\frac{1}{M}\sum_{m=1}^{M}\eta_{k}^{(T)}\gamma(\hat{P}_{k}^{m}(2|2)-p)V_{k-1}^{m}(2)\right\}_{k}$ is a martingale sequence adapted to the filtration $\mathscr{F}_{k}$ . Thus, using the result from (Li2023QLMinimax, Eqn.(139)), we can conclude that

\displaystyle\text{Var}(\overline{V}_{T}(2))\geq\mathbb{E}\left[\sum_{k=\tau^{\prime}+2}^{T}\text{Var}\left(\frac{1}{M}\sum_{m=1}^{M}\eta_{k}^{(T)}\gamma(\hat{P}_{k}^{m}(2|2)-p)V_{k-1}^{m}(2)\ \bigg{|}\mathscr{F}_{k-1}\right)\right].

(84)

We have,

$\displaystyle\text{Var}\left(\frac{1}{M}\sum_{m=1}^{M}\eta_{k}^{(T)}\gamma(\hat{P}_{k}^{m}(2\|2)-p)V_{k-1}^{m}(2)\ \bigg{\|}\mathscr{F}_{k-1}\right)$	$\displaystyle=\frac{1}{M^{2}}\sum_{m=1}^{M}\text{Var}\left(\eta_{k}^{(T)}\gamma(\hat{P}_{k}^{m}(2\|2)-p)V_{k-1}^{m}(2)\ \bigg{\|}\mathscr{F}_{k-1}\right)$
	$\displaystyle=\frac{(\eta_{k}^{(T)})^{2}}{BM}\gamma^{2}p(1-p)\left(\frac{1}{M}\sum_{m=1}^{M}(V_{k-1}^{m}(2))^{2}\right)$
	$\displaystyle\geq\frac{(1-\gamma)(4\gamma-1)}{9BM}\cdot(\eta_{k}^{(T)})^{2}\cdot(\overline{V}_{k-1}(2))^{2},$	(85)

where the first line follows from that fact that variance of sum of i.i.d. random variables is the sum of their variances, the second line from variance of Binomial random variable and the third line from Jensen’s inequality. Thus, (84) and (85) together yield,

	$\displaystyle\text{Var}(\overline{V}_{T}(2))\geq\frac{(1-\gamma)(4\gamma-1)}{9BM}\cdot\sum_{k=\tau^{\prime}+2}^{T}(\eta_{k}^{(T)})^{2}\cdot\mathbb{E}[(\overline{V}_{k-1}(2))^{2}]$
	$\displaystyle\geq\frac{(1-\gamma)(4\gamma-1)}{9BM}\cdot\frac{1}{4(1-\gamma)^{2}}\cdot\sum_{k=\max\{\tau,\tau^{\prime}\}+2}^{T}(\eta_{k}^{(T)})^{2},$		(86)

where the second line follows from the definition of $\tau^{\prime}$ . We focus on bounding the third term in the above relation. We have,

$\displaystyle\sum_{k=\max\{\tau^{\prime},\tau\}+2}^{T}\left(\eta_{k}^{(T)}\right)^{2}$	$\displaystyle\geq\sum_{k=\max\{\tau^{\prime},\tau\}+2}^{T}\left(\eta_{k}\prod_{i=k+1}^{T}(1-\eta_{i}(1-\gamma p)\right)^{2}$
	$\displaystyle\geq\sum_{k=\max\{\tau^{\prime},\tau\}+2}^{T}\left(\eta_{T}\prod_{i=k+1}^{t}(1-\eta_{\tau}(1-\gamma p))\right)^{2}$
	$\displaystyle=\eta_{T}^{2}\sum_{k=\max\{\tau^{\prime},\tau\}+2}^{T}(1-\eta_{\tau}(1-\gamma p))^{2(t-k)}$
	$\displaystyle\geq\eta_{T}^{2}\cdot\frac{1-(1-\eta_{\tau}(1-\gamma p))^{2(T-\max\{\tau^{\prime},\tau\}-1)}}{\eta_{\tau}(1-\gamma p)(2-\eta_{\tau}(1-\gamma p))}$
	$\displaystyle\geq\eta_{T}\cdot\frac{1}{4(1-\gamma)}\cdot c^{\prime},$	(87)

where the second line follows from monotonicity of $\eta_{t}$ and the numerical constant $c^{\prime}$ in the fifth step is given by the following claim whose proof is deferred to the end of the section:

\displaystyle 1-(1-\eta_{\tau}(1-\gamma p))^{2(T-\max\{\tau^{\prime},\tau\}-1)}\geq\begin{cases}1-e^{-8/9}&\text{ for constant step sizes},\\ 1-\exp\left(-\frac{8}{3\max\{1,c_{\eta}\}}\right)&\text{ for linearly rescaled step sizes}\end{cases}.

(88)

Thus, (86) and (87) together imply

	$\displaystyle\text{Var}(\overline{V}_{T}(2))$	$\displaystyle\geq\frac{(4\gamma-1)}{36BM(1-\gamma)}\cdot\sum_{k=\tau^{\prime}+2}^{T}(\eta_{k}^{(T)})^{2}$
		$\displaystyle\geq\frac{c^{\prime}(4\gamma-1)}{144(1-\gamma)}\cdot\frac{\eta_{T}}{BM(1-\gamma)}\geq\frac{c^{\prime}(4\gamma-1)}{144(1-\gamma)}\cdot\frac{1}{100},$		(89)

where the last inequality follows from the bound on $\frac{\eta_{T}}{BM}$ .

Thus, for all $N\geq 1$ , we have,

\displaystyle\mathbb{E}[(V^{\star}(2)-\overline{V}_{T}(2))^{2}]=\mathbb{E}[(V^{\star}(2)-\mathbb{E}[\overline{V}_{T}(2)])^{2}]+\text{Var}(\overline{V}_{T}(2))\geq\frac{c^{\prime\prime}}{(1-\gamma)N},

for some numerical constant $c^{\prime\prime}$ . Similar to the small learning rate case, the error rate is bounded away from a constant value irrespective of the number of agents and the number of communication rounds. Thus, even with $\textsf{CC}_{\textsf{round}}=\Omega(T)$ , we will not observe any collaborative gain in this scenario.

Proof of (88).

To establish the claim, we consider two cases:

•

$\tau^{\prime}\geq\tau$ : Under this case, we have,

	$\displaystyle(1-\eta_{\tau}(1-\gamma p))^{2(T-\max\{\tau^{\prime},\tau\}-1)}$	$\displaystyle=(1-\eta_{\tau}(1-\gamma p))^{2(T-\tau^{\prime}-1)}$
		$\displaystyle\leq(1-\eta_{\tau}(1-\gamma p))^{T-\tau^{\prime}}\leq\prod_{k=\tau^{\prime}+1}^{T}(1-\eta_{k}(1-\gamma p))\leq\frac{1}{2},$		(90)

where the last inequality follows from (82).

•

$\tau\geq\tau^{\prime}$ : For this case, we have

	$\displaystyle(1-\eta_{\tau}(1-\gamma p))^{2(T-\max\{\tau^{\prime},\tau\}-1)}$	$\displaystyle=(1-\eta_{\tau}(1-\gamma p))^{2(T-\tau-1)}$
		$\displaystyle\leq(1-\eta_{\tau}(1-\gamma p))^{T-\tau}\leq\exp\left(-\frac{2T\eta_{\tau}(1-\gamma p)}{3}\right).$		(91)

For the constant stepsize schedule, we have,

\displaystyle\exp\left(-\frac{2T\eta_{\tau}(1-\gamma p)}{3}\right)\leq\exp\left(-\frac{2T}{3}\cdot\frac{1}{(1-\gamma)T}\cdot\frac{4(1-\gamma)}{3}\right)=\exp\left(-\frac{8}{9}\right)

(92)

For linearly rescaled stepsize schedule, we have,

\displaystyle\exp\left(-\frac{2T\eta_{\tau}(1-\gamma p)}{3}\right)\leq\exp\left(-\frac{2T}{3}\cdot\frac{1}{1+c_{\eta}(1-\gamma)T/3}\cdot\frac{4(1-\gamma)}{3}\right)=\exp\left(-\frac{8}{3\max\{1,c_{\eta}\}}\right)

(93)

On combining (90), (91), (92) and (93), we arrive at the claim.

A.4 Generalizing to larger state action spaces

We now elaborate on how we can extend the result to general state-action spaces along with the obtaining the lower bound on the bit level communication complexity. For the general case, we instead consider the following MDP. For the first four states $\{0,1,2,3\}$ , the probability transition kernel and reward function are given as follows.


$\displaystyle\mathcal{A}_{0}=\{1\}$	$\displaystyle\quad\quad P(0\|0,1)=1$		$\displaystyle\quad r(0,1)=0,$	(94a)
$\displaystyle\mathcal{A}_{1}=\{1,2,\dots,\|\mathcal{A}\|\}$	$\displaystyle\quad\quad P(1\|1,a)=p$	$\displaystyle\quad P(0\|1,a)=1-p$	$\displaystyle\quad r(1,a)=1,\forall\ a\in\mathcal{A}$	(94b)
$\displaystyle\mathcal{A}_{2}=\{1\}$	$\displaystyle\quad\quad P(2\|2,1)=p$	$\displaystyle\quad P(0\|2,1)=1-p$	$\displaystyle\quad r(2,1)=1,$	(94c)
$\displaystyle\mathcal{A}_{3}=\{1\}$	$\displaystyle\quad\quad P(3\|3,1)=1$		$\displaystyle\quad r(3,1)=1,$	(94d)

where the parameter $p=\dfrac{4\gamma-1}{3\gamma}$ . The overall MDP is obtained by creating $|\mathcal{S}|/4$ copies of the above MDP for all sets of the form $\{4r,4r+1,4r+2,4r+3\}$ for $r\leq|\mathcal{S}|/4-1$ . It is straightforward to note that the lower bound on the number of communication rounds immediately transfers to the general case as well. Moreover, note that the bound on $\textsf{CC}_{\textsf{round}}$ implies the bound $\textsf{CC}_{\textsf{bit}}=\Omega\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ as every communication entails sending $\Omega(1)$ bits.

To obtain the general lower bound on bit level communication complexity, note that we can carry out the analysis in the previous section for all $|\mathcal{A}|/2$ pairs of actions in state $1$ corresponding to the set of states $\{0,1,2,3\}$ . Moreover, the algorithm $\mathscr{A}$ , needs to ensure that the error is low across all the $|\mathcal{A}|/2$ pairs. Since the errors are independent across all these pairs, each of them require $\Omega\left(\frac{1}{(1-\gamma)\log^{2}N}\right)$ bits of information to be transmitted during the learning horizon leading to a lower bound of $\Omega\left(\frac{|\mathcal{A}|}{(1-\gamma)\log^{2}N}\right)$ . Note that since we require a low $\ell_{\infty}$ error, $\mathscr{A}$ needs to ensure that the error is low across all the pairs, resulting in a communication cost linearly growing with $|\mathcal{A}|$ . Upon repeating the argument across all $|\mathcal{S}|/4$ copies of the MDP, we arrive at the lower bound of $\textsf{CC}_{\textsf{bit}}=\Omega\left(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)\log^{2}N}\right)$ .

A.5 Proofs of auxiliary lemmas

A.5.1 Proof of Lemma 2

Note that a similar relationship is also derived in Li2023QLMinimax, but needing to take care of the averaging over multiple agents, we present the entire arguments for completeness. We prove the claim using an induction over $t$ . It is straightforward to note that the claim is true for $t=0$ and all agents $m\in\{1,2,\dots,M\}$ . For the inductive step, we assume that the claim holds for $t-1$ for all clients. Using the induction hypothesis, we have the following relation between $V_{t-1}^{m}(1)$ and $\widehat{V}_{t-1}^{m}$ :

\displaystyle V_{t-1}^{m}(1)=\max_{a\in\{1,2\}}Q_{t-1}^{m}(1,a)\geq\max_{a\in\{1,2\}}\widehat{Q}_{t-1}^{m}(a)-\frac{1}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma))=\widehat{V}_{t-1}^{m}-\frac{1}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma)).

(95)

For $t\notin\{t_{r}\}_{r=1}^{R}$ and $a\in\{1,2\}$ , we have,

$\displaystyle Q_{t}^{m}(1,a)-\widehat{Q}_{t}^{m}(a)$	$\displaystyle=Q_{t-1/2}^{m}(1,a)-\widehat{Q}_{t-1/2}^{m}(a)$
	$\displaystyle=(1-\eta_{t})Q_{t-1}^{m}(1,a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)V_{t-1}^{m}(1))$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\left[(1-\eta_{t})\widehat{Q}_{t-1}^{m}(a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m})\right]$
	$\displaystyle=(1-\eta_{t})(Q_{t-1}^{m}(1\|1,a)-\widehat{Q}_{t-1}^{m}(a))+\eta_{t}\gamma\widehat{P}_{t}^{m}(1\|1,a)(V_{t-1}^{m}(1)-\widehat{V}_{t-1}^{m})$
	$\displaystyle\geq-\frac{(1-\eta_{t})}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma))-\widehat{P}_{t}^{m}(1\|1,a)\cdot\frac{\eta_{t}\gamma}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma))$
	$\displaystyle\geq-\frac{(1-\eta_{t})}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma))-\frac{\eta_{t}\gamma}{1-\gamma}\prod_{i=1}^{t-1}(1-\eta_{i}(1-\gamma))$
	$\displaystyle\geq-\frac{1}{1-\gamma}\prod_{i=1}^{t}(1-\eta_{i}(1-\gamma)).$	(96)

For $t\in\{t_{r}\}_{r=1}^{R}$ and $a\in\{1,2\}$ , we have,

$\displaystyle Q_{t}^{m}(1,a)-\widehat{Q}_{t}^{m}(a)$	$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}Q_{t-1/2}^{m}(1,a)-\frac{1}{M}\sum_{m=1}^{M}\widehat{Q}_{t-1/2}^{m}(a)$
	$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}\left[(1-\eta_{t})Q_{t-1}^{m}(1,a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)V_{t-1}^{m}(1))\right]$
	$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\frac{1}{M}\sum_{m=1}^{M}\left[(1-\eta_{t})\widehat{Q}_{t-1}^{m}(a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m})\right]$
	$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}\left[(1-\eta_{t})(Q_{t-1}^{m}(1,a)-\widehat{Q}_{t-1}^{m}(a))+\eta_{t}\gamma\widehat{P}_{t}^{m}(1\|1,a)(V_{t-1}^{m}(1)-\widehat{V}_{t-1}^{m})\right]$
	$\displaystyle\geq-\frac{1}{1-\gamma}\prod_{i=1}^{t}(1-\eta_{i}(1-\gamma)),$	(97)

where the last step follows using the same set of arguments as used in (96). The inductive step follows from (96) and (97).

A.5.2 Proof of Lemma 3

In order to bound the term $\mathbb{E}[\Delta_{t,\max}^{m}]-\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$ , we make use of the relation in (46a), which we recall

\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]

\displaystyle\geq\varphi_{t^{\prime},t}\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\left[\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma p\mathbb{E}[\Delta_{k-1,\max}^{m}]\right]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]-\varphi_{t^{\prime},t}\mathbb{E}[|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)|].

•

To aid the analysis, we consider the following recursive relation for any fixed agent $m$ :

\displaystyle y_{t}=(1-\eta_{t})y_{t-1}+\eta_{t}(\gamma py_{t-1}+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]).

(98)

Upon unrolling the recursion, we obtain,

	$\displaystyle y_{t}$	$\displaystyle=\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k})\right)y_{t^{\prime}}+\sum_{k=t^{\prime}+1}^{t}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})\right)\gamma py_{k-1}+\sum_{k=t^{\prime}+1}^{t}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})\right)\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$
		$\displaystyle=\varphi_{t^{\prime},t}y_{t^{\prime}}+\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma py_{k-1}+\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}].$		(99)

Initializing $y_{t^{\prime}}=\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]$ in (99) and plugging this into (46a), we have

\mathbb{E}[\Delta_{t,\max}^{m}]\geq y_{t}-\varphi_{t^{\prime},t}\mathbb{E}[|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)|],

where we used $\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\leq 1$ (cf. (20)). We now further simply the expression of $y_{t}$ . By rewriting (98) as

\displaystyle y_{t}=(1-\eta_{t}(1-\gamma p))y_{t-1}+\eta_{t}\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}],

it is straight forward to note that $y_{t}$ is given as

\displaystyle y_{t}=\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\right)y_{t^{\prime}}+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\left[\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}\right].

(100)

Consequently, we have,

	$\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]-\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle\geq\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\right)\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\left[\sum_{k=t^{\prime}+1}^{t}\eta_{k}^{(t)}-1\right]$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\varphi_{t^{\prime},t}\mathbb{E}[\|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)\|].$		(101)

•

We can consider a slightly different recursive sequence defined as

\displaystyle w_{t}=(1-\eta_{t})w_{t-1}+\eta_{t}(\gamma pw_{t-1}).

(102)

Using a similar sequence of arguments as outlined in (98)-(100), we can conclude that if $w_{t^{\prime}}=\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]$ , then $\mathbb{E}[\Delta_{t,\max}^{m}]\geq w_{t}+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]-\varphi_{t^{\prime},t}\mathbb{E}[|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)|]$ and consequently,

\displaystyle\mathbb{E}[\Delta_{t,\max}^{m}]

\displaystyle\geq\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k}(1-\gamma p))\right)\mathbb{E}[\overline{\Delta}_{t^{\prime},\max}]+\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]-\varphi_{t^{\prime},t}\mathbb{E}[|\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2)|].

(103)

On combining (101) and (103), we arrive at the claim.

A.5.3 Proof of Lemma 4

We begin with bounding the first term $\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$ ; the second bound follows in an almost identical derivation.

Step 1: applying Freedman’s inequality.

Using the relation $\max\{a,b\}=\frac{a+b+|a-b|}{2}$ , we can rewrite $\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$ as

$\displaystyle\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle=\mathbb{E}\left[\frac{\xi_{t^{\prime},t}^{m}(1)+\xi_{t^{\prime},t}^{m}(2)}{2}+\left\|\frac{\xi_{t^{\prime},t}^{m}(1)-\xi_{t^{\prime},t}^{m}(2)}{2}\right\|\right]$
	$\displaystyle=\frac{1}{2}\mathbb{E}\left[\left\|\frac{\xi_{t^{\prime},t}^{m}(1)-\xi_{t^{\prime},t}^{m}(2)}{2}\right\|\right]$
	$\displaystyle=\frac{1}{2}\mathbb{E}\Bigg{[}\Bigg{\|}\underbrace{\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma(\widehat{P}_{k}^{m}(1\|1,1)-\widehat{P}_{k}^{m}(1\|1,2))\widehat{V}_{k-1}^{m}}_{=:\zeta_{t^{\prime},t}^{m}}\Bigg{\|}\Bigg{]},$	(104)

where we used the definition in (40) and the fact that $\mathbb{E}[\xi_{t^{\prime},t}^{m}(1)]=\mathbb{E}[\xi_{t^{\prime},t}^{m}(2)]=0$ . Decompose $\zeta_{t^{\prime},t}^{m}$ as

\displaystyle\zeta_{t^{\prime},t}^{m}

\displaystyle=\sum_{k=t^{\prime}+1}^{t}\sum_{b=1}^{B}\widetilde{\eta}_{k}^{(t)}\frac{\gamma}{B}(P_{k,b}^{m}(1|1,1)-P_{k,b}^{m}(1|1,2))\widehat{V}_{k-1}^{m}=:\sum_{l=1}^{L}z_{l},

(105)

where for all $1\leq l\leq L$

\displaystyle z_{l}:=\frac{\gamma}{B}(P_{k(l),b(l)}^{m}(1|1,1)-P_{k(l),b(l)}^{m}(1|1,2))\widehat{V}_{k(l)-1}^{m}

with

\displaystyle k(l):=\lfloor l/B\rfloor+t^{\prime}+1;\quad b(l)=((l-1)\mod B)+1;\quad L=(t-t^{\prime})B.

Let $\{\mathscr{F}_{l}\}_{l=1}^{L}$ be a filtration such that $\mathscr{F}_{l}$ is the $\sigma$ -algebra corresponding to $\{P_{k(j),b(j)}^{m}(1|1,1),P_{k(j),b(j)}^{m}(1|1,2)\}_{j=1}^{l}$ . It is straightforward to note that $\{z_{l}\}_{l=1}^{L}$ is a martingale sequence adapted to the filtration $\{\mathscr{F}\}_{l=1}^{L}$ . We will use the Freedman’s inequality (Freedman1975Martingale; Li2023QLMinimax) to obtain a high probability bound on $|\zeta_{t^{\prime},t}^{m}|$ .

•

To that effect, note that

$\displaystyle\sup_{l}\|z_{l}\|$	$\displaystyle\leq\sup_{l}\left\|\widetilde{\eta}_{k(l)}^{(t)}\cdot\frac{\gamma}{B}\cdot(P_{k(l),b(l)}^{m}(1\|1,1)-P_{k(l),b(l)}^{m}(1\|1,2))\cdot\widehat{V}_{k(l)-1}^{m}\right\|$
	$\displaystyle\leq\widetilde{\eta}_{k(l)}^{(t)}\cdot\frac{\gamma}{B(1-\gamma)}$
	$\displaystyle\leq\frac{\eta_{t}}{B(1-\gamma)},$	(106)

where the second step follows from the bounds $|(P_{k(l),b(l)}^{m}(1|1,1)-P_{k(l),b(l)}^{m}(1|1,2))|\leq 1$ and $\widehat{V}_{k(l)-1}^{m}\leq\frac{1}{1-\gamma}$ and the third step uses $c_{\eta}\leq\frac{1}{1-\gamma}$ and the fact that $\widetilde{\eta}_{k}^{(T)}$ is increasing in $k$ in this regime. (cf. (21)).

•

Similarly,

$\displaystyle\text{Var}(z_{l}\|\mathscr{F}_{l-1})$	$\displaystyle\leq\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}\frac{\gamma^{2}}{B^{2}}\cdot\left(\widehat{V}_{k(l)-1}^{m}\right)^{2}\cdot\text{Var}(P_{k(l),b(l)}^{m}(1\|1,1)-P_{k(l),b(l)}^{m}(1\|1,2))$
	$\displaystyle\leq\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}\frac{\gamma^{2}}{B^{2}(1-\gamma)^{2}}\cdot 2p(1-p)$
	$\displaystyle\leq\frac{2\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}}{3B^{2}(1-\gamma)}.$	(107)

Using the above bounds (106) and (107) along with Freedman’s inequality yield that

\displaystyle\Pr\left(|\zeta_{t^{\prime},t}^{m}|\geq\sqrt{\frac{8\log(2/\delta)}{3B^{2}(1-\gamma)}\sum_{l=1}^{L}\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}}+\frac{4\eta_{t}\log(2/\delta)}{3B(1-\gamma)}\right)\leq\delta.

(108)

Setting $\delta_{0}=\frac{(1-\gamma)^{2}}{2}\cdot\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}]$ , with probability at least $1-\delta_{0}$ , it holds

\displaystyle|\zeta_{t^{\prime},t}^{m}|\geq\sqrt{\frac{8\log(2/\delta_{0})}{3B(1-\gamma)}\sum_{k=t^{\prime}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}}+\frac{4\eta_{t}\log(2/\delta_{0})}{3B(1-\gamma)}=:D.

(109)

Consequently, plugging this back to (104), we obtain

$\displaystyle\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle=\frac{1}{2}\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|]$
	$\displaystyle\geq\frac{1}{2}\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|\mathbbm{1}\{\|\zeta_{t^{\prime},t}^{m}\|\leq D\}]$
	$\displaystyle\geq\frac{1}{2D}\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}\mathbbm{1}\{\|\zeta_{t^{\prime},t}^{m}\|\leq D\}]$
	$\displaystyle\geq\frac{1}{2D}\left(\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]-\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}\mathbbm{1}\{\|\zeta_{t^{\prime},t}^{m}\|>D\}]\right)$
	$\displaystyle\geq\frac{1}{2D}\left(\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]-\frac{\Pr(\|\zeta_{t^{\prime},t}^{m}\|>D)}{(1-\gamma)^{2}}\right)\geq\frac{1}{4D}\cdot\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}].$	(110)

Here, the penultimate step used the fact that $\displaystyle|\zeta_{t^{\prime},t}^{m}|\leq\sum_{k=t^{\prime}+1}^{t}\frac{\widetilde{\eta}_{k}^{(t)}}{(1-\gamma)}\leq\frac{1}{(1-\gamma)}$ , and the last step used the definition of $\delta_{0}$ . Thus, it is sufficient to obtain a lower bound on $\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}]$ in order obtain a lower bound for $\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$ .

Step 2: lower bounding $\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}]$ .

To proceed, we introduce the following lemma pertaining to lower bounding $\widehat{V}_{t}^{m}$ that will be useful later.

Lemma 6.

For all time instants $t\in[T]$ and all agent $m\in[M]$ :

\displaystyle\mathbb{E}\left[\left(\widehat{V}_{t}^{m}\right)^{2}\right]\geq\frac{1}{2(1-\gamma)^{2}}.

We have,

$\displaystyle\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]$	$\displaystyle=\mathbb{E}\left[\sum_{l=1}^{L}\text{Var}\left(z_{l}\|\mathscr{F}_{l-1}\right)\right]=\mathbb{E}\left[\sum_{l=1}^{L}\mathbb{E}\left[z_{l}^{2}\|\mathscr{F}_{l-1}\right]\right]$
	$\displaystyle\geq\sum_{l=1}^{L}\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}\frac{\gamma^{2}}{B^{2}}\cdot 2p(1-p)\cdot\mathbb{E}\left[\left(\hat{V}_{k(l)-1}^{m}\right)^{2}\right]$
	$\displaystyle\geq\sum_{l=1}^{L}\left(\widetilde{\eta}_{k(l)}^{(t)}\right)^{2}\frac{\gamma^{2}}{B^{2}}\cdot 2p(1-p)\cdot\frac{1}{2(1-\gamma)^{2}}$
	$\displaystyle\geq\frac{2}{9B(1-\gamma)}\cdot\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2},$	(111)

where the third line follows from Lemma 6 and the fourth line uses $\gamma\geq 5/6$ .

Step 3: finishing up.

We finish up the proof by bounding $\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}$ for $t-\max\{t^{\prime},\tau\}\geq 1/\eta_{\tau}$ . We have

$\displaystyle\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}$	$\displaystyle\geq\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})\right)^{2}$
	$\displaystyle\overset{\mathrm{(i)}}{\geq}\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}\left(\eta_{t}\prod_{i=k+1}^{t}(1-\eta_{\tau})\right)^{2}$
	$\displaystyle=\eta_{t}^{2}\sum_{k=\max\{t^{\prime},\tau\}+1}^{t}(1-\eta_{\tau})^{2(t-k)}$
	$\displaystyle\geq\eta_{t}^{2}\cdot\frac{1-(1-\eta_{\tau})^{2(t-\max\{t^{\prime},\tau\})}}{\eta_{\tau}(2-\eta_{\tau})}$
	$\displaystyle\geq\eta_{t}\cdot\frac{1-\exp(-2)}{6}\geq\frac{\eta_{t}}{10}\geq\frac{\eta_{T}}{10},$	(112)

where (i) follows from the monotonicity of $\eta_{k}$ . Plugging (112) into the expressions of $D$ (cf. (109)) we have

	$\displaystyle D=$	$\displaystyle\sqrt{\frac{8\log(2/\delta_{0})}{3B(1-\gamma)}\sum_{k=t^{\prime}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}}+\frac{4\eta_{t}\log(2/\delta_{0})}{3B(1-\gamma)}$
		$\displaystyle\leq\frac{9}{2}\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]\cdot\sqrt{\frac{8\log(2/\delta_{0})}{3}}\left(\frac{1}{B(1-\gamma)}\sum_{k=t^{\prime}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}\right)^{-1/2}+60\cdot\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]\cdot\log(2/\delta_{0})$
		$\displaystyle\leq 3\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]\cdot\log(2/\delta_{0})\left[\sqrt{\frac{60B(1-\gamma)}{\eta_{t}}}+20\right]$
		$\displaystyle\leq 60\mathbb{E}[\|\zeta_{t^{\prime},t}^{m}\|^{2}]\cdot\log(2/\delta_{0})\left[\sqrt{\frac{3B(1-\gamma)}{20\eta_{T}}}+1\right],$

where the second line follows from (111) and (112), and the third line follows from (112). On combining the above bound with (110), we obtain,

\displaystyle\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]\geq\frac{1}{240\log(2/\delta_{0})}\cdot\frac{\nu}{\nu+1},

(113)

where $\nu:=\sqrt{\dfrac{20\eta_{T}}{3B(1-\gamma)}}$ . Note that we have,

\displaystyle\delta_{0}

\displaystyle=\frac{(1-\gamma)^{2}}{2}\cdot\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}]\geq\frac{(1-\gamma)}{9B}\cdot\sum_{k=t^{\prime}+1}^{t}\left(\widetilde{\eta}_{k}^{(t)}\right)^{2}\geq\frac{\eta_{T}(1-\gamma)}{90B}.

Combining the above bound with (113) yields us the required bound.

Step 4: repeating the argument for the second claim.

We note that second claim in the theorem, i.e., the lower bound on $\mathbb{E}\left[\max\left\{\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(1),\frac{1}{M}\sum_{m=1}^{M}\xi_{t^{\prime},t}^{m}(2)\right\}\right]$ follows through an identical series of arguments where the bounds in Eqns. (106) and (107) contain an additional factor of $M$ in the denominator (effectively replacing $B$ with $BM$ ), which is carried through in all the following steps.

A.5.4 Proof of Lemma 5

Using Eqns. (43) and (40), we can write

	$\displaystyle\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)$	$\displaystyle=\left(\prod_{k=t^{\prime}+1}^{t}(1-\eta_{k})\right)(\overline{\Delta}_{t^{\prime}}(1)-\overline{\Delta}_{t^{\prime}}(2))$
		$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}+\frac{1}{M}\sum_{m=1}^{M}\sum_{k=t^{\prime}+1}^{t}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})\right)\gamma(\widehat{P}_{k}^{m}(1\|1,1)-\widehat{P}_{k}^{m}(1\|1,2))\widehat{V}_{k-1}^{m}.$

Upon unrolling the recursion, we obtain,

\displaystyle\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)=\sum_{k=1}^{t}\sum_{m=1}^{M}\left(\eta_{k}\prod_{i=k+1}^{t}(1-\eta_{i})\right)\frac{\gamma}{M}(\widehat{P}_{k}^{m}(1|1,1)-\widehat{P}_{k}^{m}(1|1,2))\widehat{V}_{k-1}^{m}.

If we define a filtration $\mathscr{F}_{k}$ as the $\sigma$ -algebra corresponding to $\{\widehat{P}_{l}^{1}(1|1,1),\widehat{P}_{l}^{1}(1|1,2),\dots,\widehat{P}_{l}^{M}(1|1,1),\widehat{P}_{l}^{M}(1|1,2)\}_{l=1}^{k}$ , then it is straightforward to note that $\{\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)\}_{t}$ is a martingale sequence adapted to the filtration $\{\mathscr{F}_{t}\}_{t}$ . Using Jensen’s inequality, we know that if $\{Z_{t}\}_{t}$ is a martingale adapted to a filtration $\{\mathscr{G}_{t}\}_{t}$ , then for a convex function $f$ such that $f(Z_{t})$ is integrable for all $t$ , $\{f(Z_{t})\}_{t}$ is a sub-martingale adapted to $\{\mathscr{G}_{t}\}_{t}$ . Since $f(x)=|x|$ is a convex function, $\{|\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)|\}_{t}$ is a submartingale adapted to the filtration $\{\mathscr{F}_{t}\}_{t}$ . As a result,

\displaystyle\sup_{1\leq t\leq T}\mathbb{E}[|\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)|]

\displaystyle\leq\mathbb{E}[|\overline{\Delta}_{T}(1)-\overline{\Delta}_{T}(2)|]\leq\left(\mathbb{E}[(\overline{\Delta}_{T}(1)-\overline{\Delta}_{T}(2))^{2}]\right)^{1/2}.

(114)

We use the following observation about a martingale sequence $\{X_{i}\}_{i=1}^{t}$ adapted to a filtration $\{\mathscr{G}_{i}\}_{i=1}^{t}$ to evaluate the above expression. We have,

$\displaystyle\mathbb{E}\left[\left(\sum_{i=1}^{t}X_{i}\right)^{2}\right]$	$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[\left(\sum_{i=1}^{t}X_{i}\right)^{2}\bigg{\|}\mathscr{G}_{t-1}\right]\right]$
	$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[X_{t}^{2}+2X_{t}\left(\sum_{i=1}^{t-1}X_{i}\right)+\left(\sum_{i=1}^{t-1}X_{i}\right)^{2}\bigg{\|}\mathscr{G}_{t-1}\right]\right]$
	$\displaystyle=\mathbb{E}\left[X_{t}^{2}\right]+\mathbb{E}\left[\left(\sum_{i=1}^{t-1}X_{i}\right)^{2}\right]$
	$\displaystyle=\sum_{i=1}^{t}\mathbb{E}\left[X_{i}^{2}\right],$	(115)

where the third step uses the facts that $\left(\sum_{i=1}^{t-1}X_{i}\right)$ is $\mathscr{G}_{t-1}$ measure and $\mathbb{E}[X_{t}|\mathscr{G}_{t-1}]=0$ and fourth step is obtained by recursively applying second and third steps. Using the relation in Eqn. (115) in Eqn. (114), we obtain,

$\displaystyle\sup_{1\leq t\leq T}\mathbb{E}[\|\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)\|]$	$\displaystyle\leq\left(\mathbb{E}[(\overline{\Delta}_{T}(1)-\overline{\Delta}_{T}(2))^{2}]\right)^{1/2}$
	$\displaystyle\leq\left(\sum_{k=1}^{T}\mathbb{E}\left[\left(\sum_{m=1}^{M}\widetilde{\eta}_{k}^{(T)}\cdot\frac{\gamma}{M}\cdot(\widehat{P}_{k}^{m}(1\|1,1)-\widehat{P}_{k}^{m}(1\|1,2))\hat{V}_{k-1}^{m}\right)^{2}\right]\right)^{1/2}$
	$\displaystyle\leq\left(\sum_{k=1}^{T}\left(\widetilde{\eta}_{k}^{(T)}\right)^{2}\cdot\frac{2\gamma^{2}p(1-p)}{BM^{2}}\cdot\sum_{m=1}^{M}\mathbb{E}\left[\left(\hat{V}_{k-1}^{m}\right)^{2}\right]\right)^{1/2}$
	$\displaystyle\leq\left(\sum_{k=1}^{T}\left(\widetilde{\eta}_{k}^{(T)}\right)^{2}\cdot\frac{2\gamma^{2}p(1-p)}{BM(1-\gamma)^{2}}\right)^{1/2}.$	(116)

Let us focus on the term involving the step sizes. We separately consider the scenario for constant step sizes and linearly rescaled step sizes. For constant step sizes, we have,

\displaystyle\sum_{k=1}^{T}\left(\widetilde{\eta}_{k}^{(T)}\right)^{2}

\displaystyle=\sum_{k=1}^{T}\left(\eta_{k}\prod_{i=k+1}^{T}(1-\eta_{i})\right)^{2}=\sum_{k=1}^{T}\eta^{2}(1-\eta)^{2(T-k)}\leq\frac{\eta^{2}}{1-(1-\eta)^{2}}\leq\eta.

(117)

Similarly, for linearly rescaled step sizes, we have,

$\displaystyle\sum_{k=1}^{T}\left(\widetilde{\eta}_{k}^{(T)}\right)^{2}$	$\displaystyle=\sum_{k=1}^{\tau}\left(\widetilde{\eta}_{k}^{(T)}\right)^{2}+\sum_{k=\tau+1}^{T}\left(\eta_{k}\prod_{i=k+1}^{T}(1-\eta_{i})\right)^{2}$
	$\displaystyle\leq\sum_{k=1}^{\tau}\left(\widetilde{\eta}_{\tau}^{(T)}\right)^{2}+\sum_{k=\tau+1}^{T}\eta_{k}^{2}(1-\eta_{T})^{2(T-k)}$
	$\displaystyle\leq\eta_{\tau}^{2}(1-\eta_{T})^{2(T-\tau)}\cdot\tau+\eta_{\tau}^{2}\cdot\frac{1}{\eta_{T}(2-\eta_{T})}$
	$\displaystyle\leq 3\eta_{T}\cdot\eta_{T}\cdot T\cdot\exp\left(-\frac{4T\eta_{T}}{3}\right)+3\eta_{T}$
	$\displaystyle\leq\frac{9}{4e}\eta_{T}+3\eta_{T}$
	$\displaystyle\leq 4\eta_{T},$	(118)

where the second step uses $c_{\eta}\leq\log N\leq\frac{1}{1-\gamma}$ and the fact that $\widetilde{\eta}_{k}^{(T)}$ is increasing in $k$ in this regime. (See Eqn. (21)) and fifth step uses $xe^{-4x/3}\leq 3/4e$ . On plugging results from Eqns. (117) and (118) into Eqn. (116) along with the value of $p$ , we obtain,

\displaystyle\sup_{1\leq t\leq T}\mathbb{E}[|\overline{\Delta}_{t}(1)-\overline{\Delta}_{t}(2)|]

\displaystyle\leq\sqrt{\frac{8\eta_{T}}{3BM(1-\gamma)}},

(119)

as required.

A.5.5 Proof of Lemma 6

For the proof, we fix an agent $m$ . In order to obtain the required lower bound on $\widehat{V}_{t}^{m}$ , we define an auxiliary sequence $\overline{Q}_{t}^{m}$ that evolves as described in Algorithm 5. Essentially, $\overline{Q}_{t}^{m}$ evolves in a manner almost identical to $\widehat{Q}_{t}^{m}$ except for the fact that there is only one action and hence there is no maximization step in the update rule.

r\leftarrow 1

\overline{Q}^{m}_{0}=Q^{\star}(1,1)

for all

m\in\{1,2,\dots,M\}

2: for

t=1,2,\dots,T

3: for

m=1,2,\dots,M

\overline{Q}^{m}_{t-1/2}\leftarrow(1-\eta_{t})\overline{Q}^{m}_{t-1}(a)+\eta_{t}(1+\widehat{P}_{t}^{m}(1|1,1)\overline{Q}_{t-1}^{m})

5: Compute

\overline{Q}_{t}^{m}

according to Eqn. (8)

6: end for

7: end for

Algorithm 5 Evolution of

\overline{Q}

It is straightforward to note that $\widehat{Q}_{t}^{m}(1)\geq\overline{Q}_{t}^{m}$ , which can be shown using induction. From the initialization, it follows that $\widehat{Q}_{0}^{m}(1)\geq\overline{Q}_{0}^{m}$ . Assuming the relation holds for $t-1$ , we have,

	$\displaystyle\widehat{Q}_{t-1/2}^{m}(1)$	$\displaystyle=(1-\eta_{t})\widehat{Q}_{t-1}^{m}(1)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,1)\widehat{V}_{t-1}^{m})$
		$\displaystyle\geq(1-\eta_{t})\widehat{Q}_{t-1}^{m}(1)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,1)\widehat{Q}_{t-1}^{m}(1))$
		$\displaystyle\geq(1-\eta_{t})\overline{Q}_{t-1}^{m}+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,1)\overline{Q}_{t-1}^{m})$
		$\displaystyle=\overline{Q}_{t-1/2}^{m}.$

Since $\widehat{Q}_{t}^{m}$ and $\overline{Q}_{t}^{m}$ follow the same averaging schedule, it immediately follows from the above relation that $\widehat{Q}_{t}^{m}(1)\geq\overline{Q}_{t}^{m}$ . Since $\widehat{V}_{t}^{m}\geq\widehat{Q}_{t}^{m}(1)\geq\overline{Q}_{t}^{m}$ , we will use the sequence $\overline{Q}_{t}^{m}$ to establish the required lower bound on $\widehat{V}_{t}^{m}$ .

We claim that for all time instants $t$ and all agents $m$ ,

\displaystyle\mathbb{E}[\overline{Q}_{t}^{m}]=\frac{1}{1-\gamma p}.

(120)

Assuming (120) holds, we have

\displaystyle\mathbb{E}[(\widehat{V}_{t}^{m})^{2}]\geq\left(\mathbb{E}[\widehat{V}_{t}^{m}]\right)^{2}\geq\left(\mathbb{E}[\overline{Q}_{t}^{m}]\right)^{2}\geq\left(\frac{1}{1-\gamma p}\right)^{2}\geq\frac{1}{2(1-\gamma)^{2}},

as required. In the above expression, the first inequality follows from Jensen’s inequality, the second from the relation $\widehat{V}_{t}^{m}\geq\overline{Q}_{t}^{m}\geq 0$ and the third from (120).

We now move now to prove the claim (120) using induction. For the base case, $\mathbb{E}[\overline{Q}_{0}^{m}]=\frac{1}{1-\gamma p}$ holds by choice of initialization. Assume that $\mathbb{E}[\overline{Q}_{t-1}^{m}]=\frac{1}{1-\gamma p}$ holds for some $t-1$ for all $m$ .

•

If $t$ is not an averaging instant, then for any client $m$ ,

$\displaystyle\overline{Q}_{t}^{m}$	$\displaystyle=(1-\eta_{t})\overline{Q}_{t-1}^{m}+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,1)\overline{Q}_{t-1}^{m})$
$\displaystyle\implies\mathbb{E}[\overline{Q}_{t}^{m}]$	$\displaystyle=(1-\eta_{t})\mathbb{E}[\overline{Q}_{t-1}^{m}]+\eta_{t}(1+\gamma\mathbb{E}[\widehat{P}_{t}^{m}(1\|1,1)\overline{Q}_{t-1}^{m}])$
	$\displaystyle=(1-\eta_{t})\mathbb{E}[\overline{Q}_{t-1}^{m}]+\eta_{t}(1+\gamma p\mathbb{E}[\overline{Q}_{t-1}^{m}])$
	$\displaystyle=\frac{(1-\eta_{t})}{1-\gamma p}+\eta_{t}\left(1+\frac{\gamma p}{1-\gamma p}\right)=\frac{1}{1-\gamma p}.$	(121)

The third line follows from the independence of $\widehat{P}_{t}^{m}(1|1,1)$ and $\overline{Q}_{t-1}^{m}$ and the fourth line uses the inductive hypothesis.

•

If $t$ is an averaging instant, then for all clients $m$ ,

$\displaystyle\overline{Q}_{t}^{m}$	$\displaystyle=\frac{(1-\eta_{t})}{M}\sum_{j=1}^{M}\overline{Q}_{t-1}^{j}+\eta_{t}\frac{1}{M}\sum_{j=1}^{M}(1+\gamma\widehat{P}_{t}^{j}(1\|1,1)\overline{Q}_{t-1}^{j})$
$\displaystyle\implies\mathbb{E}[\overline{Q}_{t}^{m}]$	$\displaystyle=\frac{(1-\eta_{t})}{M}\sum_{j=1}^{M}\mathbb{E}[\overline{Q}_{t-1}^{j}]+\eta_{t}\frac{1}{M}\sum_{j=1}^{M}(1+\gamma\mathbb{E}[\widehat{P}_{t}^{j}(1\|1,1)\overline{Q}_{t-1}^{j}])$
	$\displaystyle=\frac{(1-\eta_{t})}{M}\sum_{j=1}^{M}\frac{1}{1-\gamma p}+\eta_{t}\frac{1}{M}\sum_{j=1}^{M}\left(1+\frac{\gamma p}{1-\gamma p}\right)=\frac{1}{1-\gamma p},$	(122)

where we again make use of independence and the inductive hypothesis.

Thus, (121) and (122) taken together complete the inductive step.

	$\displaystyle\Delta_{t}^{m}(a)$	$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}(1+\gamma\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m}-Q^{\star}(1,a))$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(\widehat{P}_{t}^{m}(1\|1,a)\widehat{V}_{t-1}^{m}-pV^{\star}(1))$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(p(\widehat{V}_{t-1}^{m}-V^{\star}(1))+(\widehat{P}_{t}^{m}(1\|1,a)-p)\widehat{V}_{t-1})$
		$\displaystyle=(1-\eta_{t})\Delta_{t-1}^{m}(a)+\eta_{t}\gamma(p\Delta_{t-1,\max}^{m}+(\widehat{P}_{t}^{m}(1\|1,a)-p)\widehat{V}_{t-1}^{m}).$

$\displaystyle\mathbb{E}[\xi_{t^{\prime},t,\max}^{m}]$	$\displaystyle=\mathbb{E}\left[\frac{\xi_{t^{\prime},t}^{m}(1)+\xi_{t^{\prime},t}^{m}(2)}{2}+\left\|\frac{\xi_{t^{\prime},t}^{m}(1)-\xi_{t^{\prime},t}^{m}(2)}{2}\right\|\right]$
	$\displaystyle=\frac{1}{2}\mathbb{E}\left[\left\|\frac{\xi_{t^{\prime},t}^{m}(1)-\xi_{t^{\prime},t}^{m}(2)}{2}\right\|\right]$
	$\displaystyle=\frac{1}{2}\mathbb{E}\Bigg{[}\Bigg{\|}\underbrace{\sum_{k=t^{\prime}+1}^{t}\widetilde{\eta}_{k}^{(t)}\gamma(\widehat{P}_{k}^{m}(1\|1,1)-\widehat{P}_{k}^{m}(1\|1,2))\widehat{V}_{k-1}^{m}}_{=:\zeta_{t^{\prime},t}^{m}}\Bigg{\|}\Bigg{]},$	(104)

The Sample-Communication Complexity Trade-off in Federated Q-Learning

1 Introduction

1.1 Main results

1.2 Related work

Single-agent Q-learning.

Federated and distributed RL.

Federated Q-learning.

Accuracy-communication trade-off in federated learning.

2 Background and Problem Formulation

2.1 Markov decision processes

2.2 Performance measures in federated Q-learning

2.3 Intermittent-communication algorithm protocols

3 Communication Complexity Lower Bound

Theorem 1.

Remark 1 (Communication complexity of policy evaluation).

Remark 2 (Extension to asynchronous Q-learning).

4 The Fed-DVR-Q Algorithm

4.1 Algorithm description

4.1.1 The RefineEstimate sub-routine

4.1.2 The compression operator

4.1.3 Setting the parameters

4.2 Performance guarantees

Theorem 2.

Optimal sample-communication complexity trade-off.

Role of minibatching.

Compression.

5 Conclusion

Appendix A Proof of Theorem 1

Introducing the “hard” MDP instance.

Establishing the performance of intermittent communication algorithms.

Generalization to larger MDPs.

A.1 Introducing the “hard” instance

Lemma 1 ((Li2023QLMinimax, Lemma 3)).

A.2 Notation and preliminary results

Useful relations of the learning rates.

Proof of (18).

Sample transition matrix.

Preliminary relations of the iterates.

A.3 Main analysis

A.3.1 Small learning rates

Proof of (28).

A.3.2 Large learning rates with small ηTB​M\frac{\eta_{T}}{BM}

Step 1: introducing an auxiliary sequence.

Lemma 2.

Step 2: deriving a recursive bound for 𝔼​[Δ¯t,max]\mathbb{E}[\overline{\Delta}_{t,\max}].

Lemma 3.

Lemma 4.

Lemma 5.

Proof of (50).

Step 3: lower bounding 𝔼​[Δ¯T,max]\mathbb{E}[\overline{\Delta}_{T,\max}].

Step 4: finishing up the proof.

Proof of (55).

Proof of (61).

A.3.3 Large learning rates with large ηTB​M\frac{\eta_{T}}{BM}

Proof of (88).

A.4 Generalizing to larger state action spaces

A.5 Proofs of auxiliary lemmas

A.5.1 Proof of Lemma 2

A.5.2 Proof of Lemma 3

A.5.3 Proof of Lemma 4

Step 1: applying Freedman’s inequality.

Step 2: lower bounding 𝔼​[|ζt′,tm|2]\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}].

Lemma 6.

Step 3: finishing up.

Step 4: repeating the argument for the second claim.

A.5.4 Proof of Lemma 5

A.5.5 Proof of Lemma 6

A.3.2 Large learning rates with small $\frac{\eta_{T}}{BM}$

Step 2: deriving a recursive bound for $\mathbb{E}[\overline{\Delta}_{t,\max}]$ .

Step 3: lower bounding $\mathbb{E}[\overline{\Delta}_{T,\max}]$ .

A.3.3 Large learning rates with large $\frac{\eta_{T}}{BM}$

Step 2: lower bounding $\mathbb{E}[|\zeta_{t^{\prime},t}^{m}|^{2}]$ .