Differential Advising in Multi-Agent Reinforcement Learning

Reinforcement learning is usually used to solve sequential decision-making problems. A sequential decision-making process can be formally modeled as a Markov decision process (MDP) [19]. An MDP is typically a tuple $\langle S,A,T,R\rangle$, where $S$ is the set of states, $A$ is a set of actions available to the agent, $T$ is the transition function, and $R$ is the reward function.

At each step, an agent observes the state $s\in S$ of the environment, and selects an action $a\in A$ based on its policy $\pi$, which is a probability distribution over available actions. After performing the action, the agent receives a real-valued reward $r$ and gets into a new state $s^{\prime}\in S$. The agent then updates its policy $\pi$ based on the reward $r$ and the new state $s^{\prime}$. Through this way, the agent can gradually accumulate knowledge and improve its policy to maximize its accumulated long-term expected reward.

An MDP can be extended to model a multi-agent reinforcement learning process as a tuple $\langle S,A_{1,...,n},T,R_{1,...,n}\rangle$, where $S$ is the cartesian product of the sets of local states from each agent, $S=S_{1}\times S_{2}\cdots\times S_{n}$, $A_{i}$ is the available action set of agent $i$, $T$ is the transition function, and $R_{i}$ is the reward function of agent $i$. In multi-agent reinforcement learning, state transitions are based on the joint actions of all the agents. Each agent’s individual reward is also based on the joint actions of all the agents.

Differential privacy is a prevalent privacy model and has been broadly applied to various applications [20, 21, 22]. Differential privacy can guarantee an individual’s privacy independent of whether the individual is in or out of a dataset [7]. Two datasets $D$ and $D^{\prime}$ are neighboring datasets if they differ in at most one record. Let $f$ be a query that maps dataset $D$ to a $k$-dimension vector in range $\mathbb{R}^{k}$: $f:D\rightarrow\mathbb{R}^{k}$. The maximal difference on the results of query $f$ is defined as sensitivity of query $\Delta f$, which determines how much perturbation is required for the privacy-preserving answer. The formal definition of sensitivity is given as follows.

The aim of differential privacy is to mask the difference in the answer of query $f$ between two neighboring datasets. To achieve this aim, differential privacy provides a randomized mechanism $\mathcal{M}$ to access a dataset. In $\epsilon$-differential privacy, parameter $\epsilon$ is defined as the privacy budget, which controls the privacy guarantee level of mechanism $\mathcal{M}$. A smaller $\epsilon$ represents a stronger privacy. The formal definition of differential privacy is presented as follows.

One of the prevalent differential privacy mechanisms is the Laplace mechanism. The Laplace mechanism adds Laplace noise to query results. We use $Lap(b)$ to represent the noise sampled from the Laplace distribution with scaling $b$. The mechanism is described as follows.

Based on the definitions of differential privacy, we provide the definitions of differential advising. The aim of providing the definitions of differential advising is to link the properties of differential privacy to the properties of differential advising. This linkage can guarantee that the differential privacy mechanisms can be used as differential advising mechanisms, as later shown in Lemma 1 and Theorem 1 in Section 5.

The reason of applying differential privacy to advising is to ensure that the knowledge learned in a state can still be used in a different but similar state, i.e., a neighboring state (Definition 4). In previous work, knowledge can only be re-used in identical states. With the introduction of the ‘neighboring state’, a piece of knowledge has more chance to be applied by agents compared with previous work. This idea is reasonable as we human can easily borrow knowledge from a similar environment.

To achieve this goal, we can consider acquiring knowledge, i.e., asking for advice, as querying to a dataset, and consider the $Q$-values of state/action pairs constituting a dataset, i.e., a $Q$-table. Then, the differential advising process can be simulated as a differentially private query process. To implement differential advising, we apply the Laplace mechanism to mask the difference between the knowledge learned in two neighboring states. Specifically, a $Q$-table, shown as below, is a two-dimensional matrix which stores the knowledge learned by an agent. Differential advising is operating on $Q$-tables.

Without loss of generality, we assume that a state $\boldsymbol{s}$ consists of $m$ dimensions: $\boldsymbol{s}=(s_{1},...,s_{m})$. Each dimension $s_{i}$ could be either an integer or a real number representing discrete or continuous states. The difference between two states $\boldsymbol{s}$ and $\boldsymbol{s}^{\prime}$ is defined as:

Two states $\boldsymbol{s}$ and $\boldsymbol{s}^{\prime}$ are neighboring, i.e., slightly different, if their difference is at most $1$, i.e., $||\boldsymbol{s}-\boldsymbol{s}^{\prime}||_{1}\leq 1$. The difference threshold is set to $1$, because this is the prerequisite of using differential privacy. As described in Definition 2, differential privacy works when two datasets have at most one record in difference. In fact, Definition 4 is slightly different from the prerequisite of differential privacy. This is because data records in a dataset are discrete, and thus $||D-D^{\prime}||_{1}\leq 1$ means that one record is different in the two datasets. In Definition 4, if states are discrete and each dimension is represented as an integer, $||\boldsymbol{s}-\boldsymbol{s}^{\prime}||_{1}\leq 1$ also means that one dimension is different in the two states, such as the multi-robot example in Section 1. If states are continuous, $||\boldsymbol{s}-\boldsymbol{s}^{\prime}||_{1}\leq 1$ could mean that the difference exists in multiple dimensions and the sum of the difference is less than $1$. We leave the further research of continuous states as one of our future works.

Similarly, the advice sensitivity is defined as follows.

As an advice is a $Q$-vector, for simplicity, we use the letter “$Q$” to represent both the advice generation function and the $Q$-function. Details will be given in the next section.

The essence of differential privacy is to guarantee that the query results of two neighboring datasets have a very high probability to be the same. Similarly, in differential advising, we set that the query results of two neighboring $Q$-tables have a very high probability to be the same, so that an agent querying to two neighboring $Q$-tables can receive almost the same knowledge. Here, two $Q$-tables are neighboring if 1) they have one record in difference and 2) the two different records correspond to two neighboring states. The query in differential advising is known as differentially private selection [23] which produces the best answer from a space of outcomes. To implement differential advising, we follow the spirit of the Laplace mechanism by adding Laplace noise on query results to mask the difference between two neighboring $Q$-tables. The detail will be given in the next section.

Table I describes the notations and terms used in this paper.

In our method, during advising, agents have to address the following three sub-problems:

• •

  whether to ask for advice;

• •

  whether to give advice;

• •

  how to use advice.

The overview of our method is outlined in Fig. 2, and the detail of our method is formally given in Algorithm 1. In summary, Algorithm 1 describes the workflow of our method, which consists of two parts: the advising part (Lines 1-15) and the learning part (Lines 16-23). The advising part of Algorithm 1 depicts the process of an advisee agent asking for and making use of advice. The detail of making use of advice is given in Algorithm 2, where the differential privacy mechanism is introduced. The learning part of Algorithm 1 is a regular reinforcement learning process.

In Lines 1-3, at time step $t$, agent $i$ observes a state $\boldsymbol{s}^{t}$, and decides whether to ask for advice from other agents (Section 4.2). Specifically, an advice is the $Q$-value vector of a state: $\boldsymbol{Q}(\boldsymbol{s})=(Q(\boldsymbol{s},a_{1}),...,Q(\boldsymbol{s},a_{k}))$. In Lines 4 and 5, if agent $i$ decides to ask for advice, agent $i$ checks whether it has a neighboring state $\boldsymbol{s}^{\prime}$ itself. If so, agent $i$ takes $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$ as a self-advice. Agent $i$, thereafter, uses Algorithm 2, a differentially private algorithm, to process this advice and selects an action based on this advice (Line 6). Here, allowing an agent to query itself can conserve its communication budget. This is called “self-advising”. In Lines 8 and 9, if agent $i$ has not visited any neighboring states, agent $i$ sends a broadcast message to all communication-reachable, i.e., neighboring, agents and its communication budget $C_{i}$ is deducted by $1$. If any neighboring agent $j$ decides to offer advice, agent $j$ sends the advice back to agent $i$ and agent $j$’s communication budget $C_{j}$ is deducted by $1$ (Section 4.3).

In Lines 10 and 11, once agent $i$ receives the advice from agent $j$, agent $i$ checks whether this advice is created based on state $\boldsymbol{s}^{t}$ or a neighboring state of $\boldsymbol{s}^{t}$. If the advice is created based on $\boldsymbol{s}^{t}$, agent $i$ directly uses this advice to choose an action (Line 12). Otherwise, in Lines 13 and 14, agent $i$ uses Algorithm 2 to modify this advice and then chooses an action by following the modified advice (Section 4.4). In the case that agent $i$ receives advice from multiple agents, it selects the advice whose state has the minimum difference to agent $i$’s current state (recall Equation 4). In the case that no advice is received or agent $i$ decides not to ask for advice, agent $i$ simply chooses an action based on its own experience (Lines 16-23). Specifically, Lines 16-23 describe a regular reinforcement learning process. In Line 16, agent $i$ randomly selects an action based on the probability distribution $\boldsymbol{\pi}(\boldsymbol{s}^{t})$ over its available actions in state $\boldsymbol{s}^{t}$. Then, in Line 17, agent $i$ performs the selected action and receives a reward $r$. In Line 18, agent $i$ uses reward $r$ to update the $Q$-value of the selected action. In Line 19, agent $i$ computes an average reward using the probability distribution in state $\boldsymbol{s}^{t}$ to multiply the $Q$-values of actions in state $\boldsymbol{s}^{t}$. After that, in Lines 20 and 21, agent $i$ adopts the average reward to update the probability distribution $\boldsymbol{\pi}(\boldsymbol{s}^{t})$. In Line 22, $\boldsymbol{\pi}(\boldsymbol{s}^{t})$ is normalized to be a valid probability distribution, where each element in $\boldsymbol{\pi}(\boldsymbol{s}^{t})$ is in $(0,1)$ and the sum of all the elements is $1$. In Line 23, the learning rate $\alpha$ is decayed to guarantee the convergence of the algorithm.

It has to be noted that in our method, an advice is a $Q$-vector instead of an action suggestion or a $Q$-function. This is because our method is differential advising rather than direct advising, and using $Q$-vectors enables advisee agents to properly modify the advice before they take it. By contrast, modifying $Q$-vectors is easier and more reliable than modifying action suggestions and $Q$-functions. Moreover, using $Q$-vectors as advice does not mean adviser agents must have a complete $Q$-table. As long as an adviser agent has the $Q$-values of the concerned state $s$ (or its neighboring state $s^{\prime}$), the adviser can offer advice.

At each time step, an agent, $i$, observes a state $\boldsymbol{s}$. Agent $i$ decides whether to ask for advice based on probability $P_{ask}$ (Equation 7). A random number between $0$ and $1$ is generated to compare with $P_{ask}$. If the random number is less than $P_{ask}$, agent $i$ asks for advice.

The calculation of $P_{ask}$ is based on 1) agent $i$’s confidence in current state $\boldsymbol{s}$, and 2) agent $i$’s remaining communication budget, $C^{\prime}_{i}$. Agent $i$’s confidence in state $\boldsymbol{s}$ depends on how many times agent $i$ has visited state $\boldsymbol{s}$, denoted as $n^{i}_{\boldsymbol{s}}$. A higher $n^{i}_{\boldsymbol{s}}$ value means a higher confidence and results in a lower probability of asking for advice. Agent $i$’s remaining communication budget $C^{\prime}_{i}$ depends on that how many times agent $i$ has asked for/provided advice from/to others. Once agent $i$ asks for advice, its communication budget is reduced by $1$: $C^{\prime}_{i}\leftarrow C^{\prime}_{i}-1$. A lower remaining communication budget results in a lower probability of asking for advice. Formally, the probability function $P_{ask}$ takes $n^{i}_{\boldsymbol{s}}$ and $C^{\prime}_{i}$ as input, and outputs a probability ranged in $[0,1)$. The calculation of probability $P_{ask}$ is shown in Equation 7, where square root is used to reduce the decay speed of $P_{ask}$, and the positive integer, $N$, is a threshold used to avoid agent $i$ using up its communication budget in very early stages.

If agent $i$ decides to ask for advice, agent $i$ checks whether it has any neighboring states. This check is based on Equation 4 to compute the difference between the new state $\boldsymbol{s}$ and the stored states of agent $i$. If the difference between $\boldsymbol{s}$ and a stored state is less than or equal to $1$, that stored state is a neighboring state of $\boldsymbol{s}$.

On one hand, if agent $i$ has a set of neighboring states, agent $i$ selects the neighboring state $\boldsymbol{s}^{\prime}$, which has the minimum difference to its current state $\boldsymbol{s}$. Agent $i$ utilizes the proposed differential advising method to modify $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$ and use the modified $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$ to take an action (Section 4.4). On the other hand, if agent $i$ does not have any neighboring states, it asks for advice from its neighboring agents (Section 4.3).

When an agent, $j$, receives an advice request message from agent $i$, agent $j$ needs to decide whether to provide advice to agent $i$. This decision is based on probability $P_{give}$ (Equation 8). The calculation of $P_{give}$ is based on 1) agent $j$’s confidence in agent $i$’s concerned state $\boldsymbol{s}$ or neighboring states of $\boldsymbol{s}$, and 2) agent $j$’s remaining communication budget $C^{\prime}_{j}$. If agent $j$ has more than one neighboring state, agent $j$ uses the one which has the minimum difference to state $\boldsymbol{s}$. The higher the confidence of agent $j$, the higher the probability agent $j$ will provide advice. The lower remaining communication budget of agent $j$, the lower probability agent $j$ will provide advice.

Formally, the probability function $P_{give}$ takes $n^{j}_{\boldsymbol{s}}$ and $C^{\prime}_{j}$ as input, and outputs a probability ranged in $[0,1)$. Here, for simplicity, we use $n^{j}_{\boldsymbol{s}}$ to denote the number of times that agent $j$ has visited either state $\boldsymbol{s}$ or a neighboring state of $\boldsymbol{s}$, and $C^{\prime}_{j}$ to denote the remaining communication budget of agent $j$. Once agent $j$ gives advice, its remaining communication budget is reduced by $1$: $C^{\prime}_{j}\leftarrow C^{\prime}_{j}-1$. It should be noted that agent $j$ does not need to know the full state space to find neighboring states, because in some environments, it is infeasible for any individual agent to know the full state space, such as the multi-robot example in Section 1. Thus, agent $j$ simply checks the states which it has visited. If all the visited states are not neighboring to the concerned state, this means that agent $j$ does not have the experience related to the concerned state and should not offer any advice to agent $i$. Formally, the calculation of $P_{give}$ is given in Equation 8.

In Equation 8, we set a thresholds $n^{i}_{\boldsymbol{s}}$ for adviser agents. If an adviser agent has visited a state for less than $n^{i}_{\boldsymbol{s}}$ times, that means the adviser agent is less experienced than the advisee agent in state $\boldsymbol{s}$ and thus the adviser agent is not allowed to offer advice to the advisee agent for that state. This setting is based on the fact that in theory, an adviser agent can generate advice for any possible states, even if the adviser agent has never visited those states. For example, an adviser agent can generate advice for a never visited state using the initialized $Q$-values of actions in that state. However, since the $Q$-value of each action can be randomly initialized, such advice is useless and may even harm the learning performance of the advisee.

When agent $i$ receives agent $j$’s advice, $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$, agent $i$ checks whether its concerned state $\boldsymbol{s}$ is the same as state $\boldsymbol{s}^{\prime}$. The aim of this check is to decide whether this advice is created based on the same state ($\boldsymbol{s}=\boldsymbol{s}^{\prime}$) or a neighboring state ($\boldsymbol{s}\neq\boldsymbol{s}^{\prime}$). If $\boldsymbol{s}=\boldsymbol{s}^{\prime}$, agent $i$ directly takes agent $j$’s advice and selects an action based on agent $j$’s advice. If $\boldsymbol{s}\neq\boldsymbol{s}^{\prime}$, agent $i$ uses the Laplace mechanism to add noise to agent $j$’s advice, $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$, shown in Algorithm 2. Here is an example. Agent $i$ arrives at state s and asks for advice from agent $j$. Agent $j$ has visited three states: $\boldsymbol{s}_{0},\boldsymbol{s}_{1},\boldsymbol{s}_{2}$ and are confident in these states. After calculation, agent $j$ detects that none of the three states is the same as state $\boldsymbol{s}$ and only $\boldsymbol{s}_{0}$ is a neighboring state of $\boldsymbol{s}$. Thus, agent $j$ sends its advice, Q-vector $Q(\boldsymbol{s}_{0})$, to agent $i$. After agent $i$ receives this advice, agent $i$ identifies that this advice is created based on a neighboring state $\boldsymbol{s}_{0}$. Agent $i$ adopts Algorithm 2 to add Laplace noise to the advice $Q(\boldsymbol{s}_{0})$, and then uses the modified advice to guide its action selection.

In Lines 2 and 3, agent $j$’s $Q$-values are adjusted by adding Laplace noise. Specifically, the Laplace noise is added to each of the $Q$-values: $Q(\boldsymbol{s^{\prime}},a_{1})+Lap_{1},m,Q(\boldsymbol{s^{\prime}},a_{k})+Lap_{k}$, where each noise, $Lap_{i}$, is a random number based on the Laplace distribution. Based on the adjusted $Q$-values, in Lines 4-6, agent $i$ computes its average reward in state $\boldsymbol{s}^{t}$ and updates its probability distribution $\pi(\boldsymbol{s}^{t},a)$. Note that the computation of the average reward uses the probability distribution in state $\boldsymbol{s}^{t}$ to multiply the $Q$-values of the available actions in state $\boldsymbol{s^{\prime}}$. As we have the assumption that an agent has the same action set in two neighboring states, this multiplication is valid. In Line 7, $\pi(\boldsymbol{s}^{t},a)$ is normalized to be a valid probability distribution.

In the Laplace mechanism in Line 3, $\Delta Q$ is the sensitivity of $Q$-values (recall Definition 5). Typically, $\Delta Q$ is based on $Q$-functions, and $Q$-functions are based on rewards. Thus, essentially, $\Delta Q$ is based on rewards, and we can use rewards to approximately compute $\Delta Q$. We have to notice that it is infeasible to accurately compute $\Delta Q$ before the learning starts, because the accurate computation of $\Delta Q$ needs the final $Q$-values of all the states, and these final $Q$-values can only be obtained when the learning finishes.

We use an example to demonstrate how to approximately compute $\Delta Q$. Definition 5 states that $\Delta Q$ is the maximum difference between two $Q$-vectors of any two neighboring states. In the multi-robot problem exemplified in Section 1, for any two neighboring states, the maximum reward difference is $10-(-5)=15$ given that the maximum reward $10$ is finding a victim and the minimum reward $-5$ is hitting an obstacle. This is because in any two neighboring states, only one dimension, i.e., one cell, is different. This different cell incurs a maximum reward difference, if the cell has a victim in a state while has an obstacle in the other state. Moreover, $Q$-value is usually updated based on a learning rate $\alpha$. Thus, in the multi-robot problem, $\Delta Q$ can be approximately computed as $\alpha\cdot 15$. When we set $\alpha=0.2$, then $\Delta Q=3$.

The rationale of Algorithm 2 is discussed as follows. According to Definition 3 in Section 3.2, the Laplace mechanism is used to add Laplace noise to the numerical output of a query $f$ to a dataset $D$, so that both the privacy and the utility of the output can be guaranteed. Typically, a query $f$ maps a dataset $D$ to $k$ real numbers: $f:D\rightarrow\mathbb{R}^{k}$. In comparison, in advising learning, the $Q$-table of an adviser agent is interpreted as a dataset, where each entry in the $Q$-table is the $Q$-value of a pair of a state and an action, $Q(s,a)$. Advice request from an advisee agent is interpreted as the query to the Q-table of the adviser agent. The output of the query is the advice of the adviser agent, which consists of $k$ real numbers, $Q(\boldsymbol{s},a_{1}),...,Q(\boldsymbol{s},a_{k})$, and $k$ is the number of actions in state $s$. As discussed in Section 3.3, we have connected the properties of differential privacy and the properties of differential advising. Hence, by adding Laplace noise to $Q(\boldsymbol{s},a_{1}),...,Q(\boldsymbol{s},a_{k})$, both the validity and utility of the advice can be guaranteed, as shown in Theorem 1, 2 and 3 in Section 5. Thus, in state $\boldsymbol{s}$, agent $i$ can still use the advice $\boldsymbol{Q}(\boldsymbol{s}^{\prime})$, which is created based on a neighboring state $\boldsymbol{s}^{\prime}$.

Lemma 1 creates a link between differential privacy and differential advising. Then, differential privacy mechanisms, e.g., the Laplace mechanism, can be used to implement differential advising.

Theorem 1 theoretically proves that the proposed method is a valid differential advising method which enables agents to use advice created based on similar states. Moreover, the parameter $\epsilon$ is used to control the amount of noise added to an advice. The tuning of $\epsilon$ is left to users.

As shown in Line 4, Algorithm 2, the average reward of an agent is: $\overline{r}=\sum_{a\in A}\pi(\boldsymbol{s},a)Q(\boldsymbol{s},a)$. We will demonstrate that with a very high probability, the average reward of an agent using differential advising is greater than the average reward without using differential advising.

In this proof, we use the average reward equation: $\overline{r}=\sum_{a\in A}\pi(\boldsymbol{s},a)Q(\boldsymbol{s},a)$. This equation is valid to correctly estimate an agent’s average reward, only if the agent has a good estimation of the $Q$-table. Hence, this theorem holds in late learning stages and reveals the final results.

Remark 1: The upper bound of $\delta$, $-\frac{\Delta Q}{\epsilon}\cdot\frac{1}{t}\cdot ln(2-2\beta)$, relies on $t$. Let $\delta(t)=-\frac{\Delta Q}{\epsilon}\cdot\frac{1}{t}\cdot ln(2-2\beta)$. Hence, $\delta(t)$ is monotonically decreasing with the increase of $t$. Specifically, when $t\rightarrow 0$, $\delta(t)\rightarrow\infty$, and when $t\rightarrow\infty$, $\delta(t)\rightarrow 0$. This means that in early stages, using differential advising can significantly increase agents’ average rewards. However, as time progresses, the improvement of agents’ average rewards decreases. This situation reflects the fact that in early stages, an agent is not knowledgable, so other agents’ knowledge can give the agent significant help. As time progresses, the agent accumulates enough knowledge, and thus other agents’ knowledge is trivial to the agent.

Theorem 2 theoretically demonstrates the effectiveness of our method by analyzing lower bound of an agent’s average reward difference between using and not using our method. The analysis result shows that the lower bound of the average reward difference is positive with a very high probability. This means that the agent using our method can receive a higher average reward than not using our method with a very high probability. Given that the average reward of using and not using the proposed method is $\overline{r}^{\prime}$ and $\overline{r}$, respectively, the parameter $\beta$ is used to measure the lower bound of the average reward difference, $\overline{r}^{\prime}-\overline{r}$. Particularly, when $\beta\rightarrow 0$, the lower bound is a small positive number; and when $\beta\rightarrow 1$, the lower bound tends to be $+\infty$. This means that the average reward difference, $\overline{r}^{\prime}-\overline{r}$, has a high probability, $1-\beta$, to be a small positive number, while has a low probability to be very large. Certainly, the average reward difference, $\overline{r}^{\prime}-\overline{r}$, cannot be $+\infty$. Thus, for the completeness of Theorem 2, as a supplement, Theorem 3 gives the upper bound of this difference, $\overline{r}^{\prime}-\overline{r}$.

According to Theorem 2 and 3, it can be found that the performance of the proposed method is affected by the advice sensitivity $\Delta Q$, and a smaller $\Delta Q$ means a better performance. Since advice sensitivity, $\Delta Q$, is based on the $Q$-values of states, we can conclude that the proposed method can work well in the environments where the $Q$-values of any pair of neighboring states have a small difference.

In Algorithm 1, let $\alpha_{t}$ be the learning rate at time step $t$. Since $\alpha_{t+1}=\frac{t}{t+1}\cdot\alpha_{t}=\frac{t}{t+1}\cdot\frac{t-1}{t}\cdot\alpha_{t-1}=...$, we can conclude that $\alpha_{t}=\frac{1}{t}\alpha_{1}$. Let $t_{i}$ be the index of the $i$th time that action $a$ is performed in state $\boldsymbol{s}$. For example, suppose that the $1$st time that action $a$ is performed in state $\boldsymbol{s}$ is at time step $6$, then $t_{1}=6$. To prove the convergence of Algorithm 1, we need the results of Lemma 4 [25] and Lemma 5 [19].

It should be noted that in Lemma 4, if $|s_{n+1}+...+s_{n+m}|\geq\mu$, the series, $s_{1}+s_{2}+...$, does not converge.

Theorem 4 theoretically proves the convergence of our method, which means that agents using our method can finally receive their expected rewards.

Two experimental scenarios are used to evaluate the proposed differential advising method, denoted as DA-RL (differential advising reinforcement learning).