Reinforcement Learning with Delayed, Composite, and Partially Anonymous Reward

Reinforcement learning (RL) enables an agent to learn a policy in an unknown environment by repeatedly interacting with it. The environment has a state that changes as a result of the actions executed by the agent. In this setup, a policy is a collection of rules that guides the agent to take action based on the observed state of the environment. Several algorithms exist in the literature that learn a policy with near-optimal regret (Jaksch et al., 2010; Azar et al., 2017; Jin et al., 2018; Fruit et al., 2018). However, all of the above frameworks assume that the reward is instantly fed back to the learner after executing an action. Unfortunately, this assumption of instantaneity may not hold in many practical scenarios. For instance, in the online advertisement industry, a customer may purchase a product several days after seeing the advertisement. In medical trials, the effect of a medicine may take several hours to manifest. In road networks, it might take a few minutes to notice the impact of a traffic light change. In many of these cases, the effect of an action is not entirely realized in a single instant, but rather fragmented into smaller components that are sequentially materialized over a long interval. Reward feedback that satisfies this property is referred to as delayed and composite reward. In several applications, the learner cannot directly observe each delayed component of a composite reward but only an aggregate of the components realized at the observation instance. For example, in the case of multiple advertisements, the advertiser can only see the total number of purchases at a given time but is completely unaware of which advertisement resulted in what fraction of the total purchase. Such feedback is referred to as anonymous reward.

Learning with delayed, composite, and anonymous rewards is gaining popularity in the RL community. While most of the theoretical analysis has been directed towards the multi-arm bandit (MAB) framework (Wang et al., 2021; Zhou et al., 2019; Vernade et al., 2020; Pike-Burke et al., 2018; Pedramfar and Aggarwal, 2023), recent studies have also analyzed Markov Decision Processes (MDPs) with delayed feedback (Howson et al., 2023; Jin et al., 2022; Lancewicki et al., 2022). However, none of these studies have incorporated both composite and anonymous rewards, which is the focus of this paper.

We consider an infinite-horizon average-reward Markov Decision Process (MDP) defined as, $M\triangleq\{\mathcal{S},\mathcal{A},r,p\}$ where $\mathcal{S},\mathcal{A}$ denote the state, and action spaces respectively, $r:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ is the reward function, and $p:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ indicates the state transition function. The function, $\Delta(\cdot)$ defines the probability simplex over its argument set. The cardinality of the sets $\mathcal{S}$ and $\mathcal{A}$ are denoted as $S,A$ respectively. Both the reward function, $r$, and the transition function, $p$ are assumed to be unknown.

A learner/agent interacts with an environment governed by the MDP stated above as follows. The interaction proceeds in discrete time steps, $t\in\{1,2,\cdots\}$. At the time $t$, the state occupied by the environment is denoted as $s_{t}\in\mathcal{S}$. The learner observes the state, $s_{t}$, and chooses an action $a_{t}\in\mathcal{A}$ following a predefined protocol, $\mathbb{A}$. As a result, a sequence of rewards $\boldsymbol{r}_{t}(s_{t},a_{t})\triangleq\{r_{t,\tau}(s_{t},a_{t})\}_{\tau=0}^{\infty}$ is generated where $r_{t,\tau}(s_{t},a_{t})$ is interpreted as the non-negative component of the vector $\boldsymbol{r}_{t}(s_{t},a_{t})$ that is realised at instant $t+\tau$. The following assumption is made regarding the reward generation process.

Assumption 1(a) dictates that the reward sequences generated from $(s,a)$ are such that the sum of their components can be thought of as samples taken from a certain distribution, $D(s,a)$, over $[0,1]$. Note that the distribution, $D(s,a)$, is independent of $t$. Assumption 1(b) explains that the expected value of the sum of the reward components generated from $(s,a)$ equals $r(s,a)\in[0,1]$. Finally, Assumption 1(c) clarifies that the reward sequences generated at different instances (either by the same or distinct state-action pairs) are presumed to be independent of each other.

At the time instant $t$, the learner observes a reward vector $\boldsymbol{x}_{t}\in\mathbb{R}^{S}$ whose $s$-th element is expressed as follows.

where $\mathrm{1}(\cdot)$ defines the indicator function. Note that the learner only has access to the lump-sum reward $\boldsymbol{x}_{t}(s)$, not its individual components contributed by past actions. This explains why the reward is termed as partially anonymous. In a fully anonymous setting, the learner can only access $||\boldsymbol{x}_{t}||_{1}$, not its elements. Although full anonymity might be desirable for many practical scenarios, from a theoretical standpoint, it is notoriously difficult to analyze. We discuss this topic in detail in section 4.2. The expected accumulated reward generated up to time $T$ can be computed as,

We would like to emphasize that $R(s_{1},\mathbb{A},M,T)$ is not the sum of the observed rewards up to time $T$. Rather, it equates to the sum of all the reward components that are generated as a consequence of the actions taken up to time $T$. We define the quantity expressed below

as the average reward of the MDP $M$ for a given protocol $\mathbb{A}$ and an initial state $s_{1}\in\mathcal{S}$. Here the expectation is obtained over all possible $T$-length trajectories generated from the initial state $s_{1}$ following the protocol $\mathbb{A}$ and the randomness associated with the reward generation process for any given state-action pair. It is well known that (Puterman, 2014) there exists a stationary deterministic policy $\pi^{*}:\mathcal{S}\rightarrow\mathcal{A}$ that maximizes the average reward $\forall s_{1}\in\mathcal{S}$ if $D(M)$, the diameter of $M$ (defined below) is finite. Also, in that case, $\rho(s_{1},\pi^{*},M)$ becomes independent of $s_{1}$ and thus can be simply denoted as $\rho^{*}(M)$. The diameter $D(M)$ is defined as follows.

where $T(s^{\prime}|s,\pi,M)$ denotes the time needed for the MDP $M$ to reach the state $s^{\prime}$ from the state $s$ following the stationary deterministic policy, $\pi$. Mathematically, $\mathrm{Pr}(T(s^{\prime}|s,\pi,M)=t)\triangleq\mathrm{Pr}(s_{t}=s|s_{1}=s,s_{\tau}\neq s,1<\tau<t,s_{\tau}\sim p(s_{\tau-1},a_{\tau-1}),a_{\tau}\sim\pi(s_{\tau})$). In simple words, given two arbitrary distinct states, one can always find a stationary deterministic policy such that the MDP, $M$, on average, takes at most $D(M)$ time steps to transition from one state to the other.

We define the performance of a protocol, $\mathbb{A}$ by the regret it accumulates over a horizon, $T$ which is mathematically expressed as,

where $\rho^{*}(M)$ is the maximum of the average reward given in $(\ref{eq_avg_reward})$, and the second term is defined in $(\ref{eq_cum_reward})$. We would like to mention that in order to define regret, we use $R(s_{1},\mathbb{A},M,T)$ rather than the expected sum of the observed rewards up to time $T$. The rationale behind this definition is that all the components of the rewards that are generated as a consequence of the actions taken up to time $T$ would eventually be realized if we allow the system to evolve for a long enough time. Our goal in this article is to come up with an algorithm that achieves sublinear regret for the delayed, composite, and anonymous reward MDP described above.

Before concluding, we would like to provide an example of an MDP with partially anonymous rewards. Let us consider the $T$ round of interaction of a potential consumer with a website that advertises $S$ categories of products. At round $t$, the state, $s_{t}\in\{1,\cdots,S\}$ observed by the website is the category of product searched by the consumer. The $s$-th category where $s\in\{1,\cdots,S\}$ has $N_{s}$ number of potential advertisements, and the total number of advertisements is $N=\sum_{s}N_{s}$. The website, in response to the observed state, $s_{t}$, shows an ordered list of $K<N$ advertisements (denoted by $a_{t}$), some of which may not directly correspond to the searched category, $s_{t}$. This may cause the consumer, with some probability, to switch to a new state, $s_{t+1}$ in the next round. For example, if the consumer is searching for “Computers”, then showing advertisements related to “Mouse” or “Keyboard” may incentivize the consumer to search for those categories of products. After $\tau$ rounds, $0<\tau\leq T$, the consumer ends up spending $r_{\tau}(s)$ amount of money for the $s$-th category of product as a consequence of previous advertisements. Note that if the same state $s$ appears in two different rounds $t_{1}<t_{2}<\tau$, the website can potentially show two different ordered lists of advertisements, $a_{t_{1}},a_{t_{2}}$ to the consumer. However, it is impossible to segregate the portions of the reward $r_{\tau}(s)$ contributed by each of those actions. Hence, the system described above can be modeled by an MDP with delayed, composite, and partially anonymous rewards.

In this section, we develop Delayed UCRL2 (DUCRL2) algorithm to achieve the overarching goal of our paper. It is inspired by the UCRL2 algorithm suggested by (Jaksch et al., 2010) for MDPs with immediate rewards (no delay). Before going into the details of the DUCRL2 algorithm, we would like to introduce the following assumption.

Note that if the maximum delay is $d_{\max}$, then using Assumption 1(a), one can show that $d\leq d_{\max}$. Therefore, $d$ can be thought of as a proxy for the maximum delay. To better understand the intuition behind Assumption 2, consider an interval $\{1,\cdots,T_{1}\}$. Clearly, the reward sequence generated at $t_{1}=T_{1}-\tau_{1}$, $\tau_{1}\in\{0,\cdots,T_{1}-1\}$, is $\mathbf{r}_{t_{1}}(s_{t_{1}},a_{t_{1}})$. The portion of this reward that is realized after $T_{1}$ is expressed by the following quantity: $\sum_{\tau\geq\tau_{1}}r_{t_{1},\tau}(s_{t_{1}},a_{t_{1}})$ which is upper bounded by $\max_{(s,a)}\sum_{\tau\geq\tau_{1}}r_{t_{1},\tau}(s,a)$. Therefore, the total amount of reward that is generated in $\{1,\cdots,T_{1}\}$ but realized after $T_{1}$ is bounded by $\sum_{\tau_{1}\geq 0}\max_{(s,a)}\sum_{\tau\geq\tau_{1}}r_{t_{1},\tau}(s,a)$. As the distribution of the reward sequence $\mathbf{r}_{t_{1}}$ is the same for all $t_{1}$ (Assumption 1(a)), one can replace the $\tau_{1}$ dependent term $t_{1}$ with a generic quantity, $t$. Thus, Assumption 2 essentially states that the spillover of the rewards generated within any finite interval $\{1,\cdots,T_{1}\}$ can be bounded by the term $d$.

We would also like to emphasize that if $d_{\max}$ is infinite, then $d$ may or may not be infinite depending on the reward sequence. For example, consider a deterministic sequence whose components are $r_{t,\tau}(s,a)=2^{-1-\tau}$, $(s,a)\in\mathcal{S}\times\mathcal{A}$, $\forall t\in\{1,2,\cdots\}$, $\forall\tau\in\{0,1,\cdots\}$. It is easy to show that one can choose $d=2$ though $d_{\max}$ is infinite. On the other hand, if $r_{t,\tau}(s,a)=\mathbf{1}(\tau=\tau^{\prime})$, $\forall(s,a)$, $\forall t,\forall\tau$ where $\tau^{\prime}$ is a random variable with the distribution $\mathrm{Pr}(\tau^{\prime}=k)=(1-p)p^{k}$, $\forall k\in\{0,1,\cdots\}$, $0<p<1$, then one can show that (4) is violated with probability at least $p^{d}$. In other words, Assumption 2 is not satisfied for any finite $d$.

The $\mathrm{DUCRL2}$ algorithm (Algorithm 1) proceeds in multiple epochs. At the beginning of epoch $k$, i.e., at time instant $t=t_{k}$, we compute two measures for all state-action pairs. The first measure is indicated as $N_{k}(s,a)$ which counts the number of times the pair $(s,a)$ appears before the onset of the $k$th epoch. The second measure is $\mathcal{E}_{j}(s,a)$ which is a binary random variable that indicates whether the pair $(s,a)$ appears at least once in the $j$th epoch, $0<j<k$. Due to the very nature of our algorithm (elaborated later), in a given epoch $j$, if $\mathcal{E}_{j}(s,a)=1$, then $\mathcal{E}_{j}(s,a^{\prime})=0$, $\forall a^{\prime}\in\mathcal{A}\setminus\{a\}$. Taking a sum over $\{\mathcal{E}_{j}(s,a)\}_{0<j<k}$, we obtain $E_{k}(s,a)$ which counts the number of epochs where $(s,a)$ appears at least once before $t_{k}$, the start of the $k$th episode.

Next, we obtain the reward estimate $\hat{r}_{k}(s,a)$ by computing the sum of the $s$-th element of the observed reward over all epochs where $(s,a)$ appears at least once and dividing it by $N_{k}(s,a)$. Also, the transition probability estimate $\hat{p}_{k}(s^{\prime}|s,a)$ is calculated by taking a ratio of the number of times the transition $(s,a,s^{\prime})$ occurs before the $k$th episode and $N_{k}(s,a)$. It is to be clarified that, due to the delayed composite nature of the reward, the observed reward values that are used for computing $\hat{r}_{k}(s,a)$ may be contaminated by the components generated by previous actions which could potentially be different from $a$. Consequently, it might appear that the empirical estimates $\hat{r}_{k}(s,a)$ may not serve as a good proxy for $r(s,a)$. However, the analysis of our algorithm exhibits that by judiciously steering the exploration, it is still possible to obtain a near-optimal regret.

Using the estimates $\hat{r}_{k}(s,a)$, $\hat{p}_{k}(\cdot|s,a)$, we now define a confidence set $\mathcal{M}_{k}$ of MDPs that is characterized by $(\ref{eq_4})$, $(\ref{eq_5})$. The confidence radius given in $(\ref{eq_4})$ is one of the main differences between our algorithm and the UCRL2 algorithm given by (Jaksch et al., 2010). Applying extended value iteration (Appendix A), we then derive a policy $\tilde{\pi}_{k}$, and an MDP $\tilde{M}_{k}\in\mathcal{M}_{k}$ that are near-optimal within the set $\mathcal{M}_{k}$ in the sense of $(\ref{eq_6})$. We keep on executing the policy $\tilde{\pi}_{k}$ until for at least one state-action pair $(s,a)$, its total number of occurrences within the current epoch, $\nu_{k}(s,a)$ becomes at least as large as $N_{k}(s,a)$, the number of its occurrences before the onset of the current epoch. When this criterion is achieved, a new epoch begins and the process described above starts all over again. Observe that, as the executed policies are deterministic, no two distinct pairs $(s,a),(s,a^{\prime})$ can appear in the same epoch for a given state, $s$.

We would like to conclude with the remark that all the quantities used in our algorithm can be computed in a recursive manner. Consequently, similar to UCRL2, the space complexity of our algorithm turns out to be $\mathcal{O}(S^{2}A)$ which is independent of $T$.

Below we state our main result.

Theorem 1 states that the regret accumulated by algorithm $\mathrm{DUCRL2}$ is $\tilde{\mathcal{O}}(DS\sqrt{AT}+d(SA)^{3})$ where $\tilde{\mathcal{O}}(\cdot)$ hides the logarithmic factors. (Jaksch et al., 2010) showed that the lower bound on the regret is $\Omega(\sqrt{DSAT})$. As our setup is a generalization of the setup considered in (Jaksch et al., 2010), the same lower bound must also apply to our model. Moreover, if we consider an MDP instance where the rewards arrive with a constant delay, $d$, then in the first $d$ steps, due to lack of any feedback, each algorithm must obey the regret lower bound $\Omega(d)$. We conclude that the regret lower bound of our setup is $\Omega(\max\{\sqrt{DSAT},d\})=\Omega(\sqrt{DSAT}+d)$. Although it matches the orders of $T$, and $d$ of our regret upper bound, there is still room for improvement in the orders of $D$, $S$ and $A$.

In this work, we addressed the challenging problem of designing learning algorithms for infinite-horizon Markov Decision Processes with delayed, composite, and partially anonymous rewards. We propose an algorithm that achieves near-optimal performance and derive a regret bound that matches the existing lower bound in the time horizon while demonstrating an additive impact of delay on the regret. Our work is the first to consider partially anonymous rewards in the MDP setting with delayed feedback.

Possible future work includes extending the analysis to the more general scenario of fully anonymous, delayed, and composite rewards, which has important applications in many domains. This extension poses theoretical challenges, and we believe it is an exciting direction for future research. Overall, we hope our work provides a useful contribution to the reinforcement learning community and inspires further research on this important topic.