A Benchmark for Low-Switching-Cost Reinforcement Learning

Reinforcement Learning (RL) has been successfully applied to solve sequential-decision problems in many real-world scenarios, such as medical domains [16], robotics [7, 12], hardware placements [20, 19], and personalized recommendation [28]. When performing RL training in these scenarios, it is often desirable to restrict the agent from adjusting its policy too often due to the high costs and risks of deployment. For example, updating a treatment strategy in medical domains requires a thorough approval process by human experts [2]; in personalized recommendations, it is impractical to adjust the policy online based on instantaneous data and a more common practice is to aggregate data in a period before deploying a new policy [28]; in problems where we use RL to learn hardware placements [20], it is desirable to limit the frequency of changes to the policy since it is costly to launch a large-scale evaluation procedure on hardware devices like FPGA; for learning to control a real robot, it may be risky or inconvenient to switch the deployed policy when an operation is being performed. In these settings, it is a requirement that the deployed policy, i.e., the policy used to interact with the environment, could keep unchanged as much as possible. Formally, we would like our RL algorithm to both produce a policy with the highest reward and at the same time reduce the number of deployed policy switches (i.e., a low switching cost) throughout the training process.

Offline reinforcement learning [15] is perhaps the most related framework in the existing literature that also has a capability of avoiding frequent policy deployment. Offline RL assumes a given transition dataset and performs RL training completely in an offline fashion without interacting with the environment at all. [18] adopt a slightly weaker offline assumption by repeating the offline training procedure, i.e., re-collecting transition data using the current policy and re-applying offline RL training on the collected data, for about 10 times. However, similar to the standard offline RL methods, due to such an extreme low-deployment-constraint, the proposed solution suffers from a particularly low sample efficiency and even produces significantly lower rewards than online SAC method in many cases [18]. In contrast with offline RL, which optimizes the reward subject to a minimal switching cost, low-switching-cost RL aims to reduce the switching cost while maintain a similar sample efficiency and final reward compared to its unconstrained RL counterpart.

In low-switching-cost RL, the central question is how to design a criterion to decide when to update the deployed policy based on the current training process. Ideally, we would like this criterion to satisfy the following four properties:

1. 1.

   Low switching cost: This is the fundamental mission. An RL algorithm equipped with this policy switching criterion should have a reduced frequency to update the deployed policy.

2. 2.

   High reward: Since the deployed policy can be different from the training one, the collected data can be highly off-policy. We need this criterion to deploy policies at the right timing so that the collected samples can be still sufficient for finally achieving the optimal reward.

3. 3.

   Sample efficiency: In addition to the final reward, we also would like the algorithm equipped with such a criterion to produce a similar sample efficiency to the unconstrained RL setting without the low-switching-cost requirement.

4. 4.

   Generality: We would like this criterion can be easily applied to a wide range of domains rather than a specific task.

From the theoretical side, low-switching-cost RL and its simplified bandit setting have been extensively studied [3, 5, 4, 22, 6, 26, 27]. The core notion in these theoretical methods is information gain. Specifically, they update the deployed policy only if the measurement of information gain is doubled, which also leads to optimality bounds for the final policy rewards. We suggest the readers refer to the original papers for details of the theoretical results. We will also present algorithmic details later in Section 4.4.

However, to our knowledge, there has been no empirical study on whether these theoretically-guided criteria are in fact effective in popular RL testbeds. In this paper, we aim to provide systematic benchmark studies on different policy switching criteria from an empirical point of view. We remark that our experiment scenarios are based on popular deep RL environments that are much more complex than the bandit or tabular cases studied in the existing theoretical works. We hope this benchmark project could serve as a starting point towards practical solutions that eventually solve those real-world low-switching-cost applications. Our contributions are summarized below.

Low switching cost algorithms were first studied in the bandit setting  [3, 5]. Existing work on RL with low switching cost is mostly theoretical. To our knowledge, [4] is the first work that studies this problem for the episodic finite-horizon tabular RL setting. [4] gave a low-regret algorithm with an $O\left(H^{3}SA\log\left(K\right)\right)$ local switching upper bound where $S$ is the number of stats, $A$ is the number of actions, $H$ is the planning horizon and $K$ is the number of episodes the agent plays. The upper bound was improved in [27, 26].

Offline RL (also called Batch RL) can be viewed as a close but parallel variant of low-switching-cost RL, where the policy does not interact with the environment at all and therefore does not incur any switching cost. Offline RL methods typically learn from a given dataset [14, 15], and have been applied to many domains including education [17], dialogue systems [11] and robotics control [13]. Some methods interpolate offline and online methods, i.e., semi-batch RL algorithms [23, 14], which update the policy many times on a large batch of transitions. However, reducing the switching cost during training is not their focus. [18] developed the only empirical RL method that tries to reduce the switching cost without the need of a given offline dataset. Given a fixed number of policy deployments (i.e., 10), the proposed algorithm collects transition data using a fixed deployed policy, trains an ensemble of transition models and updated a new deployed policy via model-based RL for the next deployment iteration. However, even though the proposed model-based RL method in [18] outperforms a collection of offline RL baselines, the final rewards are still substantially lower than standard online SAC even after consuming an order of magnitude more training samples. In our work, we focus on the effectiveness of the policy switching criterion such that the overall sample efficiency and final performances can be both preserved. In addition to offline RL, there are also some other works that aim to reduce the interaction frequency with the environment rather than the switching cost [7, 10], which is parallel to our focus.

Markov Decision Process: We consider the Markov decision model $(\mathcal{S},\mathcal{A},\gamma,r,p_{0},P)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\gamma$ is the discounted factor, $r:\mathcal{S}\ \times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function, $p_{0}$ is the initial state distribution, and $P(x^{\prime}|x,a)$ denotes the transition probability from state $x$ to state $x^{\prime}$ after taking action $a$. A policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$ is a mapping from a state to an action, which can be either deterministic or stochastic. An episode starts with an initial state $x_{0}\sim p_{0}$. At each step $h$ in this episode, the agent chooses an action $a_{h}$ from $\pi(x_{h})$ based on the current state $x_{h}$, receives a reward $r(x_{h},a_{h})$ and moves to the next state $x_{h+1}\sim P(\cdot|x_{h},a_{h})$. We assume an episode will always terminate, so each episode $e=\{(x_{h}^{e},a_{h}^{e})|0\leq h\leq H_{e}\}$ will always have a finite horizon $H_{e}$ (e.g., most practical RL environments have a maximum episode length $H_{\max}$). The goal of the agent is to find a policy $\pi^{*}$ which maximizes the discounted expected reward, $\pi^{\star}=\arg\max_{\pi}\mathbb{E}_{e}\left[\sum_{h=0}^{H_{e}}\gamma^{h}r(x^{e}_{h},a^{e}_{h})\right]$. Let $K$ denote the total transitions that the agent experienced across all the episodes during training. Ideally, we also want the agent to consume as few training samples as possible, i.e., a minimal $K$, to learn $\pi^{\star}$. A Q-function is used to evaluate the long-term value for the action $a$ and subsequent decisions, which can be defined w.r.t. a policy $\pi$ by

Deep Off-policy Reinforcement Learning: Deep Q-learning (DQN) [21] is perhaps the most popular off-policy RL algorithm leveraging a deep neural network to approximate $Q(x,a)$. Given the current state $x_{h}$, the agent selects an action $a_{h}$ greedily based on parameterized Q-function $Q_{\theta}(x_{h},a)$ and maintain all the transition data in the replay buffer.For each training step, the temporal difference error is minimized over a batch of transitions sampled from this buffer by

where $\bar{\theta}$ represents the parameters of the target Q-network, which is periodically updated from $\theta$. Rainbow [9] is perhaps the most famous DQN variant, which combines six algorithmic enhancements and achieves strong and stable performances on most Atari games. In this paper, we adopt a deterministic version²²2Standard Rainbow adds random noise to network parameters for exploration, which can be viewed as constantly switching policies over a random network ensemble. This contradicts the low-switching-cost constraint. of Rainbow DQN as the RL algorithm for the discrete action domains. We also adopt count-based exploration [24] as a deterministic exploration bonus.

For continuous action domains, soft actor-critic (SAC) [8] is the representative off-policy RL algorithm. SAC uses neural networks parameterized by $\theta$ to approximate both $Q(s,a)$ and the stochastic policy $\pi_{\theta}(a|s)$. $Q$-network is trained to approximate entropy-regularized expected return by minimizing

where $\alpha$ is the entropy coefficient. We omit the parameterization of $\pi$ since $\pi$ is not updated w.r.t $\mathcal{L}_{Q}$. The policy network $\pi_{\theta}$ is trained to maximize $\mathcal{L}_{\pi}(\theta)=\mathbb{E}_{a\sim\pi}\left[Q(x,a)-\alpha\log\pi_{\theta}(a|x)\right]$.

In standard RL, a transition $(x_{h},a_{h},x_{h})$ is always collected by a single policy $\pi$. Therefore, whenever the policy is updated, a switching cost is incurred. In low-switching-cost RL, we have two separate policies, a deployed policy $\pi_{\textrm{dep}}$ that interacts with the environment, and an online policy $\pi_{\textrm{onl}}$ that is trained by the underlying RL algorithm. These policies are parameterized by $\theta_{\textrm{dep}}$ and $\theta_{\textrm{onl}}$ respectively. Suppose that we totally collect $K$ samples during the training process, then at each transition step $k$, the agent is interacting with the environment using a deployed policy $\pi_{\textrm{dep}}^{k}$. After the transition is collected, the agent can decide whether to update the deployed $\pi_{\textrm{dep}}^{k+1}$ by the online policy $\pi_{\textrm{onl}}^{k+1}$, i.e., replacing $\theta_{\textrm{dep}}$ with $\theta_{\textrm{onl}}$, according to some switching criterion $\mathcal{J}$. Accordingly, the switching cost is defined by the number of different deployed policies throughout the training process, namely:

The goal of low-switching-cost RL is to design an effective algorithm that learns $\pi^{*}$ using $K$ samples while produces the smallest switching cost $C_{\textrm{switch}}$. Particularly in this paper, we focus on the design of the switching criterion $\mathcal{J}$, which is the most critical component that balances the final reward and the switching cost. The overall workflow of low-switching-cost RL is shown in Algorithm 1.

In the following content, we present a collection of policy switching criteria and techniques, including those inspired by the information gain principle (Sec. 4.4) as well as other non-adaptive (Sec. 4.1) and adaptive criteria (Sec. 4.2,4.3). All the discussed criteria are summarized in Algorithm 2.

In this section, we conduct experiments to evaluate different policy switching criteria on Rainbow DQN and SAC. For discrete action spaces, we study the Atari games and the GYMIC testbed for simulating sepsis treatment for ICU patients which requires low switching cost. For continuous control, we conduct experiments on the MuJoCo [25] locomotion tasks.

In this paper, we focus on low-switching-cost reinforcement learning problems and take the first empirical step towards designing an effective solution for reducing the switching cost while maintaining good performance. By systematic empirical studies on practical benchmark environments with modern RL algorithms, we find the existence of a theory-practice gap in policy switching criteria and suggest a feature-based solution can be preferred in practical scenarios.

We remark that most theoretical works focus on simplified bandits or tabular MDPs when analysing mathematical properties of information-gain-based methods, but these theoretical insights are carried out on much simplified scenarios compared with popular deep RL testbeds that we consider here. In addition, these theoretical analyses often ignore the constant factors, which can be crucial for the practical use. By contrast, feature-based methods are much harder to analyze theoretically particularly in the use of non-linear approximators, which, however, produces much better practical performances. This also raises a new research direction for theoretical studies. We emphasize that our paper not only provides a benchmark but also raises many interesting open questions for algorithm development. For example, although feature-based solution achieves the best overall performance, it does not substantially outperform the the naive non-adaptive baseline. It still has a great research room towards designing a more principled switching criteria. Another direction is to give provable guarantees for these policy switching criteria that work for methods dealing with large state space in contrast to existing analyses about tabular RL [4, 27, 26]. We believe our paper is just the first step on this important problem, which could serve as a foundation towards great future research advances.

Thanks to the strong research nature of this work, we believe our paper does not produce any negative societal impact.