Reducing Conservativeness Oriented Offline Reinforcement Learning

Reinforcement learning has gained unprecedented attention in recent years, benefiting from the powerful representation and expressive capability of deep neural networks Mnih et al. (2015); Schulman et al. (2017). However, contemporary deep reinforcement learning algorithms generally require millions of samples to achieve expert-level performance. In many real-world applications, such as healthcare and self-driving, trial-and-error collecting samples is costly, time-consuming, and even unsafe. It is becoming a major bottleneck in deep reinforcement learning for the employment in these scenarios Dulac-Arnold et al. (2019).

By utilizing the logged dataset, offline reinforcement learning is one promising solution for the above issues. The policy can be learned without further interaction with the environment. The dataset could be generated by other agents, other algorithms, and even other tasks. For instance, the Atari videos on Youtube can be employed to train the policy to imitate human gameplayAytar et al. (2018). Besides, without the chance to interact with the environment, the algorithm cannot perform hazardous options throughout the training period. However, current reinforcement learning methods easily fail to find the optimal policy due to limited and narrow datasets. Fujimoto et al. (2019) showed offline reinforcement learning suffers from the extrapolation error, which is caused by the optimistic estimation of value target of unseen state-action pairs. Numerous researchers attempt to reduce the extrapolation error by learning the policy in the constrained offline action space. Kumar et al. (2019); Wu et al. (2019) minimized the divergence between the trained policy and the offline policy. Kumar et al. (2020) proposed Conservative Q-Learning (CQL) to learn a lower bound of the expected value function and update the policy according to the conservative value function.

Although these methods use conservative strategies to behave close to the offline distribution, exorbitant conservativeness might impair the policy’s generalization ability, especially in high-dimensional settings. For CQL, learning a lower bound of the value function can suppress the effect of out-of-distribution samples. However, the discrepancy between the lower bound and the real value function might pose challenges to discover optimal actions. Especially for non-expert datasets, optimal policy might deviate far from the offline policy. The conservative lower bound concerns staying in the offline distribution and easily fails to derive the optimal policy.

Prior conservative methods always penalize samples with high uncertainty. Note that the out-of-distribution samples and those samples with small occurrences in the dataset both generally have high uncertainty. However, they are not supposed to be treated equally, especially for the skewed dataset where valuable samples are relatively sparse. Training the policy conservatively in terms of a imbalanced distribution will undermine the agent’s performance. The data imbalance always appears in supervised learning, where one or more categories occupy relatively small proportions of the training data against that of the others. In offline reinforcement learning, there is always a disproportionate percentage of state-action pairs for each component in policy mixtures. The previous methods generally neglect data imbalance problem in offline datasets. Although these methods achieve remarkable results in most D4RL evaluation benchmarks Fu et al. (2020), they fail to perform well as expected in the mixed datasets. Mixed datasets are introduced by various policies, which are prevalent in many applications. For these tasks, it is crucial to reduce the conservativeness by paying more attention to the sparse samples.

In this paper, we propose reducing conservativeness oriented offline reinforcement learning to tackle the issues. On the one hand, we expect to treat the majority and minority samples equally to handle the data imbalance problem. We first provide the samples in the dataset with pseudo class labels by clustering, and then employ resampling and reweighting to train the samples according to each transition’s class frequency. In general, the samples with rare occurrences are assumed to have high uncertainty. Enhancing the weight of these samples in training might introduce more uncertainty and enlarge the extrapolation error. Third, to reduce the extrapolation error resulting from uncertainty preference, we choose to penalize the out-of-distribution samples based on their uncertainty. In this manner, the minority samples are supposed to be attached with more importance while the trust-region of these samples will be relatively cramped. On the other hand, we introduce a tighter lower bound into the value function optimization process, which is able to reduce the conservativeness further. The tighter lower bound is expected to explore the potential optimal actions for the agents, and is crucial when presented with non-expert datasets. We evaluate our proposed method on D4RL Mujoco tasks and our own designed mixed dataset. The experimental results demonstrate that our proposed method outperforms and achieves the competitive performance compared with the state-of-the-art methods on the most tasks.

The offline reinforcement learning methods Lange et al. (2012) focus on training a policy given a fixed dataset. They have been applied in various applications, such as healthcare Gottesman et al. (2018); Wang et al. (2018), recommendation systems Strehl et al. (2010); Swaminathan & Joachims (2015); Covington et al. (2016); Chen et al. (2019), dialogue systemsZhou et al. (2017), and autonomous driving Sallab et al. (2017). In these settings, standard off-policy methods Mnih et al. (2015); Lillicrap et al. (2015) generally fail and exhibit undesirable performance due to their sensitiveness to the dataset distribution Fujimoto et al. (2019). The problem has been studied in approximate dynamic programming Bertsekas & Tsitsiklis (1995); Farahmand et al. (2010); Munos (2003); Scherrer et al. (2015). They ascribed it to the errors arising from distribution shift and function approximation. Recently, Van Hasselt et al. (2018) stated that the temporal difference algorithms would diverge when represented with function approximators and trained under off-policy data.

Some methods have been proposed to deal with the issue by restraining the policy distribution conservatively. Fujimoto et al. (2019) proposed Batch-Constrained Reinforcement Learning (BCQ) to select actions close to the offline actions when formulating Bellman target. Kumar et al. (2019) presented a bootstrapping error accumulation reduction (BEAR). They minimized Maximum Mean Discrepancy (MMD) between the trained and offline policies to alleviate the Bellman update’s error propagation. Jaques et al. (2019) utilized KL-divergence with a regularization weight to penalize the policy. Wu et al. (2019) introduced value penalty and policy penalty to regularize the learned policy towards the behavior policy. These methods tried to penalize the out-of-distribution actions and force the policy to behave in the dataset’s action space. However, these methods ignore the data imbalance problem in the logged dataset. In an optimistic view, we focus on treating the minority classes and the majority classes equally. Our propsoed method is closely related to CQLKumar et al. (2020), which learned a lower bound for the policy’s value function by penalizing the out-of-distribution state-action pairs and maximizing the in-distribution samples’ Q-value. Differently, we encourage the violation from the offline policy by estimating a tighter lower bound for the value function, which is helpful to non-expert tasks. Similarly, Agarwal et al. (2020) used an optimistic strategy and introduced a value function ensemble to stabilize the Q-function updating. Although our method is proposed in terms of reducing the conservativeness, it is essentially a conservative method.

Previous methods tend to penalize samples with high uncertainty. Yu et al. (2020) trained an auxiliary dynamics model to produce imaginary samples to augment the dataset. They reduced a reward for the generated samples according to their uncertainty. Kidambi et al. (2020) learned a pessimistic Markov Decision Process (MDP) using the offline dataset and as well as a near-optimal policy based on this pessimistic MDP. By employing an unknown state-action detector, they used a model-based method to train the policy. However, our proposed method encourages the policy to utilize samples with high uncertainty instead. Furthermore, we impose strict constraints on these sparse samples to reduce extrapolation error.

Current methods of tackling data imbalance can be divided into two categories, i.e., resampling and cost-sensitive learning. Resampling means over-sampling Shen et al. (2016) the minority classes and under-samplingHe & Garcia (2009) the frequent classes. Over-sampling could result in the over-fitting of minority classes, while under-sampling might ignore valuable information in the majority classes. Cost-sensitive learning Wang et al. (2017) assigns different weights for different samples. Most methods reweight samples according to the inverse of the class frequency. Cui et al. (2019) proposed to reweight using the effective number of the classes. In reinforcement learning, Monte Carlo dropout Srivastava et al. (2014) and random network distillation Burda et al. (2018) are common methods to estimate the frequency of each sample. Since offline datasets do not offer class labels for samples, we introduce pseudo labels to the samples using unsupervised learning methods. In this manner, our proposed method is able to perform resampling and reweighting efficiently for different clusters.

In general, a reinforcement learning problem could be formulated as a Markov decision process $(S,A,P,R,\gamma)$, with state space $S$, action space $A$, transition probability matrix $P$, reward function $R$, and discount factor $\gamma$ Sutton & Barto (2018). Each term of $P$ denotes the probability of arriving at the next state $s^{\prime}$ when selecting $a$ at state $s$. At each time step, an agent is located at $s$ and chooses action $a$. The agent would arrive at a next state $s^{\prime}$ and receive a reward $r$ from the environment. The objective of reinforcement learning is to train a policy $\pi(a|s)$ to maximize accumulative rewards in one episode as:

When the agent performs an action $a$ at state $s$ according to a policy $\pi$, the expectation of the cumulative rewards is defined as Q-value function: $Q(s,a)=\mathbb{E}_{\pi}[\sum_{t=0}\gamma^{t}r_{t}|s,a]$. And the value function is defined as $V(s)=\mathbb{E}_{\pi}[Q(s,a)]$. The Q-learning algorithms update the Q-value function according to Bellman operator $\mathcal{T}$ as:

Compared with Q-learning methods, actor-critic methods employ a policy $\pi(a|s)$ to sample actions. The update rule of Q-value function evaluation is $\mathcal{T}^{\pi}Q=r+\gamma P^{\pi}Q$, where $P^{\pi}Q$ is the expectation of Q-value with respective to transition matrix $P$ and policy $\pi$. Since it is impossible to enumerate all states and actions, the Q-value function is generally updated using an empirical Bellman operator $\hat{T}$. Recently, Haarnoja et al. (2018) employed deep neural networks to approximate the actual Q-value function and the policy. On the basis of the neural network, the Q-value loss function is then defined as:

where $\mathcal{D}$ is a replay buffer containing the collected samples. In Eq. 3, $\theta$ is the parameter of the Q-value network and $\omega$ is the parameter of the policy. In this manner, the policy can be updated as:

Offline reinforcement learning suffers from the out-of-distribution challenge. The standard reinforcement learning algorithms might derive a policy that generates out-of-distribution actions when applied in this setting. They have no chance to correct the real Q-value of these actions. Recent works focus on punishing the out-of-distribution actions. Kumar et al. (2020) proposed the CQL method by maximizing the value of in-distribution state-action pairs and minimizing the value of out-of-distribution state-action pairs. The Q-network is updated as:

In Eq. 5, $\alpha$ is a hyperparameter for controlling the degree of conservativeness with respect to the Bellman update, $\hat{Q}$ denotes the empirical Q-value function, and $\pi_{\beta}$ is used to denote the offline policy distribution.

In real-world applications, a large amount of data is generally produced by the skewed distributions. The samples of some classes outnumber those of other classes. In healthcare applications, severe events like hematorrhea are rare but important. In advertisement systems, the number of valuable purchases is much smaller than that of clicks. In this section, we present the discussion of the data imbalance in offline reinforcement learning.

The data imbalance phenomenon exists in the dataset generated by a single policy. Without loss of generality, we assume the transition model of the environment is stochastic in the single policy scenario. A suboptimal stochastic policy might produce a small portion of expert-level trajectories and a large portion of trajectories with poor performances. During the training process, those suboptimal samples dominate over the sparse samples of high quality. In standard reinforcement learning, the temporal difference method samples the experiences uniformly. The expert-level trajectories would be sampled inefficiently, which will impair the reward propagation and undermine the training process. Moreover, the weights of sparse samples might lead to large variances for off-policy evaluation.

Mixed policies are characterized by data imbalance and are ubiquitous in real-world applications. In autonomous driving, samples are produced by different drivers with different driving inclinations. Similarly, in reinforcement learning settings, the agents interact with the environment using the new updated policy at each time step. The collected samples produced by different policies lead to a mixed dataset. The induced skewed dataset consists of samples produced by different policies with different proportions. The large proportion of the samples dominates the training process of the policy compared with that of the small proportion of the samples. In the worst cases, the small proportion of samples from high-quality policies is hard to be utilized.

We examine the performance of the offline algorithm using D4RL benchmarks on the skewed datasets. We study the Mujoco tasks Walker2d and Halfcheetah by mixing the “random” and “expert” dataset with different ratios for evaluation. We create five datasets and all the datasets consist of $10^{6}$ samples for each task. And the proportions of random samples are set as $0.5$, $0.6$, $0.7$, $0.8$, and $0.9$ respectively. Accordingly, the proportions of expert-level samples decline. We train a conservative Q-learning (CQL) agent Kumar et al. (2020) for $10^{6}$ updates and run four seeds for each experiment. The results for these tasks are shown in Fig. 1. As demonstrated in Fig. 1, the performance of the trained policy decreases as the random sample proportion increases. As the above discussion, the current offline reinforcement learning methods struggle to handle the data imbalance problem.

The CQL proposed to learn a lower bound of the expected value function $V(s)$ by minimizing the Q-value of out-of-distribution state-action pairs and maximizing the Q-value of in-distribution state-action pairs. In the setting without approximation error, the gap between the value function derived by CQL and expected value function is the Pearson $\chi^{2}$ divergence between $\pi_{\beta}(a|s)$ and $\pi(a|s)$. The gap decreases to $0$ when $\pi(a|s)=\pi_{\beta}(a|s),\forall a$. We could obtain a baseline $\delta(s)$ which is the minimum of Pearson $\chi^{2}$ divergence for $s$ and a threshold as:

where $\Delta$ is the tolerance threshold of the violation of the trained policy against the offline policy $\pi_{\beta}(a|s)$. When $\Delta$ decreases, the algorithm tends to behave more conservatively. When $\Delta$ increases, the policy is allowed to deviate further from the dataset. In this manner, the policy is able to preserve the opportunity to balance stability and generalization ability. The objective of our proposed method is then defined as:

We assume that $C$ is a constant dependent on the reward function and transition matrix, $R_{\max}=\max|r(s,a)|$ and $|\mathcal{D}|$ denotes a vector of counts for all states. We show that the empirical value function $\hat{V}^{\pi}(s)$ is the lower-bound of the actual value function.

Thus, if $\alpha>\frac{CR_{\max}}{1-\gamma}\cdot\max_{s\in\mathcal{D}}\frac{1}{\mid\sqrt{|\mathcal{D}(s)|}}\cdot\left[\sum_{a}\pi(a\mid s)(\frac{\pi(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}-(1+\delta(s)))\right]^{-1},\forall s\in\mathcal{D}$, we have $\hat{V}^{\pi}(s)\leq V^{\pi}(s)$ with high probability. When $\hat{\mathcal{T}}^{\pi}=\mathcal{T}^{\pi}$, then any $\alpha>0$ guarantees $\hat{V}^{\pi}(s)\leq V^{\pi}(s),\forall s\in\mathcal{D}$.

In this section, we introduce our proposed reducing conservativeness oriented offline reinforcement learning. We first perform clustering on the dataset and providing the samples with pseudo labels. We then employ resampling and reweighting to associate the minor classes with greater importance. In order to reduce extrapolation error resulting from high uncertainty, we impose regularization constraints on the minority classes. Finally, we introduce our proposed tighter lower bound to the value function update.

In this paper, we introduce the method of reducing conservativeness oriented offline reinforcement learning. After partitioning the samples into different clusters, we reweight and resample them according to their class frequency. To reduce extrapolation error, we put more constraints on those minority classes. Besides, we introduce a tighter lower bound of the value function. Our experimental results show that our method outperforms state-of-art approaches on D4RL evaluation benchmarks and our designed mixed datasets.