The Least Restriction for Offline Reinforcement Learning

One of the main reasons behind the success of deep supervised learning [28] is the availability of large and diverse datasets such as the ImageNet [20] to train expressive deep neural networks. By contrast, almost all the RL algorithms assume that every agent has to interacts with an online environment (i.e. a real world environment or an artificial simulator environment) [14]. In this way, the agent collects its own experience for training the actor network and the critic network. Unfortunately, active data collection in the real world (autonomous driving [15], healthcare [16], etc.) would be expensive and unsafe. Moreover, building a high-fidelity simulator is not easy too.

Offline RL [14, 15] concerns the problem of learning a policy from a fixed dataset of trajectories, without any further interactions with the environment. This setting could leverage the vast amount of existing logged interactions for real world decision-making problems, like robotics [17], recommender systems [18], and dialogues [19]. The effective use of such datasets would not only make the real world RL more practical, but would also enable better generation by incorporating diverse prior experience.

In offline RL, an agent does not receive any new corrective feedback from the online environment. And the agent needs to generalize from a fixed dataset to new online environment during evaluation. In principle, off-policy RL algorithms could learn from data collected by any (unknown) policy. Nonetheless, recent work [26, 27] proposes a discouraging view that standard off-policy deep RL algorithms diverge or otherwise yield poor performance in the offline setting.

The rest of this paper is organized as the following: Section 2 explains the background of the offline RL. Section 3 analyzes the reason of why the online algorithms failing in the offline dataset. Then, Section 4 describes a creative offline RL framework, the LR, in details. Finally, Section 5 has a summary discussion.

At this section, some necessary prior knowledge for offline RL will be introduced.

Off-policy RL algorithms, which are with the critic for estimating the action-value function $Q(s,a)$, almost fail to learn on a fixed offline dataset. [1] and [2] demonstrate that, this failure is not caused by the lack of the $<s,a,r,s’>$ transition record, but the error from the $Q(s,a)$ bootstrapping. The source of this instability could be understood by examining the form of the equations for iterating the $Q(s,a)$. Although minimizing the mean squared error corresponds to a supervised regression problem, the targets $Q(s,a)$ for this regression are themselves derived from the current $Q(s^{\prime},a^{\prime})$ estimate. The targets $Q(s,a)$ are calculated by maximizing the learned $Q(s^{\prime},a^{\prime})$ with respect to the action $a^{\prime}$ at the next state $s^{\prime}$. However, the $Q(s,a)$ estimator is only reliable on inputs from the same distribution as its training set.

Have a review on the Equation (2) and (3). For estimating the value of $Q(s,a)$, all the values of $Q(s’,a’),\forall a^{\prime}\in\mathcal{A}$ must be estimated reliably. And for estimating the value of $Q(s’,a’)$, all the values of $Q(s^{\prime\prime},a^{\prime\prime})$ must be estimated reliably. And so on and so forth…… Finally, the correct value of $Q(s_{next-to-last},a_{last})$ is vitally needed. In the online setting, the agent is easily able to transfer from $<s,a,r,s^{\prime}>$ to $<s^{\prime},a^{\prime},r^{\prime},s^{\prime\prime}>$. Then, from $<s^{\prime},a^{\prime},r^{\prime},s^{\prime\prime}>$ to $<s^{\prime\prime},a^{\prime\prime},r^{\prime\prime},s^{\prime\prime\prime}>$. And so on and so forth…… End at the final state $<s_{next-to-last},a_{last},r_{last},s_{final}>$.

Unfortunately, the offline dataset is fixed, without any opportunity to supply new data. Hence, at the offline setting, when transfer from $<s,a,r,s^{\prime}>$ to $<s^{\prime},a^{\prime},r^{\prime},s^{\prime\prime}>$, many $a’\in\mathcal{A}$ are not in this dataset. These actions, which are missing or appear only a little times, are defined as $\textbf{Rare Actions}\ a_{few}$ in this paper.

In other words, at the state $s’$, selecting the action $a’_{few}$, what reward $r^{\prime}$ will get and which next state $s^{\prime\prime}$ will arrive, are all unknown, i.e. $<s’,a’_{few},?,?>$. Then, the value of $Q(s’,a’_{few})$ always keep the initial random value. As a result, naïve maximizing the $Q(s’,a’),\forall a^{\prime}\in\mathcal{A}$ usually absorb the wrong value. Furthermore, the error spread through the Bellman backup, like Equation (2). Naturally, based on the critic with wrong $Q(s,a)$ (see Equation (4) ), the actor could not perform optimally.

Overall, the key for offline RL is that preventing the Rear Actions $a_{few}$ interfering the action-value $Q(s,a)$ backups. Previous works [4], [11], [12] explicitly constrain the learned policy $\pi$ to be not far from the behavior policy $\beta$, similarly to behavior cloning. While this is enough to ensure that actions lie in the fixed dataset with high probability, it is overly restrictions. For example, if the behavior policy is close to uniform distribution, the learned policy will behave randomly too, resulting in poor performance, even the data is quite sufficient. The next section will propose a more flexible frameworks for offline RL, though.

At standard online RL, there is an interesting phenomenon that, the off-policy algorithms [6, 7, 8], whose behavior policy (also called exploration policy) $\beta$ is different from the learned policy (also called target policy) $\pi$. But they could succeed in training optimal policies. The reason is that $\pi$ is not same as $\beta$ though, they are quite close, only vary in a random noise or $\epsilon$-greedy.

Accordingly, most state-of-the-art offline RL methods force $\pi$ near the $\beta$. The earliest proposed offline RL method BCQ trains a VAE to simulate the $\beta$ distribution. When the action-value function $Q(s,a)$ is updated, the action selection depends on this VAE.

Besides straightly make $\pi$ close to $\beta$, later researchers try to restrict the “distance” between the $\pi$ distribution and the $\beta$ distribution. Under state $s$, the Rare Action $a_{few}$ one-to-one corresponds to the point, which has a low probability density at $\beta(a|s)$. Avoiding Rare Action means that a learned policy $\pi(a|s)$ has positive density only where the density of the behavior policy $\beta(a|s)$ is more than a threshold, i.e.:

Based on the above analysis, we propose a creative offline RL framework, the Least Restriction (LR) Framework, which is able to combine with almost all online algorithms.

The LR Framework firstly train a Generative Adversarial Network (GAN) [10] to simulate the dataset collected by $\beta$. For simplicity, the GAN only considers the state-action pair $(s,a)$. After the training completed, the generator of the GAN is very similar with $\beta$ and the discriminator can provide a confidence degree of a $(s,a)$ pair whether belongs to the dataset.

In fact, the offline RL algorithms do not extraordinary diverse from those online. Any online algorithm could turn into the corresponding offline one, via dropping out the procedures of interacting with the environment and iterating the experience buffer.

The extra procedures added to the primary off-policy algorithm by the LR are all at action selection. When the action-value function is being iterated (i.e. $Q(s,a)\leftarrow R+Q(s^{\prime},a^{\prime})$), an action $a’$ (under the state $s’$) has been selected by the primary algorithm. As Equation (6), the density of $a’$ at $\beta(a’|s’)$ has to be bigger than the threshold. So the selected $(s’,a’)$ is sent to the discriminator of the GAN, obtaining a confidence degree of this pair. If this degree is below the given threshold, then $a’$ is overlayed by a random noise $\cal{N}$, ($\cal{N}$ is a Gaussian noise and its mean value is zero.) until the confidence degree of $(s’,a’+\cal{N})$ is bigger than the threshold. Afterwards, the $(s’,a’+\cal{N})$ pair instead of $(s’,a’)$ is used to iterate the $Q(s,a)$. The rest procedures are same as the primary algorithm.

Almost all off-policy online algorithm with the action-value function $Q(s,a)$ (without the policy actor is okay) could combine with our LR Framework. In this section, the DDPG is served as an example. The offline LR-DDPG algorithm is summarized in Algorithm 1.

The goal in our work is to study offline reinforcement learning with fixed datasets. We firstly analyze how error propagates in standard off-policy RL algorithms. It is due to the use of Rare Actions for computing the target values in the Bellman backup. This naturally leads to that the key for offline RL is avoiding selecting a Rare Action. Armed with this insight, we develop a framework for mitigating the effect of Rare Actions, which we call MR. The LR constrains the backup to use actions that have non-negligible support under the data distribution, but without being overly strict in constraining the learned policy. This LR Framework perfectly keep the balance between the training convergence and the optimal learning. The creative LR Framework is able to combine with almost all off-policy RL algorithms with the action-value function $Q(s,a)$ (without the policy actor is okay). Hence, the novel framework proposed in this paper has a significant superiority that it could conveniently transform the advanced online RL algorithm just comes up to the offline one. This advantage may help the offline RL develop more quickly by the way of absorbing the online RL algorithms.