Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Online safe reinforcement learning (RL) has found successful applications in various domains, such as autonomous driving (Isele et al.,, 2018), recommender systems (Chow et al.,, 2017), and robotics (Achiam et al.,, 2017). It enables the learning of safe policies effectively while satisfying certain safety constraints, including collision avoidance, budget adherence, and reliability. However, collecting diverse interaction data can be extremely costly and infeasible in many real-world applications, and this challenge becomes even more critical in scenarios where risky behavior cannot be tolerated. Given the inherently risk-sensitive nature of these safety-related tasks, data collection becomes feasible only when employing behavior policies satisfies all the safety requirements.

To overcome the limitations imposed by interactive data collection, offline RL algorithms are designed to learn a policy from an available dataset collected from historical experiences by some behavior policy, which may differ from the policy we aim to learn. A desirable property of an effective offline algorithm is the assurance of robust policy improvement (RPI), which guarantees that a learned policy is always at least as good as the baseline behavior policies (Fujimoto et al.,, 2019; Laroche et al.,, 2019; Kumar et al.,, 2019; Siegel et al.,, 2020; Chen et al., 2022a, ; Zhu et al.,, 2023; Bhardwaj et al.,, 2024). We extend the property of RPI to offline safe RL called safe robust policy improvement (SRPI), which indicates the improvement should be safe as well. This is particularly important in offline safe RL. For example, in autonomous driving, an expert human driver operates the vehicle to collect a diverse dataset under various road and weather conditions, serving as the behavior policy. This policy is considered both effective and safe, as it demonstrates proficient human driving behavior while adhering to all traffic laws and other safety constraints. Achieving a policy that upholds the SRPI characteristic with such a dataset can significantly mitigate the likelihood of potential collisions and other safety concerns.

In offline RL, we represent the state-action occupancy distribution of policy $\pi$ over the dataset distribution $\mu$ using the ratio $w^{\pi}=d^{\pi}/\mu$. A commonly required assumption is that the $\ell_{\infty}$ concentrability $C_{\ell_{\infty}}^{\pi}$ is bounded, which is defined as the infinite norm of $w^{\pi}$ for all policies (Liu et al.,, 2019; Chen and Jiang,, 2019; Wang et al.,, 2019; Liao et al.,, 2022; Zhang et al.,, 2020). A stronger assumption requires a uniform lower bound on $\mu(a|s)$ (Xie and Jiang,, 2021). However, such all-policy concentrability assumptions are difficult to satisfy in practice, particularly for offline safe RL, as they essentially require the offline dataset to have good coverage of all unsafe state-action pairs. To address the full coverage requirement, other works (Rashidinejad et al.,, 2021; Zhan et al.,, 2022; Chen and Jiang,, 2022; Xie et al.,, 2021; Uehara and Sun,, 2021) adapt conservative algorithms by employing the principle of pessimism in the face of uncertainty, reducing the assumption to the best covered policy (or optimal policy) concerning $\ell_{\infty}$ concentrability. Zhu et al., (2023) introduce $\ell_{2}$ concentrability to further relax the assumption, indicating that $\ell_{\infty}$ concentrability is always an upper bound of $\ell_{2}$ concentrability (see Table 1 for detailed comparisons with previous works). While provable guarantees are obtained using single policy concentrability for unconstrained MDP as Table 1 suggests for the safe RL setting, all the existing studies (Hong et al.,, 2024; Le et al.,, 2019) still require the coverage on all the policies. Further, as Table 1 suggests, the above papers do not guarantee robust safe policy improvement. m , Our main contributions are summarized below:

1. 1.

   We prove that our algorithm, which uses average Bellman error, enjoys an optimal statistical rate of $1/\sqrt{N}$ under partial data coverage assumption. This is the first work that achieves such a result using only single-policy $\ell_{2}$ concentrability.

2. 2.

   We propose a novel offline safe RL algorithm, called Weighted Safe Actor-Critic (WSAC), which can robustly learn policies that improve upon any behavior policy with controlled relative pessimism. We prove that under standard function approximation assumptions, when the actor incorporates a no-regret policy optimization oracle, WSAC outputs a policy that never degrades the performance of a reference policy (including the behavior policy) for a range of hyperparameters (defined later). This is the first work that provably demonstrates the property of SRPI in offline safe RL setting.

3. 3.

   We point out that primal-dual-based approaches Hong et al., (2024) may require all-policy concentrability assumption. Thus, unlike, the primal-dual-based appraoch, we propose a novel rectified penalty-based approach to obtain results using single-policy concentrability. Thus, we need novel analysis techniques to prove results.

4. 4.

   Furthermore, we provide a practical implementation of WSAC following a two-timescale actor-critic framework using adversarial frameworks similar to Cheng et al., (2022); Zhu et al., (2023), and test it on several continuous control environments in the offline safe RL benchmark (Liu et al., 2023a, ). WSAC outperforms all other state-of-the-art baselines, validating the property of a safe policy improvement.

Offline safe RL: Deep offline safe RL algorithms (Lee et al.,, 2022; Liu et al., 2023b, ; Xu et al.,, 2022; Chen et al.,, 2021; Zheng et al.,, 2024) have shown strong empirical performance but lack theoretical guarantees. To the best of our knowledge, the investigation of policy improvement properties in offline safe RL is relatively rare in the state-of-the-art offline RL literature. Wu et al., (2021) focus on the offline constrained multi-objective Markov Decision Process (CMOMDP) and demonstrate that an optimal policy can be learned when there is sufficient data coverage. However, although they show that CMDP problems can be formulated as CMOMDP problems, they assume a linear kernel CMOMDP in their paper, whereas our consideration extends to a more general function approximation setting. Le et al., (2019) propose a model-based primal-dual-type algorithm with deviation control for offline safe RL in the tabular setting. With prior knowledge of the slackness in Slater’s condition and a constant on the concentrability coefficient, an $(\epsilon,\delta)$-PAC error is achievable when the number of data samples $N$ is large enough $(N=\tilde{\mathcal{O}}(1/\epsilon^{2}))$. These assumptions make the algorithm impractical, and their computational complexity is much higher than ours. Additionally, we consider a more practical, model-free function approximation setting. In another concurrent work (Hong et al.,, 2024), a primal-dual critic algorithm is proposed for offline-constrained RL settings with general function approximation. However, their algorithm requires $\ell_{2}$ concentrability for all policies, which is not practical as discussed. The reason is that the dual variable optimization in their primal-dual design requires an accurate estimation of all the policies used in each episode, which necessitates coverage over all policies. Moreover, they cannot guarantee the property of SRPI. Moreover, their algorithm requires an additional offline policy evaluation (OPE) oracle for policy evaluation, making the algorithm less efficient.

Our algorithm design builds upon the constrained actor-critic method, in which we iteratively optimize a policy and improve the policy based on the evaluation of reward and cost. Consider the following actor-critic approach for solving the optimization problem (5):

where we assume that $s_{0}$ is a fixed initial state, and $f_{r}(s,\pi)=\sum_{a\in\mathcal{A}}\pi(a|s)f_{r}(s,a),f_{c}(s,\pi)=\sum_{a\in\mathcal{A}}\pi(a|s)f_{c}(s,a).$ The policy is optimized by maximizing the reward $q$ function $f_{r}$ while ensuring that $f_{c}$ satisfies the constraint, and the two functions are trained by minimizing the Bellman error. However, this formulation has several disadvantages. $1)$ It cannot handle insufficient data coverage, which may fail to provide an accurate estimation of the policy for unseen states and actions. $2)$It cannot guarantee robust policy improvement. $3)$ The actor training step is computationally intractable especially when the policy space is extremely large.

To address the insufficient data coverage issue, as mentioned in Xie et al., (2021) the critic can include a Bellman-consistent pessimistic evaluation of $\pi,$ which selects the most pessimistic function that approximately satisfies the Bellman equation, which is called absolute pessimism. Then later as indicated by Cheng et al., (2022), instead of using an absolute pessimism, a relative pessimism approach by considering competing to the behavior policy can obtain a robust improvement over the behavior policy. However, this kind of approach can only achieve a suboptimal statistical rate of $N^{1/3},$ and fails to achieve the optimal statistical rate of $1/\sqrt{N},$ then later a weighted average Bellman error (Uehara et al.,, 2020; Xie and Jiang,, 2020; Zhu et al.,, 2023) could be treated as one possible solution for improving the order. We remark here that all the discussions here are for the traditional unconstrained offline RL. Regarding safety, no existing efficient algorithms in safe offline RL have theoretically demonstrated the property of robust policy improvement with optimal statistical rate.

Can Primal-dual based approaches achieve result using only single policy coverability?: The most commonly used approach for addressing safe RL problems is primal-dual optimization (Efroni et al.,, 2020; Altman,, 1999). As shown in current offline safe RL literature (Hong et al.,, 2024; Le et al.,, 2019), the policy can be optimized by maximizing a new unconstrained “reward" $Q-$ function $f_{r}^{\pi}(s_{0},\pi)-\lambda f_{c}^{\pi}(s_{0},\pi)$ where $\lambda$ is a dual variable. Then, the dual-variable can be tuned by taking gradient descent step. As we discussed in the introduction, all these require all policy concentrability which is not practical especially for safe RL. Important question is whether all policy concentrability assumption can be relaxed. Note that primal-dual algorithm relies on solving the min-max problem $\min_{\lambda}\max_{\pi}f_{r}^{\pi}(s_{0},\pi)-\lambda f_{c}^{\pi}(s_{0},\pi)$. Recent result (Cui and Du,, 2022) shows that single policy concentrability assumption is not enough for offline min-max game. Hence, we conjecture that using the primal-dual method we can not relax the all policy concentrability assumption. Intuitively, the primal-dual based method (Hong et al.,, 2024) rely on bounding the regret in dual domain $\sum_{k}(\lambda_{k}-\lambda^{*})(f_{c}^{\pi_{k}}-0)$, hence, all the policies $\{\pi_{k}\}_{k=1}^{K}$ encountered throughout the iteration must be supported by the dataset to evaluate the dual value $\lambda^{*}(f_{c}^{\pi_{k}}-0)$ where $\lambda^{*}$ is the optimal dual value.

Our novelty: In contrast, we propose an aggression-limited objective function $f_{r}(s_{0},\pi)-\lambda\cdot[f_{c}(s_{0},\pi)]_{+}$ to control aggressive policies, where $\{\cdot\}_{+}:=\max\{\cdot,0\}.$ The high-level intuition behind this aggression-limited objective function is that by appropriately selecting a $\lambda$ (usually large enough), we penalize all the policies that are not safe. As a result, the policy that maximizes the objective function is the optimal safe policy. This formulation is fundamentally different from the traditional primal-dual approach as it does not require dual-variable tuning, and thus, does not require all policy concentrability. In particular, we only need to bound the primal domain regret which can be done as long as the reference policy is covered by the dataset similar to the unconstrained setup.

Combining all the previous ideas together provides the design of our main algorithm named WSAC (Weighted Safe Actor-Critic). In Section 5, we will provide theoretical guarantees of WSAC and discuss its advantages over existing approaches in offline safe RL. WSAC aims to solve the following optimization problem:

where $\mathcal{L}_{\mu}(\pi,f):=\mathbb{E}_{\mu}[f(s,\pi)-f(s,a)],$ and $\mathcal{E}_{\mu}(\pi,f):=\max_{w\in\mathcal{W}}|\mathbb{E}_{\mu}[w(s,a)((f-T^{\pi}_{r}f)(s,a))]|,\hat{\mathcal{E}}_{\mu}(\pi,f):=\max_{w\in\mathcal{W}}|\mathbb{E}_{\mu}[w(s,a)((f-T^{\pi}_{c}f)(s,a))]|.$ This formulation can also be treated as a Stackelberg game (Von Stackelberg,, 2010) or bilevel optimization problem. We penalize the objective function only when the approximate cost $Q$-function $f_{c}^{\pi}$ of the policy $\pi$ is more perilous than the behavior policy ($f_{c}^{\pi}(s,\pi)\geq f_{c}^{\pi}(s,a)$) forcing our policy to be as safe as the behavior policy. Maximization over $w$ in for training the two critics can ensure that the Bellman error is small when averaged over measure $\mu\cdot w$ for any $w\in\mathcal{W},$ which turns out to be sufficient to control the suboptimality of the learned policy.

In the following theorem, we show that the solution of the optimization problem (6) is not worse than the behavior policy $\mu$ in both performance and safety for any $\beta\geq 0,\lambda>0$ than the policy $\mu$ under Assumption 3.2 with $\epsilon_{1}=0$.

The result in Theorem 4.1 shows that by selecting $\lambda$ large enough, for any $\beta\geq 0,$ the solution can achieve better performance than the behavior policy while maintaining safety that is arbitrarily close to that of the behavior policy. The Theorem verifies the design of our framework which has the potential to have a robust safe improvement.

In the next section, we will introduce our main algorithm WSAC and provide its theoretical guarantees.

In this paper, we explore the problem of offline Safe-RL with a single policy data coverage assumption. We propose a novel algorithm, WSAC, which, for the first time, is proven to guarantee the property of safe robust policy improvement. WSAC is able to outperform any reference policy, including the behavior policy, while maintaining the same level of safety across a broad range of hyperparameters. Our simulation results demonstrate that WSAC outperforms existing state-of-the-art offline safe-RL algorithms. Interesting future work includes combining WSAC with online exploration with safety guarantees and extending the approach to multi-agent settings to handle coupled constraints.