Trust Region-Based Safe Distributional
Reinforcement Learning for Multiple Constraints

Deep reinforcement learning (RL) enables reliable control of complex robots (Merel et al., 2020; Peng et al., 2021; Rudin et al., 2022). In order to successfully apply RL to real-world systems, it is essential to design a proper reward function which reflects safety guidelines, such as collision avoidance and limited energy consumption, as well as the goal of the given task. However, finding the reward function that considers all such factors involves a cumbersome and time-consuming process since RL algorithms must be repeatedly performed to verify the results of the designed reward function. Instead, safe RL, which handles safety guidelines as constraints, is an appropriate solution. A safe RL problem can be formulated using a constrained Markov decision process (Altman, 1999), where not only the reward but also cost functions are defined to provide the safety guideline signals. By defining constraints using expectation or risk measures of the sum of costs, safe RL aims to maximize returns while satisfying the constraints. Under the safe RL framework, the training process becomes straightforward since there is no need to search for a reward that reflects the safety guidelines.

While various safe RL algorithms have been proposed to deal with the safety guidelines, their applicability to general robotic applications remains limited due to the insufficiency in 1) handling multiple constraints and 2) minimizing failures, such as robot breakdowns after collisions. First, many safety-critical applications require multiple constraints, such as maintaining distance from obstacles, limiting operational space, and preventing falls. Lagrange-based safe RL methods (Yang et al., 2021; Zhang and Weng, 2022; Bai et al., 2022), which convert a safe RL problem into a dual problem and update the policy and Lagrange multipliers, are commonly used to solve these multi-constrained problems. However, the Lagrangian methods are difficult to guarantee satisfying constraints during training theoretically, and the training process can be unstable due to the multipliers (Stooke et al., 2020). To this end, trust region-based methods (Yang et al., 2020; Kim and Oh, 2022a), which can ensure to improve returns while satisfying the constraints under tabular settings (Achiam et al., 2017), have been proposed as an alternative to stabilize the training process. Still, trust region-based methods have a critical issue. Depending on the initial policy settings, there can be an infeasible starting case, meaning that no policy within the trust region satisfies constraints. To address this issue, we can sequentially select a violated constraint and update the policy to reduce the selected constraint (Xu et al., 2021). However, this can be inefficient as only one constraint is considered per update. It will be better to handle multiple constraints at once, but it is a remaining problem to find a policy gradient that reflects several constraints and guarantees to reach a feasible policy set.

Secondly, as RL settings are inherently stochastic, employing risk-neutral measures like expectation to define constraints can lead to frequent failures. Hence, it is crucial to define constraints using risk measures, such as conditional value at risk (CVaR), as they can reduce the potential for massive cost returns by emphasizing tail distributions (Yang et al., 2021; Kim and Oh, 2022a). In safe RL, critics are used to estimate the constraint values. Especially, to estimate constraints based on risk measures, it is required to use distributional critics (Dabney et al., 2018b), which can be trained using the distributional Bellman update (Bellemare et al., 2017). However, the Bellman update only considers the one-step temporal difference, which can induce a large bias. The estimation bias makes it difficult for critics to judge the policy, which can lead to the policy becoming overly conservative or risky, as shown in Section 5.3. In particular, when there are multiple constraints, the likelihood of deriving incorrect policy gradients due to estimation errors grows exponentially. Therefore, there is a need for a method that can train distributional critics with low biases.

In this paper, we propose a trust region-based safe RL algorithm called a safe distributional actor-critic (SDAC), designed to effectively manage multiple constraints and estimate risk-averse constraints with low biases. First, to handle the infeasible starting case by considering all constraints simultaneously, we propose a gradient integration method that projects unsafe policies into a feasible policy set by solving a quadratic program (QP) consisting of gradients of all constraints. It guarantees to obtain a feasible policy within a finite time under mild technical assumptions, and we experimentally show that it can restore the policy more stably than the existing method (Xu et al., 2021). Furthermore, by updating the policy using the trust region method with the integrated gradient, our approach makes the training process more stable than the Lagrangian method, as demonstrated in Section 5.2. Second, to train critics to estimate constraints with low biases, we propose a TD($\lambda$) target distribution which can adjust the bias-variance trade-off. The target distribution is obtained by merging the quantile regression losses (Dabney et al., 2018b) of multi-step distributions and extracting a unified distribution from the loss. The unified distribution is then projected onto a quantile distribution set in a memory-efficient manner. We experimentally show that the target distribution can trade off the bias-variance of the constraint estimations (see Section 5.3).

We conduct extensive experiments with multi-constrained locomotion tasks and single-constrained Safety Gym tasks (Ray et al., 2019) to evaluate the proposed method. In the locomotion tasks, SDAC shows 1.93 times fewer steps to satisfy all constraints than the second-best baselines. In the Safety Gym tasks, the proposed method shows 1.78 times fewer constraint violations than the second-best methods while achieving high returns when using risk-averse constraints. As a result, it is shown that the proposed method can efficiently handle multiple constraints using the gradient integration method and effectively lower the constraint violations using the low-biased distributional critics.

Constrained Markov Decision Processes. We formulate the safe RL problem using constrained Markov decision processes (CMDPs) (Altman, 1999). A CMDP is defined as $(S$, $A$, $P$, $R$, $C_{1,..,K}$, $\rho$, $\gamma)$, where $S$ is a state space, $A$ is an action space, $P:S\times A\times S\mapsto[0,1]$ is a transition model, $R:S\times A\times S\mapsto\mathbb{R}$ is a reward function, $C_{k\in\{1,...,K\}}:S\times A\times S\mapsto\mathbb{R}_{\geq 0}$ are cost functions, $\rho:S\mapsto[0,1]$ is an initial state distribution, and $\gamma\in(0,1)$ is a discount factor. Given a policy $\pi$ from a stochastic policy set $\Pi$, the discounted state distribution is defined as $d^{\pi}(s):=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\mathrm{Pr}(s_{t}=s|\pi)$, and the return is defined as $Z_{R}^{\pi}(s,a):=\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1})$, where $s_{0}=s,\;a_{0}=a,\;a_{t}\sim\pi(\cdot|s_{t})$, and $s_{t+1}\sim P(\cdot|s_{t},a_{t})$. Then, the state value and state action value functions are defined as: $V_{R}^{\pi}(s):=\mathbb{E}\left[Z_{R}^{\pi}(s,a)|a\sim\pi(\cdot|s)\right],\;Q_{R}^{\pi}(s,a):=\mathbb{E}\left[Z_{R}^{\pi}(s,a)\right]$. By substituting the costs for the reward, the cost value functions $V_{C_{k}}^{\pi}(s)$ and $Q_{C_{k}}^{\pi}(s,a)$ are defined. In the remainder of the paper, the cost parts will be omitted since they can be retrieved by replacing the reward with the costs. Then, the safe RL problem is defined as follows with a safety measure $F$:

where $J(\pi):=\mathbb{E}\left[Z_{R}^{\pi}(s,a)|s\sim\rho,a\sim\pi(\cdot|s)\right]+\beta\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}H(\pi(\cdot|s_{t}))|\rho,\pi,P\right]$, $\beta$ is an entropy coefficient, $H$ is the Shannon entropy, and $d_{k}$ is a threshold of the $k$th constraint.

Trust-Region Method With a Mean-Std Constraint. Kim and Oh (2022a) have proposed a trust region-based safe RL method with a risk-averse constraint, called a mean-std constraint. The definition of the mean-std constraint function is as follows:

where $\alpha\in(0,1]$ adjusts the risk level of constraints, $\mathrm{Std}[Z]$ is the standard deviation of $Z$, and $\phi$ and $\Phi$ are the probability density function and the cumulative distribution function (CDF) of the standard normal distribution, respectively. In particular, setting $\alpha=1$ causes the standard deviation part to be zero, so the constraint becomes a risk-neutral constraint. Also, the mean-std constraint can effectively reduce the potential for massive cost returns, as shown in Yang et al. (2021); Kim and Oh (2022a, b). In order to calculate the mean-std constraint, it is essential to estimate the standard deviation of the cost return. To this end, Kim and Oh (2022a) define the square value functions:

Since $\mathrm{Std}[Z]^{2}=\mathbb{E}[Z^{2}]-\mathbb{E}[Z]^{2}$, the $k$th constraint can be written as follows:

where $J_{C_{k}}(\pi):=\mathbb{E}\left[V_{C_{k}}^{\pi}(s)|s\sim\rho\right]$, $J_{S_{k}}(\pi):=\mathbb{E}\left[S_{C_{k}}^{\pi}(s)|s\sim\rho\right]$. In order to apply the trust region method (Schulman et al., 2015), it is necessary to derive surrogate functions for the objective and constraints. These surrogates can substitute for the objective and constraints within the trust region. Given a behavioral policy $\mu$ and the current policy $\pi_{\mathrm{old}}$, we denote the surrogates as $J^{\mu,\pi_{\mathrm{old}}}(\pi)$ and $F_{k}^{\mu,\pi_{\mathrm{old}}}(\pi;\alpha)$. For the definition and derivation of the surrogates, please refer to Appendix A.7 and (Kim and Oh, 2022a). Using the surrogates, a policy can be updated by solving the following subproblem:

where $D_{\mathrm{KL}}(\pi||\pi^{\prime}):=\mathbb{E}_{s\sim d^{\mu}}\left[D_{\mathrm{KL}}(\pi(\cdot|s)||\pi^{\prime}(\cdot|s))\right]$, $D_{\mathrm{KL}}$ is the KL divergence, and $\epsilon$ is a trust region size. This subproblem can be solved through approximation and a line search (see Appendix A.8). However, it is possible that there is no policy satisfying the constraints of (5). In order to tackle this issue, the policy must be projected onto a feasible policy set that complies with all constraints, yet there is a lack of such methods. In light of this issue, we introduce an efficient feasibility handling method for multi-constrained RL problems.

Distributional Quantile Critic. Dabney et al. (2018b) have proposed a method for approximating the random variable $Z^{\pi}_{R}$ to follow a quantile distribution. Given a parametric model, $\theta:S\times A\mapsto\mathbb{R}^{M}$, $Z_{R}^{\pi}$ can be approximated as $Z_{R,\theta}$, called a distributional quantile critic. The probability density function of $Z_{R,\theta}$ is defined as follows:

where $M$ is the number of atoms, $\theta_{m}(s,a)$ is the $m$th atom, $\delta$ is the Dirac function, and $\delta_{a}(z):=\delta(z-a)$. The percentile value of the $m$th atom is denoted by $\tau_{m}$ ($\tau_{0}=0,\tau_{i}=i/M$). In distributional RL, the returns are directly estimated to get value functions, and the target distribution can be calculated from the distributional Bellman operator (Bellemare et al., 2017): $\mathcal{T}^{\pi}Z_{R}(s,a):\stackrel{{\scriptstyle D}}{{=}}R(s,a,s^{\prime})+\gamma Z_{R}(s^{\prime},a^{\prime})$, where $s^{\prime}\sim P(\cdot|s,a)$ and $a^{\prime}\sim\pi(\cdot|s^{\prime})$. The above one-step distributional operator can be expanded to the $n$-step one: $\mathcal{T}_{n}^{\pi}Z_{R}(s_{0},a_{0}):\stackrel{{\scriptstyle D}}{{=}}\sum_{t=0}^{n-1}\gamma^{t}R(s_{t},a_{t},s_{t+1})+\gamma^{n}Z_{R}(s_{n},a_{n})$, where $a_{t}\sim\pi(\cdot|s_{t})$ for $t=1,...,n$. Then, the critic can be trained to minimize the following quantile regression loss (Dabney et al., 2018b):

$D$ is a replay buffer, $\bar{\tau}_{m}:=(\tau_{m-1}+\tau_{m})/2$, and $\mathcal{L}^{\bar{\tau}_{m}}_{\mathrm{QR}}(\theta_{m})$ denotes the quantile regression loss for a single atom. The distributional quantile critic can be plugged into existing actor-critic algorithms because only the critic modeling part is changed.

The proposed method comprises two key components: 1) a feasibility handling method required for multi-constrained safe RL problems and 2) a target distribution designed to minimize estimation bias. This section sequentially presents these components, followed by a detailed explanation of the proposed method.

Safe Reinforcement Learning. Garcıa and Fernández (2015) and Gu et al. (2022) have researched and categorized safe RL methodologies from various perspectives. In this paper, we introduce safe RL methods depending on how to update policies to reflect safety constraints. First, trust region-based safe RL methods (Achiam et al., 2017; Yang et al., 2020; Kim and Oh, 2022a) find policy update directions by approximating the safe RL problem as a linear-quadratic constrained linear program and update policies through a line search. Yang et al. (2020) also employ projection to meet a constraint; however, their method is limited to a single constraint and does not show to satisfy the constraint for the infeasible starting case. Second, Lagrangian-based methods (Stooke et al., 2020; Yang et al., 2021; Liu et al., 2020) convert the safe RL problem to a dual problem and update the policy and dual variables simultaneously. Last, expectation-maximization (EM) based methods (Liu et al., 2022; Zhang et al., 2022) find non-parametric policy distributions by solving the safe RL problem in E-steps and fit parametric policies to the found non-parametric distributions in M-steps. Also, there are other ways to reflect safety other than policy updates. Qin et al. (2021); Lee et al. (2022) find optimal state or state-action distributions that satisfy constraints, and Bharadhwaj et al. (2021); Thananjeyan et al. (2021) reflect safety during exploration by executing only safe action candidates. In the experiments, only the safe RL methods of the policy update approach are compared with the proposed method.

Distributional TD($\lambda$). TD($\lambda$) (Precup et al., 2000) can be extended to the distributional critic to trade off bias-variance. Gruslys et al. (2018) have proposed a method to obtain target distributions by mixing $n$-step distributions, but the method is applicable only in discrete action spaces. Nam et al. (2021) have proposed a method to obtain target distributions using sampling to apply to continuous action spaces, but this is only for on-policy settings. A method proposed by Tang et al. (2022) updates the critics using newly defined distributional TD errors rather than target distributions. This method is applicable for off-policy settings but has the disadvantage that memory usage increases linearly with the number of TD error steps. In contrast to these methods, the proposed method is memory-efficient and applicable for continuous action spaces under off-policy settings.

Gradient Integration. The proposed feasibility handling method utilizes a gradient integration method, which is widely used in multi-task learning (MTL). The gradient integration method finds a single gradient to improve all tasks by using gradients of all tasks. Yu et al. (2020) have proposed a projection-based gradient integration method, which is guaranteed to converge Pareto-stationary sets. A method proposed by Liu et al. (2021) can reflect user preference, and Navon et al. (2022) proposed a gradient-scale invariant method to prevent the training process from being biased by a few tasks. The proposed method can be viewed as a mixture of projection and scale-invariant methods as gradients are clipped and projected onto a trust region.

We evaluate the safety performance of the proposed method in single- and multi-constrained robotic tasks. For single constraints, the agent performs four tasks provided by Safety Gym (Ray et al., 2019), and for multi-constraints, it performs bipedal and quadrupedal locomotion tasks.

A limitation of the proposed method is that the computational complexity of the gradient integration is proportional to the square of the number of constraints, whose qualitative analysis is presented in Appendix E.1. Also, we conducted quantitative analyses in Appendix E.2 by measuring wall clock training time. In the mini-cheetah task which has three constraints, the training time of SDAC is the third fastest among the four safe RL algorithms. Gradient integration is not applied when the policy satisfies constraints, so it may not constitute a significant proportion of training time. However, its influence can be dominant as the number of constraints increases. In order to resolve this limitation, the calculation can be speeded up by stochastically selecting a subset of constraints (Liu et al., 2021), or by reducing the frequency of policy updates (Navon et al., 2022). The other limitation is that the mean-std defined in (2) is not a coherent risk measure. As a result, mean-std constraints can be served as reducing uncertainty rather than risk, although we experimentally showed that constraint violations are efficiently reduced. To resolve this, we can use the CVaR constraint, which can be estimated using an auxiliary variable, as done by Chow et al. (2017). However, this solution can destabilize the training process due to the auxiliary variable, as observed in experiments of Kim and Oh (2022b). Hence, a stabilization technique should be developed to employ the CVaR constraint.

We have presented the trust region-based safe distributional RL method, called SDAC. Through the locomotion tasks, it is verified that the proposed method efficiently satisfies multiple constraints using the gradient integration. Moreover, constraints can be stably satisfied in various tasks due to the low-biased distributional critics trained using the proposed target distributions. In addition, the proposed method is analyzed from multiple perspectives through various ablation studies. However, to compensate for the computational complexity, future work plans to devise efficient methods when dealing with large numbers of constraints.

This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01190, [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceptable Machine Learning, 34%), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (NRF-2022R1A2C2008239, General-Purpose Deep Reinforcement Learning Using Metaverse for Real World Applications, 33%), and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-01341, AI Graduate School Program, CAU, 33%).