Guarded Policy Optimization with Imperfect Online Demonstrations

We consider an infinite-horizon Markov decision process (MDP), defined by the tuple $M=\left\langle\mathcal{S},\mathcal{A},P,R,\gamma,d_{0}\right\rangle$ consisting of a finite state space $\mathcal{S}$, a finite action space $\mathcal{A}$, the state transition probability distribution $P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,1]$, the reward function $R:\mathcal{S}\times\mathcal{A}\to[R_{\min},R_{\max}]$, the discount factor $\gamma\in(0,1)$ and the initial state distribution $d_{0}:\mathcal{S}\to[0,1]$. Unless otherwise stated, $\pi$ denotes a stochastic policy $\pi:\mathcal{S}\times\mathcal{A}\to[0,1]$. The state-action value and state value functions of $\pi$ are defined as $Q^{\pi}(s,a)=\mathbb{E}_{s_{0}=s,a_{0}=a,a_{t}\sim\pi\left(\cdot\mid s_{t}\right),s_{t+1}\sim p\left(\cdot\mid s_{t},a_{t}\right)}\left[\sum_{t=0}^{\infty}\gamma^{t}R\left(s_{t},a_{t}\right)\right]$ and $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}Q^{\pi}(s,a)$. The optimal policy is expected to maximize the accumulated return $J(\pi)=\mathbb{E}_{s\sim d_{0}}V^{\pi}(s)$.

The Teacher-Student Framework (TSF) models the shared control system as the combination of a teacher policy $\pi_{t}$ which is pretrained and fixed and a student policy $\pi_{s}$ to be learned. The actual actions applied to the agent are deduced from a mixed policy of $\pi_{t}$ and $\pi_{s}$, where $\pi_{t}$ starts generating actions when intervention happens. The details of the intervention mechanism are described in Sec. 3.2. The goal of TSF is to improve the training efficiency and safety of $\pi_{s}$ with the involvement of $\pi_{t}$. The discrepancy between $\pi_{t}$ and $\pi_{s}$ on state $s$, termed as policy discrepancy, is the $L_{1}$-norm of output difference: $\|\pi_{t}(\cdot|s)-\pi_{s}(\cdot|s)\|_{1}=\int_{\mathcal{A}}\left|\pi_{t}(a|s)-\pi_{s}(a|s)\right|\mathrm{d}a.$ We define the discounted state distribution under policy $\pi$ as $d_{\pi}(s)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\operatorname{Pr}\left(s_{t}=s;\pi,d_{0}\right)$, where $\operatorname{Pr}^{\pi}\left(s_{t}=s;\pi,d_{0}\right)$ is the state visitation probability. The state distribution discrepancy is defined as the difference in $L_{1}$-norm of the discounted state distributions deduced from two policies: $\|d_{\pi_{t}}-d_{\pi_{s}}\|_{1}=\int_{\mathcal{S}}\left|d_{\pi_{t}}(s)-d_{\pi_{s}}(s)\right|\mathrm{d}s$.

In intervention-based RL, the teacher policy and the student policy act together and become a mixed behavior policy $\pi_{b}$. The intervention function $\mathcal{T}(s)$ determines which policy is in charge. Let $\mathcal{T}(s)=1$ denotes the teacher policy $\pi_{t}$ takes control and $\mathcal{T}(s)=0$ means otherwise. Then $\pi_{b}$ can be represented as $\pi_{b}(\cdot|s)=\mathcal{T}(s)\pi_{t}(\cdot|s)+(1-\mathcal{T}(s))\pi_{s}(\cdot|s)$.

Both Eq. 1 and Eq. 2 bound the state distribution discrepancy by the difference in per-state policy distributions, but the upper bound with intervention is squeezed by the intervention rate $\beta$. In practical algorithms, $\beta$ can be minimized to reduce the state distribution discrepancy and thus relieve the performance drop during test time. Based on Thm. 3.2, we further prove in Appendix A.1 that under the setting of intervention-based RL, the accumulated returns of behavior policy $J(\pi_{b})$ and student policy $J(\pi_{s})$ can be similarly related. The analysis in this section does not assume a certain form of the intervention function $\mathcal{T}(s)$. Our analysis provides the insight on the feasibility and efficiency of all previous algorithms in intervention-based RL (Kelly et al., 2019; Peng et al., 2021; Chisari et al., 2021). In the following section, we will examine different forms of intervention functions and investigate their properties and performance bounds, especially with imperfect online demonstrations.

The theorem shows that $J(\pi_{b})$ can be lower bounded by the return of the teacher policy $\pi_{t}$ and an extra term relating to the entropy of the teacher policy. It implies that action-based intervention function $\mathcal{T}_{\text{action}}$ is indeed helpful in providing training data with high return. We discuss the tightness of Thm. 3.3 and give an intuitive interpretation of $\sqrt{H-\varepsilon}$ in Appendix A.2.

A drawback of the action-based intervention function is the strong assumption on the optimal teacher, which is not always feasible. If we turn to employ a suboptimal teacher, the behavior policy would be burdened due to the upper bound in Eq. 4. We illustrate this phenomenon with the example in Fig. 2 where a slow vehicle in gray is driving in front of the ego-vehicle in blue. The student policy is aggressive and would like to overtake the gray vehicle to reach the destination faster, while the teacher intends to follow the vehicle conservatively. Therefore, $\pi_{s}$ and $\pi_{b}$ will propose different actions in the current state, leading to $\mathcal{T}_{\text{action}}=1$ according to Eq. 3. The mixed policy with shared control will always choose to follow the front vehicle and the agent can never accomplish a successful overtake.

In safety-critical scenarios, the step-wise training cost $c(s,a)$, $i.e.,$ the penalty on the safety violation during training, can be regarded as a negative reward. We define $\hat{r}(s,a)=r(s,a)-\eta c(s,a)$ as the combined reward, where $\eta$ is the weighting hyperparameter. $\hat{V},\hat{Q}$ and $\hat{\mathcal{T}}_{\text{value}}$ are similarly defined by substituting $r$ with $\hat{r}$ in the original definition. Then we have the following corollary related to expected cumulative training cost, defined by $C(\pi)=\mathbb{E}_{s_{0}\sim d_{0},a_{t}\sim\pi\left(\cdot\mid s_{t}\right),s_{t+1}\sim p\left(\cdot\mid s_{t},a_{t}\right)}\left[\sum_{t=0}^{\infty}\gamma^{t}c\left(s_{t},a_{t}\right)\right]$.

In Eq. 7 the upper bound of behavior policy’s training cost consists of three terms: the cost of teacher policy, the threshold in intervention $\epsilon$ multiplied by coefficients and the superiority of $\pi_{b}$ over $\pi_{t}$ in cumulative reward. The first two terms are similar to those in Eq. 6 and the third term means a trade-off between training safety and efficient exploration, which can be adjusted by hyperparameter $\eta$.

Comparing the lower bound performance guarantee of action-based and value-based intervention function (Eq. 4 and Eq. 6), the performance gap between $\pi_{b}$ and $\pi_{t}$ can both be bounded with respect to the threshold for intervention $\varepsilon$ and the discount factor $\gamma$. The difference is that the performance gap when using $\mathcal{T}_{\text{action}}$ is in an order of $O(\frac{1}{(1-\gamma)^{2}})$ while the gap with $\mathcal{T}_{\text{value}}$ is in an order of $O(\frac{1}{1-\gamma})$. It implies that in theory value-based intervention leads to better lower-bound performance guarantee. In terms of training safety guarantee, value-based intervention function $\mathcal{T}_{\text{value}}$ has better safety guarantee by providing a tighter safety bound with the order of $O(\frac{1}{1-\gamma})$, in contrast to $O(\frac{1}{(1-\gamma)^{2}})$ of action-based intervention function (see Theorem 1 in (Peng et al., 2021)). We show in the Sec. 4.3 that the theoretical advances of $\mathcal{T}_{\text{value}}$ in training safety and efficiency can both be verified empirically.

Justified by the aforementioned advantages of the value-based intervention function, we propose a practical algorithm called Teacher-Student Shared Control (TS2C). Its workflow is listed in Appendix B. To obtain the teacher Q-network $Q^{\pi_{t}}$ in the value-based intervention function in Eq. 5, we rollout the teacher policy $\pi_{t}$ and collect training samples during the warmup period. Gaussian noise is added to the teacher’s policy distribution to increase the state coverage during warmup. With limited training data the Q-network may fail to provide accurate estimation when encountering previously unseen states. We propose to use teacher Q-ensemble based on the idea of ensembling Q-networks (Chen et al., 2021). A set of ensembled teacher Q-networks $\mathbf{Q}^{\phi}$ with the same architecture and different initialization weights are built and trained with the same data. To learn $\mathbf{Q}^{\phi}$ we follow the standard procedure in (Chen et al., 2021) and optimize the following loss:

where $y=\mathbb{E}_{s^{\prime}\sim\mathcal{D},a^{\prime}\sim\pi_{t}(\cdot|s^{\prime})+\mathcal{N}(0,\sigma)}\left[r+\gamma\mathrm{Mean}\left[\mathbf{Q}^{\phi}\left(s^{\prime},a^{\prime}\right)\right]\right]$ is the Bellman target and $\mathcal{D}$ is the replay buffer for storing sequences $\{(s,a,r,s^{\prime})\}$. Teacher will intervene when $\mathcal{T}_{\text{value}}$ returns 1 or the output variance of ensembled Q-networks surpasses the threshold, which means the agent is exploring unknown regions and requires guarding. We also use $\mathbf{Q}^{\phi}$ to compute the state-value functions in Eq. 5, leading to the following practical intervention function:

Eq. 2 shows that the distributional shift and the performance gap to oracle can be reduced with smaller $\beta$, i.e., less teacher intervention. Therefore, we minimize the amount of teacher intervention via adding negative reward to the transitions one step before the teacher intervention. Incorporating intervention minimization, we use the following loss function to update the student’s Q-network parameterized by $\psi$:

where $y^{\prime}=\mathbb{E}_{s^{\prime}\sim\mathcal{D},a^{\prime}\sim\pi_{b}(\cdot|s^{\prime})}\left[r-\lambda\mathcal{T}_{\text{TS2C}}(s^{\prime})+\gamma Q^{\psi}\left(s^{\prime},a^{\prime}\right)\right.$ $-\left.\alpha\log\pi_{b}(a^{\prime}|s^{\prime})\right]$ is the soft Bellman target with intervention minimization. $\lambda$ is the hyperparameter controlling the intervention minimization. $\alpha$ is the coefficient for maximum-entropy learning updated in the same way as Soft Actor Critic (SAC) (Haarnoja et al., 2018). To update the student’s policy network parameterized by $\theta$, we apply the objective used in SAC as:

The majority of the experiments are conducted on the lightweight driving simulator MetaDrive (Li et al., 2022a). One concern with TSF algorithms is that the student may simply record the teacher’s actions and overfit the training environment. MetaDrive can test the generalizability of learned agents on unseen driving environments with its capability to generate an unlimited number of scenes with various road networks and traffic flows. We choose 100 scenes for training and 50 held-out scenes for testing. Examples of the traffic scenes from MetaDrive are shown in Appendix C. In MetaDrive, the objective is to drive the ego vehicle to the destination without dangerous behaviors such as crashing into other vehicles. The reward function consists of the dense reward proportional to the vehicle speed and the driving distance, and the terminal +20 reward when the ego vehicle reaches the destination. Training cost is increased by 1 when the ego vehicle crashes or drives out of the lane. To evaluate TS2C’s performance in different environments, we also conduct experiments in several environments of the MuJoCo simulator (Todorov et al., 2012).

Two sets of algorithms are selected as baselines to compare with. One includes traditional RL and IL algorithms without the TSF. By comparing with these methods we can demonstrate how TS2C improves the efficiency and safety of training. Another set contains previous algorithms with the TSF, including Importance Advising (Torrey & Taylor, 2013) and EGPO (Peng et al., 2021). The original Importance Advising uses an intervention function based on the range of the Q-function: $I(s)=\max_{a\in A}Q_{\mathcal{D}(s,a)}-\min_{a\in A}Q_{\mathcal{D}(s,a)}$, where $Q_{\mathcal{D}}$ is the Q-table of the teacher policy. Such Q-table is not applicable in the Metadrive simulator with continuous state and action spaces. In practice, we sample $N$ actions from the teacher’s policy distribution and compute their Q-values on a certain state. The intervention happens if the range, i.e., the maximum value minus the minimum value, surpass a certain threshold $\varepsilon$. The EGPO algorithm uses an intervention function similar to the action-based intervention function introduced in section 3.2. All algorithms are trained with 4 different random seeds. In all figures the solid line is computed with the average value across different seeds and the shadow implies the standard deviation. We leave detailed information on the experiments and the result of ablation studies on hyperparameters in Appendix C.

We further investigate the intervention behaviors under different intervention functions. As shown in Fig. 4(c), the average intervention rate $\mathbb{E}_{s\sim d_{\pi_{b}}}\mathcal{T}(s)$ of TS2C drops quickly as soon as the student policy takes control. The teacher policy only intervenes during a very few states where it can propose actions with higher value than the students. The intervention rate of EGPO remains high due to the action-based intervention function: the teacher intervenes whenever the student act differently.

We also show different outcomes of action-based and value-based intervention functions with screenshots in the MetaDrive simulator. In Fig. 7 the ego vehicle happens to drive behind a traffic vehicle which is in an orange trajectory. With action-based intervention the teacher takes control and keeps following the front vehicle, as shown in the green trajectory. In contrast, with the value-based intervention the student policy proposes to turn left and overtake the front vehicle as in the blue trajectory. Such action has higher return and therefore is tolerable by $T_{\text{TS2C}}$, leading to a better agent trajectory.