Reinforcement Learning with Supervision from Noisy Demonstrations

We consider an agent interacting with the environment over a sequence of steps, which can be formalized as a Markov Decision Process $\mathcal{M}=(\mathcal{S},\mathcal{A},\gamma,\mathcal{P},\mathcal{R})$. Here $\mathcal{S}$ is a set of states, $\mathcal{A}$ is a set of actions, $\gamma\in[0,1)$ is the disount factor, $\mathcal{P}$ is the state transition probabilities, and $\mathcal{R}:(\mathcal{S}\times\mathcal{A})\rightarrow\mathcal{R}$ is the reward function. At each step $t$, with the state $s_{t}\in\mathcal{S}$, the agent takes an action $a_{t}\in\mathcal{A}$ according to the policy $\pi$, and receives a reward $r_{t}\in\mathcal{R}$ from the environment. The target is to maximize the discounted sum of rewards over all steps.

RL algorithms typically learn an effective control policy after many millions of steps, which is unacceptable in most real tasks. To overcome this problem, we can provide the agent with expert demonstrations to more effectively learn the policy $\pi(\cdot)$. Formally, a trajectory is a sequence of observations, actions and rewards, $\sigma=((s_{1},{a^{*}}_{1},r_{1}),(s_{2},{a^{*}}_{2},r_{2}),...,(s_{T},{a^{*}}_{T},r_{T}))$, where ${a^{*}}_{i}$ is the $i$-th action from experts, and $s_{T}$ is the terminal state. For convenience, we denote $(s_{i},{a^{*}}_{i})$ as an instance corresponding to the $i$-th step of the trajectory $\sigma$.

As discussed in the Introduction, the trajectory $\sigma$ collected from expert demonstration is usually imperfect, because it may contain serious noise or even some misleading information. For example, to train an agent to play Go game with human demonstrations, we should be aware that a trajectory win the game does not mean that it is optimal. The optimal demonstrations are hard to obtain caused by data collection noise or produced by the immaturity of the expert. It is more common that some parts of the demonstrations are optimal while the others are not. A naive approach to handle noisy demonstrations is to simply filter out the trajectories with low reward. However, a low reward does not necessarily imply that all the demonstration steps in the trajectory are useless. Instead, even for a noisy trajectory, some steps of it may still provide important information for the policy optimization. Moreover, even for the same step of a trajectory, it may contribute differently at different stages of the learning process, as the model changes.

Based on the above observations, we propose to adaptively estimate the potential utility of each instance at each iteration during the policy learning. Formally, we introduce a weight variable $w_{i}$ for each demonstration instance $(s_{i},{a^{*}}_{i})$ to estimate the potential value. Obviously, more useful instances should receive higher values of $w_{i}$, while noisy instance should receive lower values of $w_{i}$. After that, we try to learn the policy $\pi(\cdot)$ by allowing the agent to interact with the environment and exploit the supervised information from weighted demonstration instances.

First of all, we compare the proposed framework with existing reinforcement learning with expert demonstrations (RLED) methods in Figure 1. The framework of previous approaches (Gao et al., 2018; Piot et al., 2014; Kim et al., 2013; Chemali and Lazaric, 2015; Silver et al., 2016) are demonstrated in Figure 1(a). They typically work in a two-steps way. In the first step, a supervised learning algorithm is employed to pre train the policy network based on the demonstrations. Then in the second step, the policy is optimized by allowing the agent to explore the environment. Obviously, the demonstration learning and environment exploring are performed separately, making it less possible to adaptively exploit the supervised information from the demonstration trajectories. More importantly, all the instances of the trajectories are uniformly used, without considering their potential utility. When the demonstration trajectories are imperfect, some noisy instances will mislead the policy optimization, and subsequently hurt the learning performance.

In contrast, the proposed LfND framework is demonstrated in Figure 1(b). First of all, it simultaneously learns from the demonstrations and the environment in a joint framework, which allows the agent to adaptively utilize the information from two sources. Unlike previous methods that assume the demonstrations are perfect, the proposed LfND framework introduces a weight variables $w_{i}$ for each instance $(s_{i},a^{*}_{i})$ to estimate its potential utility. Noticing that because the utility of a specific instance varies as the model changes, the weights are adaptively updated in different learning iterations. By optimizing the weighted loss over all instances, it is expected to emphasize the positive effects of good demonstrations while eliminate the negative effects of noisy demonstrations.

In the following part of this section, we will firstly introduce the joint training of the policy $\pi(\cdot)$ based on the noisy demonstrations and environment exploring, and then discuss on how to calculate the weights to accurately estimate the potential utility of demonstration instances.

To allow joint training of the agent by simultaneously learning from the demonstrations and exploring the environment, we define a joint objective function as in Eq. 1.

where $\ell_{d}$ and $\ell_{e}$ denote the loss for demonstration learning and environment exploring, respectively, and $\lambda$ is a trade-off parameter.

For demonstration learning, as previously discussed, given a set of noisy trajectories, if we can estimate the potential utility of each instance accurately at each time, it is more effective to utilize these supervised information for policy joint optimization. In other words, our target is to utilize the right demonstration instances at the right time to learn an effective policy. Specifically, in the set of demonstration trajectories $\Sigma=\{\sigma^{1},\sigma^{2},...,\sigma^{m}\}$, each trajectory $\sigma^{i}$ consists of $n^{i}$ (state, action, reward) instances as follows:

Given a state $s^{i}_{j}$, based on the currently learned parameters $\theta$ of the policy network, the agent will return an action:

Note that we assume both $a^{i}_{j}$ and ${a^{*}}^{i}_{j}$ is a distribution over all actions, where each element describes the probability of taken the corresponding action. Then, to learn the policy $\pi(\cdot)$ from demonstration, we try to minimize the distance between $a^{i}_{j}$ and ${a^{*}}^{i}_{j}$, forcing the agent to behave similarly to the expert. Formally, we define a weighted cross-entropy loss function over all trajectories as Eq. 2.

Obviously, by minimizing the loss function, the policy $\pi$ will be optimized to produce consistent actions with the experts, while the negative effect of noisy demonstrations will be eliminated by the weight $w_{j}^{i}$. Note that we only introduce this straightforward implementation to validate our idea of learning from demonstrations. Other more advanced strategies may be employed to further improve the performance.

For environment exploring, we focus on policy gradient methods, which are popular and can handle tasks with continuous actions compared to Q-learning methods. As an example, we can employ the trust region policy optimization (TRPO) method (Schulman et al., 2015) to define the loss function $\ell_{e}$ as follows:

where $\pi^{old}$ is the policy parameters before the update, $\hat{A}$ is an estimator of the advantage function, and the $\beta$ is the parameters of max trust region between the new policy and the old policy.

Alternatively, one can also use proximal policy optimization (PPO) algorithm (Schulman et al., 2017) to define $\ell_{e}$ as follows:

where $r_{t}(\theta)$ denote the probability ratio $r_{t}(\theta)=\frac{\pi(\bm{a|s})}{\pi^{old}(\bm{a|s})}$. The second term of Eq 4, $clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}$, modifies the surrogate objective by clipping the probability ratio, which prevents a big gap between the new and old policies (Schulman et al., 2017).

Finally, by substituting Eqs. 2 and 3 or 4 into Eq. 1, we get the final objective function of the proposed LfND approach. It can be observed that both the two terms of objective function are consistently trying to learn the policy distribution, and can be effectively solved with policy gradient methods.

Next, we discuss the estimation of the potential utility of each instance, i.e., calculating the weights $w_{j}^{i}$ for each instance $(s^{i}_{j},{a^{*}}^{i}_{j})$. Intuitively, given a state $s_{j}^{i}$, if the action given by the expert will lead to a higher expected reward than the currently learned policy, it is likely that the expert policy is superior to the agent policy, and thus such demonstration could be utilized to improve the policy network. We thus define the weight as follows:

where

Here, $\mathds{1}{\{\cdot\}}$ is the indicator function that the value is 1 when the condition is met, otherwise it is 0. $Q_{\sigma^{i}}(s_{j}^{i},{a^{*}}_{j}^{i})$ is the state-action value function calculated from expert, which estimates the long-term reward of performing action ${a^{*}}_{j}^{i}$ in state $s_{j}^{i}$. Note that this term is an accurate value because the whole trajectory $\sigma^{i}$ is given. $V_{\pi}(s_{j}^{i})$ is the state value function from policy $\pi$, which estimates the value of the state $s_{j}^{i}$. It can be observed Eq. 5 has a similar form with the advantage function $A_{\pi}(s,a)=Q_{\pi}(s,a)-V_{\pi}(s)$ (Baird III, 1993; Wang et al., 2015). The difference is that advantage function aims at describing how good it is to select action $a$ in state $s$ under the current policy $\pi$, and its $Q_{\pi}$ function is approximated by policy $\pi$.

Remarks: As we discussed before, the utility of a specific demonstration may vary over different learning stages as the model updates. For example, at the begining, the parameter of policy network $\theta$ is usually initialized to a small number that makes the $V_{\pi}(s_{j}^{i})$ is also a small number. Thus nearly all instances will be used to optimize the policy $\pi$ because the $Q_{\sigma^{i}}(s_{j}^{i},{a^{*}}_{j}^{i})-V_{\pi}(s_{j}^{i})$ will be greater than zero for most instances. As the learning process goes on, the $V_{\pi}(\cdot)$ will increase, and the weights of some less useful instances will be zero so that these instances will not participate in the policy optimization. Note that, it can not work if we use advantage function $Q_{\pi}(s,a)-V_{\pi}(s)$ instead of $Q_{\sigma^{i}}(s_{j}^{i},{a^{*}}_{j}^{i})-V_{\pi}(s_{j}^{i})$, because $Q_{\pi}$ and $V_{\pi}$ are both approximated by policy $\pi$, which can not estimate weights of instances.

Another case is that even the demonstration $(s_{j}^{i},{a^{*}}_{j}^{i})$ is noise, it is still possible to contribute to the policy optimizatioin. Intuitively, this instance may have a negative effect on optimizing a policy with high performance, but it still has a positive effect on the randomly initialized policy or a policy with poor performance. This also demonstrate that the proposed approach can dynamically adjust the weights through the current policy $\pi$, and adaptively utilize the right instances at right time to jointly optimize the policy.

Although the weight defined in Eq. 5 can eliminate the negative effects of noisy demonstrations, it is unable to distinguish demonstrations with different degrees of positive effects. Therefore, we provide some other optional solutions to the weight estimation. Specifically, we define a linear form of weight as shown in Eq 6.

where $\delta$ is a hyperparameter, say, $\delta=10$.

Similarly, we also define a logarithmic form of the weight as shown in Eq 7.

Obviously, Eq 6 and Eq 7 can better describe the importance of different demonstration instances. The comparison among these different implementations will be studied in the experiment section.

The process of the approach is summarized in Algorithm 1. Firstly, environment $\mathcal{E}$, observation space $\mathcal{O}$, action space $\mathcal{A}$ and set of trajectory $\Sigma$ are given. Then the parameters $\theta$ represents the policy network is randomly initialized. We also introduce the memory $\mathcal{H}$ to store tuples of agent interactions, which is initialized as an empty set. At each iteration, some trajectories generated by interaction between agent and environment are stored in memory $\mathcal{H}$, and then $w_{j}^{i}$ for each demonstration instance will be caculated by Eq. 6 or Eq. 7, finally $\theta$ will be optimized until convergence or desirable performance. When updating $\theta$ by minimizing the loss function $\ell$ in Eq. 1, $\ell_{e}(\theta)$ can be solved with existing policy gradient algorithms.

To validate the effectiveness of the proposed approach, we perform experiments in six commonly used MuJoCo (Todorov et al., 2012) environments, which are implemented in OpenAI Gym (Brockman et al., 2016). HalfCheetah: a simulated cheetah robot system with 17-dimensional state and 3-dimensional action, whose goal is to make the robot run as far as possible. Hopper: a one-legged robot, which is expected to hop forward as fast as possible in this environment. There is a 11-dimensional state space and a 3-dimensional action space. Humanoid: there is a bipedal robot in the Humanoid system with 376-dimensional state and 17-dimensional action. Its goal is to make it walk forward as fast as possible without falling over. HumanoidStandup: the HumanoidStandup system with 376-dimensional state and 17-dimensional action is trying to make the robot standing up as long as possible. InvertedPendulum: the goal of this environment is to prevent an agent which moves along a frictionless track from falling over. In this environment, a 4-dimensional state and a 1-dimensional continuous action is provided. Walker2d: the goal of Walker2d system is to make the robot run as fast as possible. The state is in 17-dimensional and the action is in 6-dimensional. Screen shots of the six environments are shown in Figure 2. For each environment, 10 demonstration trajectories are provided by a well trained agent, among which 5 trajectories contain noisy demonstrations. Specifically, all trajectories are generated by the trained agent, and each trajectory is from the initial state to the terminal state, where the noisy demonstrations are produced by the immature agent.

We respectively employ two state-of-the-art reinforcement learning algorithms, i.e., TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017), as the base model to implement our approach as well as other compared methods. The following methods are compared in the experiments:

• •

  Trust Region Policy Optimization (TRPO): A RL method performs strategy search in the trust region, and tries to ensure that the strategy of the next iteration will be better than the current strategy (Schulman et al., 2015).

• •

  Proximal Policy Optimization (PPO): A RL method that tries to optimize the lower bound of the clipped and unclipped objectives (Schulman et al., 2017).

• •

  Imitation Learning only (IL): Supervised imitation from expert demonstrations without any environment interaction.

• •

  Learning policy by AlphaGo (LbA): This method combines exploration and demonstration based on policy gradient method as in AlphaGo. Specifically, it firstly pre trains the policy network with expert demonstrations by supervised learning, and then trains the network by exploring the environment (Silver et al., 2016).

• •

  Policy Optimization from Demonstration (POfD): A latest RLED method that optimizes the JensenShannon (JS) divergence between the current policy and expert’s policy by adversarial training on demonstrations (Kang et al., 2018).

• •

  LfND: The approach proposed in this paper.

• •

  LfND without weight (LfND-noW): A degenerated version of the proposed method, which uses all the instances of the trajectories uniformly in the learning process.

The parameters of the two base models are set as recommended in the corresponding literatures. Specifically, for TRPO, the penalty coefficient $\beta$ is set to 0.01. For PPO, the parameter $\epsilon$ is set to 0.2. We use the same parameter setting for both our approach and the compared methods. The trade-off parameter $\lambda$ of LfND is set to 1 as default. We use linear form of weight in experiments and set $\delta=10$ as default.

We evaluate the performance of compared methods by plotting the reward curve with the number of training iterations increases. The results with TRPO as the base model are presented in Figure 3, while the results with PPO as the base model are shown in Figure 4. It is worthy to note that when comparing with a specific RL base algorithm, we use the same algorithm to implement LfND and all other methods for fair comparison. In addition, we also show the average reward from noisy demonstrations in the figures, denoted by Expert.

From Figures 3 and 4, we can observe that no matter which base model is used, the proposed LfND approach outperforms the other methods in most cases. LfND can achieve higher reward with fewer training iterations in general. The IL method which imitates the demonstration without exploring environment is not effective in all environments, and typically cannot reach the average reward of demonstrations. This phenomenon implies that it is important to learn the policy by both exploiting the demonstration and exploring the environment, especially when the demonstrations are noisy. When comparing with the methods that only exploring the environment, i.e., TRPO or PPO, our method is always superior by utilizing the supervised information from demonstration. In contrast, the other two methods POfD and LbA, which also combine environment exploring and demonstration exploiting, are less robust. They outperforms TRPO or PPO in some environments but loss in the others, probably misled by the noisy demonstrations since they utilize all trajectories without distinction. The LfND-noW approach, which is a variant of the proposed approach without weighting scheme, can improve the performance quickly at the early stage but fail to achieve higher performance persistently. One possible reason is that at the early stage, the learned policy is quite poor, and thus most demonstrations are superior than it and can provide useful information to guide the training. After some iterations, when the policy getting more effective, the noisy demonstrations will hurt the performance. This also validates that a specific demonstration may contribute differently at different learning stages, and thus it is important to adaptively adjust the weights of demonstrations as in LfND. The results in Figures 3 and 4 are consistent in general, validating that the proposed strategy can be effectively incorporated with different reinforcement learning base models.

To further validate the superiority of our method, we also show the average and maximum reward achieved by different algorithms within 500 learning iterations. The mean results over 5 times repeated experiments with different random seeds are recorded. Results on average reward and maximum reward are reported in Tables 1 and  2, respectively. It can be observed that the proposed approach LfND outperforms the other methods in most cases with regard to both average reward and maximum reward. In other words, the LfND method can consistently keep a higher performance in the learning process, and also obtain a better final policy.

In this subsection, to further examine the robustness of the proposed method, we perform the experiments with different noise ratios of demonstrations. Specifically, we vary the noise ratio from 0 to 1 with interval of 0.1, where 0 ratio means there is no noise demonstration. For each ratio, we perform the experiments for five methods and record their average reward over 500 learning iterations. Then we calculate the improvement ratio of the proposed method over the other methods. Figure 5 shows the overall improvements of our method over other methods, one color for a method. Due to space limitation, we report the results with TRPO as the base model in three environments, i.e., HalfCheetah, Humanoid and Walker2d. It can be observed that the LfND approach always outperforms other compared methods with all ratios of noise demonstrations. With the increase of noise ratio, the superiority of LfND over the other methods becomes more significant. These results indicate that the proposed LfND method can effectively and robustly learn the policy from noisy demonstrations.

In this subsection, we further perform experiments to examine whether potential utility of demonstrations are accurately estimated by our algorithm. Specifically, we firstly calculate the average weight of each demonstration instance during the learning process. Then we sort all the instances in increasing order of their average weights, and separate them into 10 groups with equal size. A smaller group ID implies smaller weights for the demonstrations in the group. After that, we fed the demonstrations from each group into the imitation learning (IL) algorithm to learn a policy, and record its performance on average reward, maximum reward and final reward during 500 iterations, respectively.

In Figure 6, the reward curves with different groups are plotted in HalfCheetah, Humanoid and Walker2d. The IL_Mean, the IL_Final and the IL_Max denote the mean, final, maximum rewards respectively in 10 groups. It can be observed that the higher weight of the demonstrations are used for imitation learning, the higher the performance.

Intuitively, high quality demonstration can improve the performance of imitation learning, which implies that the LfND approach can accurately distinguish the quality of different demonstrations by estimating its potential utility. In other words, these results also validate that the LfND approach can effectively emphasize more helpful demonstration while filter out noisy ones.

As discussed previously, we may weight the instances in different forms. In this subsection, we further compare the results of our method with four different weighting strategies: oneZero denotes the one-zero form of weight as in Eq 1; linear10 and linear20 denote the linear forms of weight with hyperparameter of 10 and 20 respectively as in Eq 6; log denotes the logarithmic form of weight as in Eq 7.

In Figure 7, we plot the reward curves in HalfCheetah, Humanoid and Walker2d environments for six methods: PPO, TRPO, LfND-oneZero, LfND-linear10, LfND-linear20, LfND-log. Firstly it can be observed that LfND with all weighting strategies can achieve better performance than the base RL algorithms. LfND-log, LfND-linear10 and LfND-linear20 outperforms LfND-oneZero in all cases. One intuitive reason is that both linear weight form and log weight form can emphasize the more helpful demonstrations while filter out noisy ones. These results also validate that the superiority of LfND shown in previous experiments is from the proposed strategy of adaptively exploiting noisy demonstrations.