Co-Imitation Learning without Expert Demonstration

We consider an agent interacting with the environment over a sequence of steps, which can be formalized as a Markov Decision Process $\mathcal{M}=(\mathcal{S},\mathcal{A},\gamma,\mathcal{P},\mathcal{R})$. Here, $\mathcal{S}$ is a set of states, $\mathcal{A}$ is a set of actions, $\gamma\in[0,1)$ is the discount factor, $\mathcal{P}$ is the state transition probabilities, and $\mathcal{R}:(\mathcal{S}\times\mathcal{A})\rightarrow\mathcal{R}$ is the reward function. At each step $t$, given the state $s_{t}\in\mathcal{S}$, the agent takes an action $a_{t}\in\mathcal{A}$ according to the policy $\pi$, and receives a reward $r_{t}\in\mathcal{R}$ from the environment. The target is to maximize the discounted sum of rewards over all step.

RL algorithms typically learn an effective control policy after many millions of interactions with the environment, and its learning process is rather inefficient. To overcome this deficiency, Self-Imitation Learning (SIL) Oh et al. (2018) stores past trajectories and allows the agent to self imitate its past good experiences to more effectively learn the policy $\pi$. Formally, the past episodes with cumulative rewards are stored in a replay buffer: $\mathcal{D}=\{(s_{t},a_{t},R_{t})\}$, where $R_{t}=\sum_{k=t}^{\infty}\gamma^{k-t}r_{k}$ is the discounted sum of rewards. Then, it tries to exploit agent’s past good experiences from the replay buffer to further improve the policy.

However, as discussed in the Introduction, SIL only attempts to mining the supervision from its own past experiences. Since there is no external information provided, the performance improvement could be less significant. In other words, SIL only emphasizes the past good experiences that the agent has utilized in previous training process, and thus limits its role in the environment exploration. Here we consider a “maze-search” problem as a supportive example. When there is only one agent to explore the maze, it is hard for the agent to walk out of the maze because it is easy to overestimate some observations by itself. Luckily, if we allow two agents to explore different areas of the maze and let them exchange their observations, it will become much easier for them to find the exit of the maze. Intuitively, different agents can generate different action decision policies, and subsequently may generate experiences with different views. Based on this motivation, we try to train two agents by exploiting their experiences from two views to further improve the learning performance in RL.

The framework of Co-Imitation Learning (CoIL) is demonstrated in Figure 2. It alternately explores the environment and exploits the peer agent’s past experience in a joint framework, which allows the agents to effectively utilize the information from two sources. Specifically, in the first step, by exploring the environment, two agents will independently obtain a series of trajectories (dotted line) and sparse rewards (red circle), which will be stored in their replay buffers. Then, by estimating the potential utility of each piece of trajectory, the agents can selectively imitate each other from the peer agent’s good experiences (black solid arrow) in the second step. As a result, the policies learned by cooperation can complement and reinforce each other, and thus can lead to a better performance and drive more effective exploration.

The goal of co-imitation learning (CoIL) is to train two agents to interact with the environment simultaneously, and let them imitate each other with peer agent’s past good experiences. Without loss of generality, in addition to the current agent $\pi$, we introduce its peer agent as $\hat{\pi}$ parameterized by $\hat{\theta}$, and store its past experiences in a peer replay buffer: $\hat{\mathcal{D}}=\{(\hat{s_{t}},\hat{a_{t}},\hat{R_{t}})\}$, where $\hat{s_{t}}$ is the state at time-step $t$, $\hat{a_{t}}$ is the action produced by $\hat{\pi}$, and $\hat{R_{t}}=\sum_{k=t}^{\infty}\gamma^{k-t}\hat{r_{k}}$ is the discounted sum of rewards.

Then, for agent $\pi$, to exploit the peer agent’s past good experiences in the replay buffer $\hat{\mathcal{D}}$, we follow the loss form of Oh et al. (2018) and propose the following actor-critic loss $\mathcal{L}^{co}$ for CoIL.

The loss function consists of two parts, which are defined as follows respectively,

where $(\cdot)_{+}=max(\cdot,0)$, $V_{\theta}(\cdot)$ is the state value function of the current agent, and $\beta^{co}$ is the trade-off hyperparameter for the value loss.

It is worth noticing that we use peer agent’s experiences in the replay buffer $\hat{D}$ to update the current agent $\pi$ by minimizing $\mathcal{L}^{co}$ in Eq 1. On one hand, by minimizing $\mathcal{L}_{policy}^{co}$, the policy $\pi$ will be optimized to produce consistent good action decisions with the peer agent. On the other hand, for an unseen state $\hat{s}$ in peer replay buffer $\hat{D}$, the state value function $V_{\theta}(\hat{s})$ will be optimized to have a more accurate estimation by minimizing $\mathcal{L}_{value}^{co}$.

Next, to make sure that the agents imitate the good experiences as much as possible but not the misleading ones, we propose to estimate the potential utility of each state-action pair based on the difference between the cumulative reward and the state value. Formally, given a state-action pair $(\hat{s},\hat{a})$ from peer agent’s experiences $\hat{D}$, its potential utility to learn $\pi$ is estimated as follows,

Intuitively, if the action given by the peer agent $\hat{\pi}$ will lead to a higher cumulative reward than the current agent $\pi$, it is likely that this state has not been seen by the agent $\pi$ but has a high potential contribution to the policy training. Obviously, such experience should be imitated by the agent to improve the current policy network $\pi_{\theta}$. After imitating these useful experiences, when the current agent $\pi$ meets these states again, it is expected to take better actions according to these past experiences instead of random decisions.

Next, for environment exploring, we perform reinforcement learning algorithm to update two agents $\pi,\hat{\pi}$. Our co-imitation learning framework can be implemented with any actor-critic methods. In this paper, we follow the settings in Oh et al. (2018), and employ the advantage actor-critic (A2C) Frobeen (2017) and proximal policy optimization (PPO) as the basic algorithm of reinforcement learning. For convenience, the loss of RL is denoted as $\mathcal{L}^{rl}$.

The process of the proposed co-imitation learning (CoIL) approach is summarized in Algorithm 1. Firstly, the environment $\mathbf{E}$, observation space $\mathcal{O}$, and action space $\mathcal{A}$ are given. Then the parameters $\theta$ and $\hat{\theta}$ of the two agents are randomly initialized. We introduce two episode buffers $\mathcal{E},\hat{\mathcal{E}}$ to store the two agent’s past experiences, respectively, which are initialized as empty sets. Moreover, we also initialize two replay buffers $\mathcal{D},\hat{\mathcal{D}}$ as an empty set for performing co-imitation. At each iteration, past experiences of two agents generated by interacting with environment are collected in the episode buffers $\mathcal{E}$ and $\hat{\mathcal{E}}$ respectively. Then, to update the replay buffers, we compute the cumulative rewards $R$ and $\hat{R}$ for all instances in $\mathcal{E}$ and $\hat{\mathcal{E}}$, and clear the episode buffers. After that, we perform reinforcement learning algorithm (minimizing $\mathcal{L}^{rl}$) and co-imitating (minimizing $\mathcal{L}^{co}$) from the replay buffers $M$ times to update the policy networks for each agent.

In this subsection, we evaluate the performance of compared methods by plotting the reward curve with the number of training iterations increases. The results with PPO as the base algorithm are demonstrated in Figure 3. We implement our CoIL approach base on PPO algorithm shown as PPO+CoIL-$i$, which denotes the performance of the $i$-th agent. Moreover, we also implement a combination with self-imitation learning shown as PPO+SIL.

From Figure 3, we can observe that in both standard and delayed-reward environments, the proposed CoIL approach significantly outperforms the other methods in all cases. CoIL can achieve higher reward with fewer training iterations in general. When comparing with the PPO method that only explores the environment, the proposed CoIL method and the SIL method are always superior by utilizing the supervised information from past experiences. This phenomenon implies that it is more effective to learn policy by simultaneously exploiting the past experiences and exploring the environment. The SIL method, which aims to exploit past good experience of the agent itself, can improve learning based on PPO in some cases but fail to achieve higher performance in other cases (such as in HalfCheetah, DelayedHalfCheetah, and DelayedAnt). One possible reason is that SIL only exploits its own supervision without external information. In constract, the proposed CoIL approach, which allows the agent to obtain external supervision by imitating the peer agent’s past good experiences, can complement and reinforce each other, and thus lead to a higher performance. Importantly, the proposed CoIL approach can significantly improve the performance of the Spinning Up benchmark Achiam (2018) implementation to a new level on all tasks. Especially in HalfCheetah environment, CoIL can get a reward of around 5000, while other methods can only get less than 2000. And in Swimmer environment, CoIL can achieve a reward of 360, which is significantly higher than the best results reported in previous literatures.

In standard MuJoCo environments, the reward structure is smooth and dense so that agent always receives a reasonable amount of reward and learns much faster in this type of domain. Thus, we further validate the robustness of our approach by delaying the reward the agent received from the environment. The result is demonstrated in the bottom row of Figure 3. In general, all compared methods for the delayed-reward tasks do not perform as well as those in the standard environments. However, it is clearly shown that the gap between CoIL and other methods is larger on delayed-reward tasks compared to standard tasks. More surprisingly, the CoIL methods can achieve competitive even higher performance in DelayedAnt and DelayedWalker2d environments. The results in both standard and delayed environments consistently validate the effectiveness of co-imitation learning, i.e., the agents can benefit from each other by imitating the peer agent’s experience.

In this subsection, to examine the effectiveness of the proposed methods in sparse-reward environment, we follow the setting in Oh et al. (2018) to employ A2C as the base algorithm and perform experiments in various MiniGrid environments. Figure 4 demonstrates the performance comparison in four environments, where A2C+CoIL and A2C+SIL respectively represents the combination of co-imitation learning and self-imitation learning with A2C.

From Figure 4, it can be observed that the proposed CoIL approach significantly outperforms the other methods in all cases. The policy learned by the proposed CoIL approach can find the goal with fewer training iterations. Especially CoIL can solve KeyCorridorS3R1 task, while A2C and SIL both fail. It worth noting that the trajectory sequences generated on these tasks have strong dependence. For example, in the DoorKey domain, the agent needs to pick up the key, open the door and get to the goal, then achieves a sparse reward. Due to the sequential dependency for tasks, the policy learned by A2C has a low chance to find the goal, and thus leads to the slow convergence (such as in Empty and DoorKey) or even failure on some tasks (such as in KeyCorridor). The SIL method also fails in some environments due to similar reasons. Intuitively, if the current agent cannot find the goal or cannot produce good experiences, the SIL method will degenerate to standard A2C algorithm, and thus will not promote the learning ability to drive effective exploration. The proposed CoIL approach can effectively alleviate this issue. Intuitively, two agents have a greater chance to find the goal and generate good experiences. After exchanging past experiences and selectively imitating each other, two agents can benefit from each other’s good decisions.

Moreover, to investigate whether our CoIL approach is complementary to count-based method (BEBOLD), we further perform experiments on other four relatively hard environments. The results in Figure 5 show a consistent conclusion. When combined with BEBOLD approach, the exploration ability of all the compared methods are improved. It is clearly shown that the proposed approach still outperforms the other algorithms.This result also validates that co-imitation learning and count-based exploration can be well incorporated to solve the hard-exploration problems.

To study how different methods can effect the exploration of the agent, we further analyze the visualized results of visitation counts in DoorKey and MultiRoom-N6 task. The target is to let the agent across the rooms and reach the green goal square. Moreover, we define visitation counts $N(s)$ at every state as the metric to measure the effectiveness of an exploration strategy. Figure 1 and  6 shows the heatmap of visitation counts at 1.5M and 800K steps respectively. It can be observed that the policy learned by our CoIL method can drive agent to explore more effectively. This visualized result also validate the superiority of the proposed co-imitation learning.