Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks

A standard reinforcement learning problem is formalized as a Markov decision process (MDP), defined as a tuple $\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$ (Sutton and Barto, 2018), where

• •

  $\mathcal{S}$ is a set of environment states which encodes relevant information for an agent’s decision.

• •

  $\mathcal{A}$ is a set of agent actions.

• •

  $\mathcal{P}$ is the state transition function which describes $p(s^{\prime}|s,a)$, i.e., the probability of entering state $s^{\prime}$ when an agent takes action $a$ in state $s$.

• •

  $\mathcal{R}$ is a reward function. $r(s,a,s^{\prime})$ denotes the scalar reward agent received on transition from $s$ to $s^{\prime}$ under action $a$.

• •

  $\gamma\in[0,1]$ is a discount factor that indicates how much the agent values an immediate reward compared to a future reward.

As a concrete example, Atari Montezuma’s Revenge (Bellemare et al., 2013) (Fig. 1) is one of the most challenging video games for reinforcement learning research. The game has rich visual features, complicated game dynamics, and very sparse rewards. Modeled as an MDP, the state is the game image frame, or a stack of frames to capture temporal information. The agent controls the avatar by choosing an action from a discrete set of actions at every timestep. The agent receives the reward from the game engine in the form of game scores. We will use this game as a running example throughout the survey.

Additionally, $\pi:\mathcal{S}\times\mathcal{A}\mapsto[0,1]$ is a policy which specifies the probability distribution of selecting actions in a given state. The goal for a learning agent is to find an optimal policy $\pi^{*}$ that maximizes the expected cumulative reward. One could optimize $\pi$ directly, while alternatively many of the algorithms are based on value function estimation, i.e., estimating the state value function $V^{\pi}(s)$ or the action-value function $Q^{\pi}(s,a)$.

The state value function for a given policy $\pi$ is defined as (Sutton and Barto, 2018)

A corresponding action value function, $Q^{\pi}(s,a)$, also exists and is given by

and the advantage function $A^{\pi}(s,a)$, is defined as

The state value function $V^{\pi}(s)$ measures the expected cumulative reward to be in a particular state $s$ and following policy $\pi$ afterward. The action-value function $Q^{\pi}(s,a)$ defines the same quantity but for taking a particular action $a$ when in state $s$ and following policy $\pi$ afterward. The advantage function tells us the relative gain (“advantage”) that could be obtained by taking a certain action compared to the average action taken at that state (Wang et al., 2016).

Several successful RL algorithms that seek to estimate these quantities directly have been developed, including Q Learning (Watkins and Dayan, 1992) and advantage actor-critic (see, e.g., Sutton and Barto (2018)). For example, Q-learning seeks to learn the state-action value function for the optimal policy, $Q^{\pi^{*}}(s,a)$, and the policy is then given by $\pi^{*}(s)=\arg\max_{a}Q^{\pi^{*}}(s,a)$. Nowadays, deep neural networks are often used as function approximators to estimate and optimize $\pi$, $V$, and $Q$.

An important challenge in RL is to balance exploration vs. exploitation when an agent selects its action. Exploration allows the agent to improve its current knowledge. Exploitation chooses the greedy action to maximize reward by exploiting the agent’s current knowledge. A simple strategy ($\epsilon$-greedy) chooses a random action with probability $\epsilon$ and chooses the greedy action (the action with the highest Q value) with probability $1-\epsilon$ (Sutton and Barto, 2018). A more sophisticated strategy uses a Boltzmann distribution for selecting actions based on the current estimate of Q function (Sutton and Barto, 2018):

where $\tau$ is a temperature constant that controls the exploration rate.

The learning frameworks surveyed in this paper are inspired by, an extension of, or combined with traditional imitation learning algorithms. The standard imitation learning setting (Fig. 2 and Fig. 4a) can be formulated as MDP\$R$, i.e. there is no reward function $\mathcal{R}$ available. Instead, a learning agent (the imitator) records expert (the demonstrator, could be a human expert or an artificial agent) demonstrations in the format of state-action pairs $\{(s_{t},a^{*}_{t})\}$ at each timestep, and then attempts to learn the task using that data.

One approach is for the agent to learn to mimic the demonstrated policy using supervised learning, which is known as behavioral cloning (Bain and Sommut, 1999). A second approach to imitation learning is called inverse reinforcement learning (IRL) (Abbeel and Ng, 2004) which involves learning a reward function based on the demonstration data and learning the imitation policy using RL with the learned reward function. These two approaches constitute the major learning frameworks used in imitation learning. Comprehensive reviews of these two approaches can be found in Argall et al. (2009); Hussein et al. (2017); Osa et al. (2018); Arora and Doshi (2018); Fang et al. (2019). More recently, generative adversarial imitation learning (GAIL) (Ho and Ermon, 2016) has been proposed, which utilizes the notion of generative adversarial networks (GAN) (Goodfellow et al., 2014).

Importantly, all of these approaches assume that $(s_{t},a^{*}_{t})$ pairs are the only learning signal to the agent and that both $s_{t}$ and $a^{*}_{t}$ are available to the agent. Unfortunately, access to the optimal actions, $a^{*}_{t}$, is not always plausible, as the task might be too complex for human demonstrators to perform. Therefore, there has recently been a great deal of interest in the research community in learning frameworks that utilize learning signals other than optimal action information, and it is these techniques that we review in this survey. Before doing so, however, we briefly describe the three IL frameworks described above, i.e., (1) behavioral cloning (BC), (2) inverse reinforcement learning (IRL), and (3) generative adversarial imitation learning (GAIL).

Next, we introduce several sequential decision-making tasks that are commonly used today to test the algorithms discussed above. Before deep learning, sequential decision-making models and learning algorithms were often confined to task domains with low dimensional state space such as 2D gridworld and mountain car (Sutton and Barto, 1998). The emergence of deep neural networks (LeCun et al., 2015) has enabled these models and algorithms to solve significantly more challenging tasks. These tasks include Atari 2600 video games (Bellemare et al., 2013; Machado et al., 2018) in which the state space could be high-dimensional raw game images. The platform has 60 unique video games that span a variety of dynamics, visual features, reward mechanisms, and difficulty levels for both humans and AIs. Montezuma’s Revenge (Fig. 1) is one of the most difficult games due to very sparse rewards. Hence it is one of the most challenging games in terms of exploration.

Another example is robotic locomotion tasks using MuJoCo (Todorov et al., 2012), a simulator with a physics engine and multi-joint dynamics to study complex dynamical systems in contact-rich behaviors. It is the first full-featured simulator designed from the ground up for the purpose of model-based optimization, and in particular optimization through contacts (Todorov et al., 2012).

Recently, much effort has been spent on moving from simulation to real-world applications, from the navigation robots (e.g., TurtleBot (MacGlashan et al., 2017)), robotic manipulators (e.g., Sawyer robot arm (Xu et al., 2018a)), to autonomous driving vehicles (Yu et al., 2018). These are typically tasks with high-dimensional state space at which humans are particularly good. Examples of the tasks can be seen in Fig. 3. In some of the tasks such as board games, reinforcement learning agents have already surpassed human expert performance (Silver et al., 2016), and could perform even better without human knowledge (Silver et al., 2017, 2018). For example, Silver et al. (2017) have shown that a pure RL agent that learns to play Go by itself from scratch can outperform an IL/RL hybrid agent (Silver et al., 2016) which first learns to imitate expert human Go players’ moves. However, RL agents and algorithms still face significant challenges in solving many of the tasks we discuss in this survey.

In the RL setting (a fixed MDP), a stationary reward function $R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is defined as a means by which to specify a fixed task. Due to this stationarity, RL algorithms are able to seek policies that exhibit a notion of optimality with respect to a statistic dependent upon this distribution. That is, RL algorithms seek $\pi^{*}=\text{arg}\;\text{max}_{\pi}J(\pi)$, where $J(\pi)=\mathbb{E}_{\pi}\left[\sum_{t}\gamma^{t}R(s_{t},a_{t})\right]$ is well-defined. The RL community has proposed several algorithms to accomplish this task, including policy gradient techniques (Sutton et al., 2000) and actor-critic techniques (Grondman et al., 2012).

Due to the success of RL, some researchers have proposed techniques for learning from human feedback that interpret the feedback as the reward function itself (Isbell et al., 2001; Tenorio-Gonzalez et al., 2010). Intuitively, this interpretation amounts to assuming that the human feedback provides an instantaneous rating of the agent’s current decision. For example, Pilarski et al. (2011) propose a technique for learning from human feedback that uses the feedback as the reward function in an actor-critic algorithm. During training, the human trainer provides a positive ($r=+0.5$) or negative ($r=-0.5$) reward to the learning agent. In the absence of a human-delivered reward, they simply assume the feedback is neural ($r=0$). For their experimental setting, they find that the proposed algorithm can learn useful policies when the human feedback is consistent, but that policy quality degrades when the feedback becomes inconsistent (i.e., becomes less stationary). Another such technique is Advise (Griffith et al., 2013; Cederborg et al., 2015), in which the human feedback is used as the reward signal in a policy-gradient-like algorithm. More specifically, the probability of an action being a good action is:

where $\Delta s,a$ is the difference between the number of “right” and “wrong” labels that the human provided for action $a$ in state $s$. Notably, this approach coarsely takes human error into consideration using the $\mathcal{C}$ parameter. Let $\mathcal{C}$ denote the probability that an evaluation of an action choice is correctly provided by the human teacher ($\mathcal{C}$ = 0.5 is a random non-informative teacher, and $\mathcal{C}$ = 1 is a flawless teacher) (Cederborg et al., 2015). Suppose $P_{q}(a)$ is the probability of selecting action $a$ by an RL agent (e.g., according to Boltzmann distribution, Eq. 4), the final policy during learning is determined using both $P_{c}$ and $P_{q}$:

In this way the algorithm combines knowledge it learned from interacting with the environment and knowledge it gained from human evaluative feedback.

Alternatively, several methods interpret the feedback signal as a value-like quantity (Knox and Stone, 2009; MacGlashan et al., 2017). Intuitively, this interpretation amounts to assuming that the human feedback provides a rating of the agent’s current decision with respect to some forecast of future behavior.

One such technique is the TAMER algorithm (training an agent manually via evaluative reinforcement) (Knox and Stone, 2009), in which it is assumed that the human has in mind a desired policy $\pi_{H}$, and the feedback given at a time instant $t$, $H(s_{t},a_{t})$ roughly corresponds to $Q^{\pi_{H}}(s_{t},a_{t})$ (defined in Eq. 2). TAMER agents use supervised learning with all the feedback collected up to time $t$ to calculate the current estimate of $H$, $\hat{H}$, e.g., through minimizing a standard squared loss (Warnell et al., 2018):

Then the agent acts, in the next state, according to the policy

in a fashion similar to Q Learning since we interpret $\hat{H}^{*}$ as an approximation for $Q^{\pi_{H}}$. Because the TAMER algorithm interprets the human feedback to be the value corresponding to a fixed (ideal) policy $\pi_{H}$, the implicit assumption made is that the feedback given is independent of the agent’s current policy and depends only on the quality of an agent’s action selection. This type of human feedback model is called policy-independent models.

Alternatively, we could have policy-dependent models in which the feedback depends on the agent’s current policy. An action selection may be rewarded or punished more depending on how often the agent would typically be inclined to select it. For example, the human may greatly reward the agent for deviating from its current policy to take a slightly better action (though this action may still be sub-optimal), and stop rewarding this action as the agent consistently adopts this action (MacGlashan et al., 2017). This phenomenon is known as diminishing returns and is policy-dependent (MacGlashan et al., 2017). The COACH (convergent actor-critic by humans) framework has leveraged the idea of policy-dependent feedback and assumes instead that the human feedback corresponds to the advantage (Eq. 3) for the current policy (MacGlashan et al., 2017). Intuitively, the advantage function communicates how much better or worse the agent’s behavior is when deviating from its current policy. Algorithmically, COACH uses the feedback to replace the advantage function in calculating the policy gradient in an advantage actor-critic algorithm. Note that the human trainers do not need to provide feedback at every timestep like other evaluative feedback approaches.

With the advent of deep learning, several researchers in the community have recently begun attempting to use these techniques in the context of more challenging, high-dimensional state spaces. For example, Warnell et al. (2018) propose a technique that enables the use of TAMER for pixel-level state spaces in Atari games. To overcome the difficulty faced by trying to learn functions over such state spaces from sparse feedback, the authors propose to use a combination of a pre-trained deep autoencoder for state representation and a feedback replay buffer to allow for off-policy updates. Arumugam et al. (2019) report that a similar approach is successful when applying COACH to pixel-level state spaces. Aside from demonstrating the utility of learning from human feedback algorithms in high-dimensional state spaces, Warnell et al. (2018) also reported that agents trained using human-provided feedback ultimately learned policies that outperformed that of the human trainers themselves. This result would seem to support the hypothesis that the performance of agents that can learn from human feedback is not capped by the trainer’s expertise to perform the task. However, such performance could be affected by the trainer’s expertise in providing evaluative feedback.

Several extensions to the above algorithms have been proposed in the literature. Notably, several have studied combining human-provided evaluative feedback with existing reward functions (Cederborg et al., 2015; Knox and Stone, 2010, 2012; Arakawa et al., 2018) with the goal of augmenting reinforcement learning. Saunders et al. (2018) look explicitly at situations in which humans block catastrophic actions, and interpret these blocking actions as evaluative feedback when learning in combination with an existing reward function. This method is particularly useful for RL tasks that require safe exploration.

Evaluative feedback is usually communicated through button presses by humans, some other works have sought to infer feedback signals from multi-modal evaluative signals humans naturally emit during social interactions, including gestures (Najar et al., 2020), facial expressions (Broekens, 2007; Arakawa et al., 2018; Cui et al., 2020), electroencephalogram (EEG) based brain waves signals (Xu et al., 2020; Akinola et al., 2020), and implied feedback when humans refrain from giving explicit feedback (Loftin et al., 2014; Joachims et al., 2017). Other body-language and vocalization modalities not aimed at explicit communication, such as tone of voice, subtle head gestures, and hand gestures, could also be modeled and leveraged during training in the future (Cui et al., 2020). Other works have looked at methods by which to elicit more feedback from human trainers (Li et al., 2016b) or to explicitly account for situations in which the human trainer may not be paying attention (Kessler Faulkner et al., 2019).

Each of the algorithms presented above interprets human feedback in slightly different ways, resulting in different policy update rules. Using synthetic feedback, MacGlashan et al. (2017) showed that the convergence of these algorithms depends critically on whether the actual feedback matches the assumed one. Critically, the nature of the feedback could potentially vary across tasks and trainers. Loftin et al. (2016) has shown that instead of providing balanced feedback, human trainers could be more reward-focused (provides explicit rewards for correct actions and ignore incorrect ones), punishment-focused, or even inactive. Factors such as previous experience in training pets and feedback from the agent can affect the trainer’s strategies (Loftin et al., 2016). Additionally, the nature of the feedback can be altered by the instruction given to the trainers. For example, Cederborg et al. (2015) has shown that they can manipulate the meaning of human trainer’s silence by differing the instructions given: The trainers were told that their silence meant positive/negative to the agent. Not surprisingly, the agent needs to adjust their interpretations of silence accordingly to perform well (Cederborg et al., 2015). Therefore, these factors need to be carefully controlled in practice. One potential future research direction is to study methods that explicitly attempt to be robust to many types of feedback or methods that attempt to infer the human feedback type and adapt to that type in real time (Grizou et al., 2014; Loftin et al., 2016; Najar et al., 2020).

In learning from human preferences, the targets to be evaluated are trajectories instead of state-action pairs as in learning from human evaluative feedback. The advantage of evaluating trajectories is that the feedback is typically very sparse, resulting in a drastic reduction in human effort. However, there are two potential concerns in leveraging human preference. First, although the amount of feedback is less, the time or cognitive efforts in watching the two trajectories to make a preference choice could be more. Second, selecting the optimal trajectory length is challenging. Shorter trajectories allow humans to provide feedback of high granularity at the cost of more frequent human interactions. From the human trainer’s perspective, the ideal trajectory length should be subjective, meaning that human trainers could select and adjust their preferred trajectory length during the training process. From the learning agent’s perspective, the ideal length could also be adaptive, meaning that the agent could adaptively adjust the trajectory length to query humans to receive feedback of desired granularity.

Recent work by Bhatia et al. (2020) raised an important issue in preference learning: the human preference could be multi-criteria in nature, i.e., different behaviors are preferred under different criteria. For example, a driving policy $\tau_{1}$ is preferred in terms of comfort, while $\tau_{2}$ is preferred when speed is the only concern. Interestingly, a third policy $\tau_{3}$ which is a linear combination of $\tau_{1}$ and $\tau_{2}$ may be preferred among all three when considering both criteria (Bhatia et al., 2020). The authors propose a novel framework to solve this problem by decomposing the single overall comparison and ask humans to provide preferences along simpler criteria (Bhatia et al., 2020). The authors lay the groundwork from a game-theoretic perspective but many questions are yet to be answered.

Le et al. (2018) has proposed a hierarchical guidance framework that assumes a two-level hierarchy, in which a high-level agent learns to choose a goal $g$ given a state, while low-level agents learn to execute a sub-policy (option) to accomplish the chosen goal (Fig. 4d). Note that an action to terminate the current option needs to be added to the low-level agent’s action space, and this termination action can be demonstrated by a human and learned by the agent. Human trainers were asked to provide three types of feedback: 1) a positive signal if the high-level goal $g_{t}$ and low-level sub-policy $a_{t}$ are both correct; 2) the correct high-level goal if the chosen one is incorrect; 3) the correct low-level action $a^{*}_{t}$ if the high-level goal was chosen correctly but the low-level sub-policy is incorrect. At each level, the learning task becomes a typical imitation learning problem, therefore conventional IL algorithms such as behavioral cloning and DAgger (Ross et al., 2011) can be applied.

Perhaps the most exciting result comes from a hybrid of hierarchical imitation learning and RL. One approach is to train the agent to choose high-level goals via imitation learning, and let the agent learn low-level policies via RL by itself. This approach was shown to be substantially more sample efficient than conventional imitation learning. For example, Andreas et al. (2017) only required humans to provide policy sketches which are high-level symbolic subtask labels. The policy of each subtask is learned by the RL agent on its own and no longer requires human demonstration. A similar approach has been shown to be successful on Atari Montezuma’s Revenge (Le et al., 2018). Another approach is to train both high-level and low-level policies via imitation learning, then fine-tune them using RL later (Gupta et al., 2020).

Compared to simulated learning agents, in physical robot learning safety and sample efficiency are critical issues for IL and RL algorithms. Therefore, incorporating human prior knowledge through hierarchical task structuring to make robot learning tractable has been long studied and implemented. The classic approach is to define and represent the learning task at a more abstract level using human knowledge. As an example, actions can be defined as high-level goals such as “turn 90 degrees clockwise”, meanwhile fine-grained motor commands that accomplish these goals can be handled by low-level controllers. An early survey of previous robotic research on this topic is provided by Kober et al. (2013).

In contrast to providing only high-level goals for simulated agents (Andreas et al., 2017), in robotic tasks humans often need to provide low-level demonstrations due to safety and sample efficiency concerns. The works that leverage this type of demonstration can be roughly classified into segmentation-based or non-segmentation-based approaches depending on whether the task hierarchy is provided by humans.

In segmentation-based approaches, the task hierarchy is not provided, hence the aim is to extract meaningful segments from the low-level demonstration trajectories. For example, in contrast to Andreas et al. (2017), Krishnan et al. (2017) and Henderson et al. (2018a) set up the learning task in the opposite way which attempts to discover high-level options from low-level demonstration data. In this setting, only low-level demonstrations are collected, the options are latent variables of the trainer that can be inferred in a fashion similar to Expectation Maximization (Krishnan et al., 2017). Similarly, other methods aim to learn low-level primitives (Kipf et al., 2019; Sharma et al., 2018), latent conditioned policies (Hausman et al., 2017), goal-conditioned policies (Gupta et al., 2020), or skills (Konidaris et al., 2012; Kroemer et al., 2015) which meaningfully segment the low-level demonstrations. If information about high-level goals is also provided, they can be used to help infer meaningful segmentation boundaries. For example, Codevilla et al. (2018) has successfully combined high-level navigation commands with low-level control signals in a framework named conditional imitation learning for autonomous driving tasks.

In contrast, the task hierarchy can be provided explicitly or implicitly to eliminate the need for task segmentation. Humans can explicitly define the task hierarchy and feed such information to the robots (Mohseni-Kabir et al., 2015). Alternatively, humans can provide demonstrations subtask by subtask, therefore options, subtasks, subprograms, subroutines, or individual skills are learned first in isolation and then combined (Friesen and Rao, 2010).

A fixed, rigid task hierarchy has a poor ability to generalize. Recently, a framework named neural programming (NTP) has been developed that can decompose a demonstrated task into modular and reusable neural programs in a hierarchical manner (Reed and De Freitas, 2015; Li et al., 2016a; Xu et al., 2018a; Fox et al., 2018). The task hierarchy is only provided by humans during the training phase until the agent has learned to do task segmentation on its own. Neural programs are structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism (Fox et al., 2018). They are represented by neural networks that can learn to represent and execute compositional programs from demonstrations (Reed and De Freitas, 2015). The demonstration is still provided as low-level actions, the learning algorithm attempts to learn and reuse primitive network modules from the demonstration. A task manager, which is often a trainable task-agnostic network, decides which subprogram to run next and feeds task specification to the next program. Training this high-level manager requires ground-truth high-level task labels provided by human (Xu et al., 2018a). The low-level policy is represented as a neural program that takes a task specification as its input argument.

The benefit of this kind of hierarchical modular approach comes from the observation that in many robotic tasks there are shared components, and a learned task component (e.g., a particular skill) can often generalize across tasks. In a multitask setting, learned task components can be transferred between tasks so the required human demonstration effort could be drastically reduced (Fox et al., 2019; Xu et al., 2018b).

In some of the above robot learning works, asking humans to provide high-level actions requires additional human effort. However, in physical robot experiments, human annotation is often less costly and risky than demonstrations or teleoperations. If providing extra annotation can reduce the cost of using physical robots, such extra effort is justified and desirable (Fox et al., 2019).

We have seen that humans can either provide low-level action demonstrations, or high-level goals, or both types of guidance together. The choice that is suitable for a particular task domain depends on at least two factors. The first concern is the relative effort in specifying goals vs. providing demonstrations. High-level goals are often clear and easy to be specified in tasks such as navigation (Andreas et al., 2017). On the contrary in tasks like Tetris providing low-level demonstration is easier since high-level goals are not easy to represent and be communicated. The second concern is safety and sample efficiency. Only providing high-level goals requires the agents to learn low-level policies by themselves through trial-and-error, perhaps with many more samples, which is suitable for simulated agents but not for physical robots. Therefore in robotic tasks, low-level action demonstrations are often required.

One way to further reduce human effort in hierarchical imitation learning is to leverage evaluative feedback. Evaluative feedback can be naturally incorporated in the hierarchical imitation learning framework, in which human trainers provide evaluative feedback (yes or no) on either high-level or low-level actions of the learning agent, an approach that has been partially explored by Mohseni-Kabir et al. (2015) and (Le et al., 2018). Moreover, as mentioned earlier, it is natural to extend hierarchical imitation to incorporate human preferences over the outcome of options, instead of asking humans to provide the correct option labels, as done in Pinsler et al. (2018).

Hierarchical imitation learning is naturally related to hierarchical reinforcement learning (Sutton et al., 1999; Dietterich, 2000; Barto and Mahadevan, 2003), which is an active research field with its own exciting progress (Kulkarni et al., 2016; Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018). Another closely related research field here is multi-agent reinforcement learning (Ghavamzadeh et al., 2006) since the multi-agent setting implicitly contains a two-level hierarchy: One at the individual agent’s level and the other at the group level. For a recent survey on this topic, please see Hernandez-Leal et al. (2019). A potential research direction is to leverage human guidance, in the forms of demonstration or feedback, in the settings of hierarchical RL or multi-agent learning systems. As we have seen in Le et al. (2018), a reasonable starting point is to ask humans to demonstrate or evaluate high-level decisions.

Because IfO depends on observations of an expert agent, processing these observations perceptually is extremely important. Previous works have used multiple approaches for this part of the problem. One such approach is to record the expert’s movements using sensors placed directly on the expert agent (Ijspeert et al., 2011). Using these recordings, techniques have been proposed that can allow humanoid or anthropomorphic robots to mimic human motions, e.g., arm-reaching movements (Ijspeert et al., 2002; Bentivegna et al., 2002), biped locomotion (Nakanishi et al., 2004), and human gestures (Calinon and Billard, 2007). Another approach to the perception problem is that of motion capture (Field et al., 2009), which typically uses visual markers on the demonstrator to infer movement. IfO techniques built upon this approach have been used for a variety of tasks, including locomotion, acrobatics, and martial arts (Peng et al., 2018a; Merel et al., 2017; Setapen et al., 2010). The methods discussed above often require costly instrumentation and pre-processing (Holden et al., 2016), and therefore cannot be used in conjunction with more passive resources such as YouTube videos.

Recently, however, convolutional neural networks and advances in visual recognition have provided promising tools to work towards visual imitation where the expert demonstration consists of raw video information (e.g., pixel color values) alone. Even with such tools, the imitating agent is still faced with several challenges: (1) embodiment mismatch, and (2) viewpoint difference. Embodiment mismatch arises when the demonstrating agent has a different embodiment from that of the imitator. For example, the video could be of a human performing a task, but the goal may be to train a robot to do the same. Since humans and robots do not look exactly alike (and may look quite different), the challenge is in how to interpret the visual information such that IfO can be successful. One IfO method developed to address this problem learns a correspondence between the embodiments using autoencoders in a supervised fashion (Gupta et al., 2018). The autoencoder is trained in such a way that the encoded representations are invariant with respect to the embodiment features. Another method learns the correspondence in an unsupervised fashion with a small amount of human supervision (Sermanet et al., 2018). The second IfO perceptual challenge is the viewpoint difference that arises when demonstrations are not recorded in a controlled environment. For instance, the video background may be cluttered, or there may be a mismatch in the point of view present in the demonstration video and that with which the agent sees itself. One IfO approach that attempts to address this issue learns a context translation model to translate an observation by predicting it in the target context (Liu et al., 2018). The translation is learned using data that consists of images of the target context and the source context, and the task is to translate the frame from the source context to that of the target. Another approach uses a classifier to distinguish between the data that comes from different viewpoints and attempts to maximize the domain confusion in an adversarial setting during the training (Stadie et al., 2017). Consequently, the extracted features can be invariant with respect to the viewpoint.

Another main component of IfO is control, i.e., the approach used to learn the imitation policy, typically under the assumption that the agent has access to clean state demonstration data $\{s_{t}\}$. Since the action labels are not available, this is a very challenging problem, and many approaches have been discussed in the literature. Previously, this problem was referred to as trajectory tracking where the goal was to follow a time parameterized reference (Yang and Kim, 1999; Aguiar and Hespanha, 2007). Most of the algorithms developed for trajectory tracking do not involve machine learning at all and require the state features and reference points to be well-defined such as joint angles, velocities, etc. (Yang and Kim, 1999; Caracciolo et al., 1999). Therefore, it is not clear how to directly scale these algorithms to visual imitation (imitating directly from raw pixel data).

With the rise of deep learning, however, many new learning-based algorithms have recently been proposed to tackle the IfO control problem. We organize them here into two general groups: (1) model-based algorithms, and (2) model-free algorithms. Model-based approaches to IfO are characterized by the fact that they learn some type of dynamics model during the imitation process. Most of these algorithms learn an inverse dynamics model which is a mapping from state-transitions $\{(s_{t},s_{t+1})\}$ to actions $\{a_{t}\}$ (Hanna and Stone, 2017). The goal of these algorithms is to retrieve the missing demonstration action labels. To do so, they interact with the environment, collect state action data, and then learn an inverse dynamics model. Applying this learned model on two consecutive demonstrated states would output the missing taken action that had resulted in that state transition. After retrieving the actions, the learning problem can be treated as a conventional imitation learning problem. Recently, many algorithms are developed with this high-level idea (Nair et al., 2017; Torabi et al., 2018a; Pavse et al., 2020; Pathak et al., 2018; Guo et al., 2019; Robertson and Walter, 2020; Jiang et al., 2020; Radosavovic et al., 2020). Some other algorithms use forward dynamics model instead which is a mapping from state-action pairs, $\{(s_{t},a_{t})\}$, to the next states, $\{s_{t+1}\}$. One algorithm of this type is developed by Wu et al. [2020] in which a forward dynamics model is learned which is used to predict the future state of the agent and then future state similarity is used to learn an imitation policy. There is another algorithm (Edwards et al., 2018) that learns forward dynamics model. This algorithm hypothesizes that the state transitions are caused by the actions taken by the agent. The actions are unknown and therefore the algorithm considers a latent (unreal) action space and learns a policy in that latent space that best describes the state transitions. Since the actions generated by this learned policy are not real, next the agent takes a few interactions with the environment to make corrections to the action labels. To be more specific, this algorithm creates an initial hypothesis for the imitation policy by learning a latent policy $\pi(z|s_{t})$ that estimates the probability of latent (unreal) action $z$ given the current state $s_{t}$. Since actual actions are not needed, this process can be done offline without any interaction with the environment. To learn the latent policy, they use a latent forward dynamics model which predicts $s_{t+1}$ and a prior over $z$ given $s_{t}$. Then they use a limited number of environment interactions to learn an action-remapping network that associates the latent actions with their corresponding correct actions. Since most of the process happens offline, the algorithm is efficient with regard to the number of interactions needed.

The other broad category of IfO control approaches is that of model-free algorithms. Model-free techniques attempt to learn the imitation policy without any sort of model-learning step. Within this category, there are two fundamentally different types of algorithms. One is adversarial methods which are inspired by the generative adversarial imitation learning (GAIL) algorithm described in Section 2.2. In GAIL, the goal is to bring the state-action distribution of the imitator close to that of the demonstrator. However, since in IfO the imitator does not have access to the actions, the proposed algorithms attempt to bring the state distribution (Merel et al., 2017; Henderson et al., 2018a), or state transition distribution (Stadie et al., 2017; Torabi et al., 2018b, 2019b, 2019c, 2019a; Zolna et al., 2018; Sun et al., 2019; Yang et al., 2019; Chaudhury et al., 2019) of the imitator close to that of the demonstrator. The overall scheme of these algorithms is as follows. They use a GAN-like architecture in which the imitation policy is interpreted as the generator. The imitation policy is executed in the environment to collect data, $\{(s_{t}^{i},a_{t}^{i})\}$, and either the states or the state transitions are fed into the discriminator, which is trained to differentiate between the data that comes from the imitator and data that comes from the demonstrator. The output value of the discriminator is then used as a reward to update the imitation policy using RL. Another class of model-free approaches developed for IfO control is that utilizes reward engineering. Here, reward engineering means that, based on the expert demonstrations, a manually designed reward function is used to find imitation policies via RL. Importantly, the designed reward functions are not necessarily the ones that the demonstrator used to produce the demonstrations—rather, they are simply estimates inferred from the demonstration data. Most of the algorithms of this type use the negative of the Euclidean distance of the states of the imitator and the demonstrator (or an embedded version of them) as the reward at each time step (Kimura et al., 2018; Sermanet et al., 2018; Dwibedi et al., 2018; Gupta et al., 2018; Liu et al., 2018, 2020). Another approach of this type is developed by Goo and Niekum (2019) in which the algorithm uses a formulation similar to shuffle-and-learn Misra et al. (2016) to train a neural network that learns the order of frames in the demonstration. The network in a supervised fashion gets two observations and outputs a value between zero and one. The closer the value to one, the higher the chance of observations being in the right order. This neural network is then used as a surrogate reward function to train a policy. Aytar et al. (2018) also take a similar approach, learning an embedding function for the video frames based on the demonstration. They use the closeness between the imitator’s embedded states and some checkpoint embedded features as the reward function.

Regarding the perception component of the IfO problem, adversarial training techniques have led to several recent and exciting advances in the computer vision community. One such advance is in the area of pose estimation (Cao et al., 2017; Wang et al., 2019), which enables detection of the position and orientation of the objects in a cluttered video through keypoint detection—such keypoint information may also prove useful in IfO. While there has been a small amount of effort to incorporate these advances in IfO (Peng et al., 2018b), there is still much to investigate.

Another recent advancement in computer vision is in the area of visual domain adaptation (Wang and Deng, 2018), which is concerned with transferring learned knowledge to different visual contexts. For instance, the recent success of CycleGAN (Zhu et al., 2017) suggests that modified adversarial techniques may be applicable to IfO problems that require solutions to embodiment mismatch, though it remains to be seen if such approaches will truly lead to advances in IfO.

Regarding the control component of the IfO problem, very few of the mentioned IfO algorithms discussed have been successfully tested on physical robots, such as Sermanet et al. (2018); Liu et al. (2018). That is, most discuss results only in simulated domains. For instance, while adversarial control methods currently provide state-of-the-art performance for several baseline experimental IfO problems, these methods exhibit high sample complexity and have therefore only been applied to relatively simple simulation tasks. Thus, an open problem in IfO is that of finding ways to adapt these techniques such that they can be used in scenarios for which high sample complexity is prohibitive, i.e., tasks in robotics. Furthermore, there have been few works investigating the combination of IfO with other learning frameworks (Brown et al., 2019; Pavse et al., 2020; Schmeckpeper et al., 2020). There is room to investigate how different types of learning paradigms could be incorporated in IfO to improve the overall task learning performance.

The first learning objective using these datasets could be training an agent to imitate human gaze behaviors, i.e., learning to attend to certain features of a given image. The problem was formalized as a visual saliency prediction problem in computer vision research (Itti et al., 1998). Recently this area has made tremendous progress due to deep learning as large-scale eye-tracking datasets became available for images (Papadopoulos et al., 2014; Li et al., 2014; Xu et al., 2014; Bylinskii et al., 2015b, a; Krafka et al., 2016), videos (Mathe and Sminchisescu, 2014; Wang et al., 2018), and 360-degree videos (Zhang et al., 2018b; Xu et al., 2018c). Visual saliency is a well-developed field in computer vision. We direct interested readers to recent review papers on the topics of saliency evaluation metrics (Bylinskii et al., 2019), saliency model performance analyses (Bylinskii et al., 2016; He et al., 2019) and a closely related field called salient object detection (Borji et al., 2015).

In our context of sequential decision-making tasks, the saliency prediction problem can be formalized as follows:

Given a state $s_{t}$, learn to predict human gaze positions $w_{t}$, i.e., learn $P(w|s)$.

Note that $w_{t}$ could be a set of positions since the human can look at multiple regions of the image. In practice, discrete human gaze positions are converted into a continuous distribution (Bylinskii et al., 2019). So the agent should learn to predict this probability distribution over the given image. This can be done using supervised learning where Kullback-–Leibler divergence can be used as the loss function to calculate the difference between the ground truth distribution $P$ and predicted distribution $Q$ (Bylinskii et al., 2019):

where $i,j$ are pixels indices and $\epsilon$ is a small regularization constant and determines how much zero-valued predictions are penalized. Recent works have trained convolutional neural networks to accomplish this learning task (Li et al., 2018; Zhang et al., 2020b; Palazzi et al., 2018; Deng et al., 2019; Chen et al., 2020). Example gaze prediction results in the format of saliency maps can be seen in Fig. 11. A notable challenge here is egocentric gaze prediction in which the spatial distribution of the gaze is highly biased towards the image center, a problem further addressed by Palazzi et al. (2018); Tavakoli et al. (2019).

In computer vision, traditional saliency prediction does not involve active tasks nor human decisions. The humans look at static images or videos in a free-viewing manner without performing any particular task and only the eye movements are recorded and modeled. Meanwhile, the aforementioned datasets all require humans to perform a task while collecting their gaze and action data. From a decision-learning perspective, human attention may provide additional information about their decisions, therefore it is intuitive to leverage learned attention models to guide the learning process of human decisions. The learning problem can be formalized as follows:

Given a state $s_{t}$ and human gaze positions $w_{t}$, learn to predict human action $a_{t}$, i.e., learn $P(a|s,w)$.

Intuitively, knowing where humans would look provides useful information on what action they will take. To incorporate human attention into action learning, there are at least three common methods: as an additional channel of information, as a mask on the input to filter out unimportant information, or as a secondary optimization objective. For example, in training a neural network, the above methods correspond to concatenating a gaze map with the input image, masking the input image with the gaze map, and adding gaze prediction as an auxiliary loss term in the objective function, respectively (Zhang et al., 2020a). The most popular way is to treat the predicted gaze distribution of an image as a filter or a mask. This mask can be applied to the image to generate a representation of the image that highlights the attended visual features.

Experimental results have shown that including gaze information leads to higher accuracy in recognizing or predicting human actions, in reaching (Ravichandar et al., 2018), human-to-human interaction (Zuo et al., 2018), driving (Xia et al., 2018; Liu et al., 2019; Chen et al., 2019; Xia et al., 2020), meal preparation (Li et al., 2018; Shen et al., 2018; Sudhakaran et al., 2019; Huang et al., 2020), and video game playing (Zhang et al., 2018a, 2020b).

Once the agent has learned both the attention and decision models from human data, it can perform the task on its own. It has been shown that incorporating a learned gaze model into imitation learning agents leads to a large performance increase, comparing to agents without attention information (Zhang et al., 2020b; Saran et al., 2020; Chen et al., 2020). For real-world tasks like autonomous driving, it is reasonable to expect a similar improvement when incorporating human attention models. Due to physical constraints and safety reasons, this is yet to be explored but preliminary tests in simulated environments are possible.

In general, human gaze is a good indicator of the underlying decision-making mechanism, it bridges perception and decision-making by indicating the current behavioral target. The gaze data can be collected in parallel with actions. One concern with this approach is the hardware and software required to collect human gaze data. Recent progress in computer vision has improved eye tracker accuracy and portability by a significant margin. Appearance-based algorithms using convolutional neural networks have been shown to have better tracking accuracy and are more robust to visual appearance variations (Zhang et al., 2015; Wood et al., 2015; Krafka et al., 2016; Shrivastava et al., 2017; Zhang et al., 2017; Park et al., 2018), compared to more traditional approaches like hand-crafted feature-based or model-based algorithms. Advanced tracking software can estimate gaze in real-time from head poses and appearance without specialized hardware on low-cost devices such as webcams (Papoutsaki et al., 2016) and mobile tablets and phones (Huang et al., 2017; Krafka et al., 2016).

Gaze data can be collected in parallel when providing other types of feedback, and potentially be combined with previously introduced learning methods. Saran et al. (2020) has shown that incorporating gaze information into imitation from observation (IfO) and inverse reinforcement learning can lead to a large performance increase in Atari games. Since attention is an intermediate mechanism between perception and action, it becomes very useful when action information is missing in the case of IfO. In learning evaluative feedback and preference, gaze data might reveal more information to the learning agent to explain why the human gives a particular evaluation. Attention learning is closely related to hierarchical imitation, since gaze is a good indicator of the current high-level behavioral goal which might help an imitator to infer this goal. However, the problem of inferring behavioral goals from human attention needs to be solved first.

In general, researchers collect their own human guidance data. However, this type of data is often expensive to collect. An effort that could greatly facilitate research in this field is to create publicly available benchmark datasets. Collecting and reusing such datasets may be difficult for some interactive learning methods, in which the guidance (such as evaluative feedback) depends on the changing policy as it is being learned. But, for other approaches, data can be collected in advance and shared. In Table 1 we provide links to existing datasets that are publicly available. Another concern is reproducibility in RL (Henderson et al., 2018b). When collecting human guidance data, factors such as individual expertise, experimental setup, data collection tools, dataset size, and experimenter bias could introduce large variances in the final performance. Therefore, evaluating algorithms using a standard dataset could save effort and assure a fair comparison between algorithms.

Leveraging human guidance to train an agent naturally follows a teacher-student paradigm. Much effort has been spent on making the student more intelligent. However, understanding the behavior of human teachers is equally important. Thomaz and Breazeal (2008) pioneered the effort in understanding human behavior in teaching learning agents. As RL agents become more powerful and attempt to solve more complex tasks, the human teachers’ guiding behaviors could become more complicated and require further study.

Studying this aspect of human behavior, especially the limitations of human teachers, allows one to design a teaching environment that is more effective and produces more useful guidance data. Amir et al. (2016) studied human attention limits while monitoring the learning process of an agent and proposed an algorithm for the human and the agent to jointly identify states where feedback is most needed to reduce human monitoring cost. Ho et al. (2016) showed the differences in behavior when a human trainer is intentionally teaching (showing) versus merely doing the task. They found that humans modify their policies to reveal the goal to the agent when in the showing mode but not in doing mode. They further showed that imitation learning algorithms can benefit substantially more from the data collected in the showing mode (Ho et al., 2016). An important factor to consider is the human trainer’s knowledge of the task. Laskey et al. (2016) have shown that using a hierarchy of human supervisors with different expertise levels can substantially reduce the burden on the experts.

Understanding the variations in human guidance signals allows algorithms to learn more effectively. We have already seen the debate on how to interpret human evaluative feedback in complex tasks. A helpful way to resolve this debate is to conduct human studies with diverse subject pools to investigate whether real-life human feedback satisfies their algorithmic assumptions and what factors affect the human feedback strategy (Cederborg et al., 2015; Loftin et al., 2016; MacGlashan et al., 2017).

The best learning results often come from an interactive teaching and learning process in a teacher-student paradigm which involves active instruction by the human and active learning by the agent. Two factors justify an interactive learning paradigm in our context: (1) We only have a partial understanding of human guidance behaviors; and (2) the nature of human guidance may vary during training according to the behaviors of the learning agents. As shown by Cooperative IRL (Hadfield-Menell et al., 2016), an iterative and interactive learning process can greatly enhance learning. Therefore the same idea may benefit the process of learning from human guidance as well.

For evaluative feedback, we have seen a debate on how we interpret human feedback. We have also seen methods that are robust to many types of feedback or that can infer and adapt to different human feedback types (Grizou et al., 2014; Loftin et al., 2016; Najar et al., 2020). However, perhaps an alternative is to allow the agent to actively query humans for a certain type of feedback that best informs the agent. Learning from human preference is naturally an interactive process when the learning agents actively query for human preferences as we have discussed. Additionally, the aforementioned challenge of selecting optimal trajectory length for query likely requires two-way communication between the human and the agent. In hierarchical imitation, imitation from observation, and attention learning we rarely see examples of interactive learning (Mohseni-Kabir et al., 2015), since the human data is often collected off-line (see Table 1) before training as in standard imitation learning. From DAgger (Ross et al., 2011) it is clear that interactive training can also benefit imitation learning, however, making the three learning paradigms above interactive remains to be explored.

The learning frameworks discussed in this paper are often inspired by real-life biological learning scenarios that correspond to different learning stages and strategies in lifelong learning. Imitation and reinforcement learning correspond to learning completely by imitating others and learning completely through self-generated experience, where the former may be used more often in the early stages of learning and the latter could be more useful in the late stages. The other learning strategies discussed are often mixed with these two to allow an agent to utilize signals from all possible sources. For example, it is widely known that children learn largely by imitation and observation (Bandura et al., 1961) at their early stage of learning. Then the children gradually learn to develop joint attention with adults through gaze following (Goswami, 2008). Later children begin to adjust their behaviors based on the evaluative feedback and preference received when interacting with other people. Once they developed the ability to reason abstractly about task structure, hierarchical imitation becomes feasible. At the same time, learning through trial and error from reinforcement is always one of the most common types of learning (Skinner, 1938). The human’s ability to learn from all types of resources continue to develop through a lifetime. We have compared these learning strategies within an imitation and reinforcement learning framework. Under this framework, it is possible to develop a unified learning paradigm that accepts multiple types of human guidance. We start to notice efforts towards this goal (Abel et al., 2017; Waytowich et al., 2018; Goecks et al., 2019; Woodward et al., 2020; Najar et al., 2020; Bıyık et al., 2020).

In conclusion, the goal of this survey is to serve as a high-level overview of five recent learning frameworks that leverage human guidance to solve sequential decision-making tasks, especially in the context of deep reinforcement learning. We compare and contrast these frameworks by reviewing the motivation, assumption, and implementation of each framework. We hope this will allow researchers in the related areas to see the connections between the works being surveyed, and inspire more research to be done in this field.