Physical Interaction as Communication:
Learning Robot Objectives Online from Human Corrections

Physical interaction is a natural means for collaboration and communication between humans and robots. From compliant designs to reliable prediction algorithms, recent advances in robotics have enabled humans and robots to work in close physical proximity. Despite this progress, seamless physical interaction—where robots are as responsive, intelligent, and fluid as their human counterparts—remains an open problem.

One key challenge is determining how robots should respond to direct physical contact. Fast and safe responses to external forces are generally necessary, and have been studied extensively within the field of physical human-robot interaction (pHRI). A traditional controls approach is to treat the human’s interaction force as a perturbation to be rejected or ignored. Here the robot assumes that it is an expert agent and follows its own predefined trajectory regardless of the human’s actions (De Santis et al., 2008). Alternatively, the robot can treat the human as the expert, so that the human guides the passive robot throughout their preferred trajectory. Whenever the robot detects an interaction it stops moving and becomes transparent, enabling the human to easily adjust the robot’s state (Jarrassé et al., 2012). Impedance control—the most prevalent paradigm for pHRI (Haddadin and Croft, 2016; Hogan, 1985)—combines aspects of the previous two control strategies. Here the robot tracks a predefined trajectory, but when the human interacts the robot complies with the human’s applied force. Under this approach the human can intuitively alter the robot’s state while also receiving force feedback from the robot.

In each of these different response strategies for pHRI the robot returns to its pre-planned trajectory as soon as the human stops interacting. In other words, the robot remains confident that its original trajectory is the correct way to complete the task. Since this robot trajectory is optimal with respect to some underlying objective function, these response paradigms effectively maintain a fixed objective function during pHRI. Hence, the human’s interactions do not change the robot’s understanding of the task; instead, external forces are simply disturbances which should be reacted to, rather than information which should be reasoned about.

In this work we assert that physical human interactions are often intentional, and occur because the robot is doing something that the human believes is incorrect. The fact that the human is physically intervening to fix the robot’s behavior implies that the robot’s trajectory—and therefore the underlying objective function used to produce this trajectory—is wrong. Under our framework we consider the forces that the human applies as observations about the true objective function that the robot should be optimizing, which is known to the human but not by the robot. Accordingly, human interactions should no longer be thought of as only disturbances that perturb the robot from its pre-planned trajectory, but rather as corrections that teach the robot about the desired behavior during the task.

This insight enables us to formalize the robot’s response to pHRI as an instance of a partially observable dynamical system, where the robot is unsure of its true objective function, and human interactions provide information about that objective. Solving this system defines the optimal way for the robot to respond to pHRI. We derive an approximation of the solution to this system that works in real-time for continuous state and action spaces, enabling robot arms to react to pHRI online and adjust how they complete the current task. Due to the necessity of fast and reactive schemes, we also derive an online gradient-descent solution that adapts inverse reinforcement learning approaches to the pHRI domain. We find that this solution works well in some settings, while in others user corrections are noisy and result in unintended learning. We alleviate this problem by introducing a restriction to our update rule focused on extracting only what the person intends to correct, rather than assuming that every aspect of their correction is intentional. Finally, we compare our approximations to a full solution, and experimentally test our proposed learning method in user studies with a robotic manipulator.

We make the following contributions¹¹1Note that parts of this work have been published at the Conference on Robotic Learning (Bajcsy et al., 2017) and the Conference on Human-Robot Interaction (Bajcsy et al., 2018).:

Overall, this work demonstrates how we can leverage the implicit communication which is present during physical interactions. Learning from implicit human communication applies not only to pHRI, but conceivably also to other kinds of actions that people take.

In this work, we enable robots to leverage physical interaction with a human during task execution to learn a human’s objective function. We also account for imperfections in the way that people physically interact to correct robot behavior. Prior work has separately addressed (a) control strategies for reacting to pHRI without learning the human’s objective and (b) learning the human’s objective offline from kinesthetic demonstrations. An exception is work on shared autonomy, which learns the human’s objective in real-time, but only when that objective is parameterized by the human’s goal position. Finally, we discuss related work on algorithmic teaching, which describes how humans can optimally teach robots as well as how humans practically teach robots.

More relevant here are works on shared control, where the robot has an objective function, and uses that objective function to select optimal control feedback during pHRI (Jarrassé et al., 2012; Medina et al., 2015; Losey et al., 2018). In Li et al. (2016) the authors formulate pHRI with game theory. The robot has an objective function which depends on the error from a pre-defined trajectory, the human’s effort, and the robot’s effort. During the task the robot learns the relative weights of these terms from human interactions, resulting in a shared controller that becomes less stiff when the human exerts more force. Rather than only learning the correct robot stiffness—as in Li et al. (2016)—our work more generally learns the correct robot behavior. We note that each of these control methods (Hogan, 1985; Haddadin et al., 2008; Jarrassé et al., 2012; Medina et al., 2015; Li et al., 2016; Losey et al., 2018) enables the robot to safely respond to human interactions in real-time. However, once the human stops interacting, the robot resumes performing its task in the same way as it had planned before human interactions.

Most relevant to our research are IRL approaches that learn from corrections to the robot’s trajectory rather than complete demonstrations (Jain et al., 2015; Karlsson et al., 2017; Ratliff et al., 2006). Within these works, the human corrects some aspect of the demonstrated trajectory during the current iteration, and the robot improves its trajectory the next time it performs the task. By contrast, we use human interactions to update the robot’s behavior during the current task. Our solution for real-time learning is analogous to online Maximum Margin Planning (Ratliff et al., 2006) or coactive learning (Jain et al., 2015; Shivaswamy and Joachims, 2015), but we derive this solution as an approximately optimal response to pHRI. Moreover, we also show how this learning method can be adjusted to accommodate unintentional human corrections.

As we move towards online learning, we also point out research where the robot learns a discrete set of candidate reward functions offline, and then changes between these options based on the human’s real-time physical corrections (Yin et al., 2019). We view this work as a simplified instance of our approach, where the robot has sufficient domain knowledge to limit the continuous space of rewards to a few discrete choices.

Although not part of shared autonomy, we also point out research where the robot’s trajectory changes online due to physical human interactions. In some works—such as Mainprice and Berenson (2013); Sisbot et al. (2007)—the robot alters its trajectory to avoid physical human interaction. More related to our approach are works where the robot embraces physical corrections to adapt its behavior. For example, in Losey and O’Malley (2018); Khoramshahi et al. (2018); Khoramshahi and Billard (2019); Losey and O’Malley (2020) the robot maintains a parameterized desired trajectory or dynamical system, and updates the parameters in real-time to minimize the error between the resultant trajectory and the human’s corrections. These works directly update the robot’s desired trajectory based on corrections; by contrast, we learn a reward function from human corrections, which can—in turn—be used to generate dynamical systems or desired trajectories. Learning a reward function is advantageous here because it enables the robot to generalize what it has learned within the task, e.g., because the human has corrected the robot closer to one table, the robot will move closer to a second table as well.

Consider a robot performing a task autonomously and in close proximity to a human end-user. The human observes this robot and can physically interact with the robot to alter its behavior. Returning to our running example from Fig. 1, imagine a robotic manipulator that is carrying a coffee mug from the top of a cabinet down to a table while the human sits nearby. Importantly, the robot is either not doing this task correctly (e.g., the robot is carrying the cup at such an angle that coffee will spill) or the robot is not doing the task according to the human’s personal preferences (e.g., the robot is carrying the coffee too far above the table). In both of these cases the human is incentivized to physically interact with the robot and correct its behavior: but how should the robot respond? Here we formalize pHRI as a dynamical system where the robot does not know the correct objective function that the human wants it to optimize and the human’s interactions are informative about this objective. Importantly, this formalism defines what it means for a robot to respond in the right or optimal way to physical human interactions. Furthermore, certain strategies for responding to pHRI can be justified as approximate solutions to this formalism.

With this formalism the robot should take actions $u_{r}$ to maximize the reward in Equation (1) across every timestep. This is challenging, however, because the robot does not know the true objective parameters $\theta$: only the human knows $\theta$. Different end-users have different objectives, which can change from task-to-task and even day-to-day. We thus think of $\theta$ as a hidden part of the state known only by the human. If the robot did know $\theta$, then pHRI would reduce to an instance of a Markov decision process (MDP), where the states are $x$, the actions are $u_{r}$, the reward is (1), and the robot understands what it means to complete its task optimally. But since the actual robot is uncertain about $\theta$, we must reason over this uncertainty during pHRI.

Solving this POMDP yields the robot’s optimal response to pHRI during the task³³3The most general formulation for pHRI is that of a cooperative inverse reinforcement learning (CIRL) game (Hadfield-Menell et al., 2016), which, when solved, yields the optimal human and robot policies.. We point out that this POMDP is atypical, however, because the observations $u_{h}$ additionally affect the robot’s reward $r$, similar to Javdani et al. (2018), and alter the robot’s state $x$ via the transition distribution $T$. Because the human’s actions can change both the state and reward, solving this POMDP suggests that the robot should anticipate future human actions, and choose control inputs $u_{r}$ that account for the predicted human inputs $u_{h}$, similar to Hoffman and Breazeal (2007).

Although we have demonstrated that pHRI is an instance of a POMDP, solving POMDPs exactly is at best computationally expensive and at worst intractable (Kaelbling et al., 1998). POMDP solvers have made significant progress (Silver and Veness, 2010; Somani et al., 2013); however, it still remains difficult to compute online solutions for continuous state, action, and observation spaces. For instance, when evaluated on a toy problem ($S=\mathbb{R}^{4}$, $O=\mathbb{R}^{8}$), recent developments do not obtain exact solutions within one second (Sunberg and Kochenderfer, 2017). The lack of efficient POMDP solvers for large, continuous state, action, and observation spaces is particularly challenging here since (a) the dimension of our state space $S$ is twice the number of robot DoF, $2n$, plus the number of task-related features, $N$, and (b) we are interested in real-time solutions that enable the robot to learn and act while the human is interacting (i.e. we need millisecond-to-second solutions). Accordingly, in this section we introduce three approximations to our pHRI formalism that enable online solutions. First, we separate finding the optimal robot policy from estimating the human’s objective. Next, we simplify the observation model and use a maximum a posteriori (MAP) estimate of $\theta$ as opposed to the full belief over $\theta$. Finally, when finding the optimal robot policy and estimating $\theta$, we move from policies to trajectories. These approximations show how our solution is derived from the complete POMDP formalism outlined in the last section, but now enable the robot to learn and react in real-time with continuous state, action, and belief spaces.

Intuitively, under this QMDP the robot is always exploiting the information it currently has, and never actively tries to explore for new information. A robot using the policy from Equation (3) does not anticipate any human actions $u_{h}$, and so the robot solves for its optimal policy as if it were completing the task in isolation. Recall that we previously pointed out that physical human interactions can influence the robot’s state. In practice, however, we do not necessarily want to account for these actions when planning—the robot should not rely on the human to move the robot. Due to our QMPD approximation the robot never relies on the human for guidance: but when the human does interact, the robot leverages $u_{h}$ to learn about $\theta$ in Equation (4). In summary, the robot only considers $u_{h}$ for its information value.

At every timestep $t$, we first replan a trajectory $\xi=x^{0:T}\in\Xi$ which optimizes the task-related reward from Equation (1) over the $T$-step planning horizon. If our features $\phi$ only depend on the state $x$, then the cumulative task-related reward becomes:

Here $\Phi(\xi)$ is the total feature count along trajectory $\xi$. Using the cumulative reward function in Equation (5), the robot finds the optimal trajectory $\xi_{r}^{t}$ from its current estimate $\hat{\theta}^{t}$:

We can solve Equation (6) for the optimal trajectory using trajectory optimization tools (Schulman et al., 2014; Karaman and Frazzoli, 2011). Whenever $\hat{\theta}$ is updated from pHRI during task execution, the robot’s trajectory will be replanned using that new estimate to match the the learned objective.

To track the robot’s planned trajectory we leverage impedance control. Impedance control—as originally proposed by Hogan (1985)—is the most popular controller for pHRI (Haddadin and Croft, 2016), and ensures that the robot responds compliantly to human corrections (De Luca et al., 2006). Let $x^{t}=(q^{t},\dot{q}^{t})$, where $q^{t}$ is the robot’s current configuration, and $q^{t}_{r}\in\xi^{t}_{r}$ is the desired configuration at timestep t. After feedback linearization (Spong et al., 2006), the equation of motion of a robot arm under impedance control becomes:

Here $M_{r}$, $B_{r}$, and $K_{r}$ are the desired inertia, damping, and stiffness rendered by the robot. These parameters determine what impedance the human perceives: for instance, lower $K_{r}$ makes the robot appear more compliant. In our experiments, we implement a simplified impedance controller without feedback linearization:

This control input drives the robot towards its desired state $x^{t}\in\xi_{r}^{t}$, and evaluating Equation (8) over all states yields the robot’s policy. To summarize, we first solve the trajectory optimization problem from Equation (6) to get the current robot trajectory $\xi_{r}^{t}$, and then compliantly track that trajectory using Equation (8). Notice that if the robot never updates $\hat{\theta}$ then $\xi_{r}^{t}=\xi_{r}^{t-1}$, and this approach reduces to using impedance control to track an unchanging robot trajectory.

Instead of attempting to directly relate $u_{h}$ to $\theta$, as in our original observation model, we propose an intermediate step: interpret each human action $u_{h}$ via an intended trajectory, $\xi_{h}$, which the human would prefer for the robot to execute. We leverage trajectory deformations (Losey and O’Malley, 2018) to get the intended trajectory $\xi_{h}$ from the robots planned trajectory $\xi_{r}$ and the humans physical interaction $u_{h}$. Following Losey and O’Malley (2018), we propagate the human’s interaction force along the robot’s trajectory:

where $\mu>0$ scales the magnitude of the deformation. The symmetric positive definite matrix $A$ defines a norm on the Hilbert space of trajectories and dictates the shape of the deformation Dragan et al. (2015). The input vector is $U_{h}=u_{h}$ at the current time, and $U_{h}=0$ at all other times. During experiments we use the velocity norm for $A$ (Dragan et al., 2015), but other options are possible.

Our deformed trajectory minimizes the distance from the previous trajectory while keeping the end-points the same and moving the corrected point to its new configuration (Dragan et al., 2015). Whereas using the Euclidean norm to measure distance would return the same trajectory as before with the current waypoint teleported to where the user corrected it, using a band-diagonal norm $A$ (e.g., the velocity norm) serves to couple each waypoint along the trajectory to the one before it and the one after it. This formalizes the effect proposed by elastic strips by Brock and Khatib (2002) and elastic bands by Quinlan and Khatib (1993).

Now rather than evaluating the $Q$-value of $u_{h}+u_{r}$ given $\theta$, like we did in Equation (2), we can compare the human’s intended trajectory $\xi_{h}$ to the robot’s original trajectory $\xi_{r}$ and relate these differences to $\theta$. We assume that the human provides a intended trajectory $\xi_{h}$ that approximately maximizes their cumulative task-related reward from Equation (5) while remaining close to $\xi_{r}$:

Moving forward we treat $P(\xi_{h}\mid\xi_{r};\theta)$ as our observation model. Note that this observation model is analogous to Equation (2) but in trajectory space. In other words, Equation (10) yields a distribution over intended trajectories given $\theta$ and the current robot trajectory. Here the correspondence between the human’s effort $\|u_{h}\|^{2}$ and the change in trajectories $\|\xi_{h}-\xi_{r}\|^{2}$ stems from the deformation in Equation (9). In conclusion, we can leverage our simplified observation model (10) to tractably reason about the meaning behind the human’s physical interaction.

So far we have determined how to choose the robot’s actions given $\hat{\theta}$, the current MAP estimate of the human’s objective. We have also derived a tractable observation model. Next, we apply this observation model to update $\hat{\theta}$ based on human interactions. By using online gradient descent we arrive at an update rule for $\hat{\theta}$ which adjusts the weights of all the features based on a single human correction. We refer to this method as all-at-once learning. We also relate all-at-once learning to prior works on online Maximium Margin Planning (MMP) and Coactive Learning.

Substituting our normalized observation model from Equation (12) and the prior from Equation (13) back into Equation (5), the MAP estimate $\hat{\theta}^{t+1}$ is the solution to:

In Equation (14) the $\lambda\|\xi_{h}-\xi_{r}\|^{2}$ terms have dropped out because this penalty for human effort does not explicitly depend on $\theta$. Intuitively, our estimation problem (14) states that we are searching for the objective $\theta$ that maximally separates the reward associated with $\xi_{h}$ and $\xi_{r}$, while also regulating the size of the change in $\theta$.

We solve Equation (14) by taking the gradient with respect to $\theta$ and then setting the result equal to zero. Substituting in our cumulative reward function from Equation (5), we obtain the all-at-once update rule:

Given the current MAP estimate $\hat{\theta}^{t}$, the robot’s trajectory $\xi_{r}^{t}$, and the human’s intended trajectory $\xi_{h}^{t}$, we determine an approximate MAP estimate at timestep $t+1$ by comparing the feature counts. Note that the update rule in (5) is actually the online gradient descent algorithm (Bottou, 1998) applied to our normalized observation model (12).

Interestingly, our all-at-once update rule is a special case of the update rules from two related IRL works. Equation (5) is the same as the Preference Perceptron for coactive learning—introduced in Shivaswamy and Joachims (2015) and applied for manipulation tasks by Jain et al. (2015)—if $\xi_{h}$ was the robot’s original trajectory $\xi_{r}$ with a single corrected waypoint. Similarly, Equation (5) is analogous to online Maximum Margin Planning without the loss function if the correction $\xi_{h}$ was treated as a new demonstration (Ratliff et al., 2006). These findings also align with work from Choi and Kim (2011), who show that other IRL methods can be interpreted as a MAP estimate. What is unique in our work is that we demonstrate how the online gradient-descent update rule in Equation (5) results from a POMDP with hidden state $\theta$ where physical human interactions are interpreted as intended trajectories.

We derived an update rule to learn the human’s objective from their physical interactions with the robot. This all-at-once approach changes the weight of all the features that the human adjusts during their correction. In practice, however, the human’s interactions (and their intended trajectory) may result in unintended corrections which mistakenly alter features the human meant to leave untouched. For example, when the human’s action intentionally causes $\xi_{h}$ to move closer to the table, the same correction may accidentally also change the orientation of the coffee. In order to address unintended corrections, we here assume that the human’s intended trajectory $\xi_{h}$ should change only a single feature. We explain how to determine which feature the human is trying to change, and then modify the update rule from Equation (5) to obtain one-at-a-time learning.

Without loss of generality, assume the human is trying to change the $i$-th entry of the robot’s MAP estimate $\hat{\theta}$ during the current timestep $t$. The ideal human correction of $\xi_{r}^{t}$ should accordingly change the feature count in the direction:

Recall that $\xi_{r}^{t}$ is optimal with respect to the current estimate $\hat{\theta}^{t}$, and so changing $\hat{\theta}^{t}$ will alter $\Phi(\xi_{r}^{t})$. Put another way, if the human is an optimal corrector, and their interaction was meant to alter just the weight on the $i$-th feature, then we would expect them to correct the current robot trajectory $\xi_{r}^{t}$ such that they produce a feature difference $\Delta\Phi^{t}$ exactly in the direction of the vector $J_{i}$ from Equation (16).

Because the human is imperfect, they will not exactly match Equation (16). Instead, we model the human as making corrections $\Delta\Phi^{t}$ in the direction of $J_{i}$. This yields an observation model from which the robot can find the likelihood of observing a specific feature difference $\Delta\Phi^{t}$ given that the human is attempting to update the $i$-th feature:

Recalling that the robot observes the feature difference $\Delta\Phi^{t}=\Phi(\xi_{h}^{t})-\Phi(\xi_{r}^{t})$, then we estimate which feature the human most likely wants to change using:

Once the robot solves for the most likely feature the human wants to change, $i^{*}$, it can now find the human’s intended feature difference $\Delta\Phi^{t}_{h}$. Recall that, if the human wanted to only update feature $i^{*}$, their intended feature difference would ideally be in the direction $J_{i^{*}}$. Thus, we choose $\Delta\Phi^{t}_{h}\propto J_{i^{*}}$ as our intended feature difference.

Given this assumption, computing the intended feature difference $\Delta\Phi^{t}_{h}\propto J_{i^{*}}$ reduces to projecting the observed feature difference $\Delta\Phi^{t}$ induced by the human’s action $u_{h}$ onto the $i^{*}$-th axis:

This fulfills our original requirement for the intended feature difference $\Delta\Phi^{t}_{h}$ to only have one non-zero entry. Moreover, once we substitute our simplification of $J_{i}$ back into our feature estimation problem (6), we get a simple yet intuitive heuristic for finding $i^{*}$: only the feature which the user has changed the most during their correction should be updated. Our one-at-a-time update rule is therefore similar to the gradient update from Equation (5), but with a single feature weight update using Equation (19):

Instead of updating the estimated weights associated with all the features like in Equation (5), we now only update the MAP estimate for the feature which has the largest change in feature count. Overall, isolating a single feature at every timestep is meant to mitigate the effects of unintended learning from noisy physical interactions⁵⁵5We note that all the features are normalized to have the same sensitivity..

Before introducing all-at-once and one-at-a-time learning, we showed how approximate solutions to pHRI involve (a) safely tracking the optimal trajectory and (b) updating the MAP estimate based on human interactions. Now that we have derived update rules for $\hat{\theta}$, we will circle back and present our algorithm for learning from pHRI. We also include practical considerations for implementation.

To compare our real-time learning approach with optimal offline solutions and current online baselines, as well as to test both all-at-once and one-at-a-time learning, we conduct human-robot interaction simulations in a controlled environment. Here the robot is performing a pick-and-place task: the robot is carrying a cup of coffee for the simulated human. The simulated human physically interacts with the robot to correct its behavior.

We expect the full QMDP solution to outperform our Learning approximation. From Fig. 4, we observe that the robot learns $\theta$ faster when using the QMDP, and that the robot completes the task with less regret. Both QMDP and Learning outperform Impedance, where the robot does not learn from pHRI. We note that here the simulated human behaves differently than our observation model (2): rather than maximizing their $Q$-value, the human is guiding the robot along their desired trajectory. When the simulated human does follow our observation model, we obtain very similar results: the normalized regret becomes $0.55$ for QMDP and $0.62$ for Learning. To ensure that the learning rate is consistent between the QMDP and Learning methods, we selected $\alpha$ such that $\hat{\theta}^{1}$ equalled $b^{1}(\theta=1)$ when the simulated human followed our observation model (2). From these simulations we conclude that the Learning approximation for online performance is worse than the full QMDP solution, but the difference between these methods is negligible when compared to Impedance.

In Figs. 5 and 6 we show the robot’s trajectory after $N$ human corrections. Notice that Deformations result in local changes which aggregate over time, while—when we learn from these deformations—Learning replans the entire trajectory. Our findings are summarized in Fig 7: it takes fewer corrections to track the human’s desired trajectory with Learning, and the human also expends more effort with Learning. To make the comparison consistent, here we used the same propagation method from (9) to get the Deformations and the intended trajectory for Learning. Based on our results, we conclude that Learning leads to more efficient online performance than Deformations alone, and, in particular, Learning requires less human effort to complete the task correctly.

We utilize two different simulated humans: (a) an optimal human, who exactly guides the robot towards their desired trajectory, and (b) a noisy human, who imperfectly corrects the robot’s trajectory. Like in our previous simulations, the human intervenes to correct the robot when the robot’s error with respect to their desired trajectory exceeds an acceptable margin of error: let us now refer to this as the optimal human. By contrast, the noisy human takes actions sampled from a Gaussian distribution which is centered at the optimal human’s action. This distribution is biased in the direction of the human such that the noisy human tends to accidentally pull the robot closer to their body when correcting the table feature. Due to this noise and bias, the noisy human may unintentionally correct the human feature.

Our final simulation compares All-at-Once and One-at-a-Time learning for optimal and noisy humans. The results for an optimal human are shown in Figs. 8 and 9, while the results for the noisy human are depicted in Figs. 10 and 11. We find that the performance of All-at-Once and One-at-a-Time are identical when the human acts optimally: the robot accurately learns the importance of table, and does not change the weight of human. When the person acts noisily, however, One-at-a-Time learning causes better performance. More specifically, the noisy user corrected the All-at-Once robot during an average of $5.24$ timesteps, but only corrected the One-at-a-Time robot $3.56$ timesteps. Inspecting Fig. 11, we observe that the noisy human unintentionally taught the human feature at the beginning of the task, and had to exert additional effort undoing this mistake on All-at-Once robots. We conclude that there is a benefit to One-at-a-Time learning when the human behaves noisily, since updating only one feature per timestep mitigates accidental learning.

To evaluate the benefits of using physical interaction to communicate we conducted two user studies with a 7-DoF robotic arm (JACO2, Kinova). In the first study, we tested whether learning from pHRI is useful when humans interact, and compared our online learning approach to a state-of-the-art response that treated interactions as disturbances (Learning vs. Impedance). In the second study, we tested how the robot should learn from end-users, and compared one-at-a-time learning to all-at-once learning (One-at-a-Time vs. All-at-Once). During both studies the participants and the robot worked in close physical proximity. In all experimental tasks, the robot began with the wrong objective function, and participants were instructed to physically interact with the robot to correct its behavior⁶⁶6For video footage of the experiment, see: https://youtu.be/I2YHT3giwcY.

In this work we recognize that when humans physically interact with and correct a robot’s behavior their corrections become a source of information. This insight enables us to formulate pHRI as a partially observable dynamical system: the robot is unsure of its true objective function, and human interactions become observations about that latent objective. Solving this dynamical system results in robots that respond to pHRI in the optimal way. These robots update their understanding of the task after each human interaction, and then change how they complete the rest of the current task based on this new understanding.

Interestingly, this derivation enables us to interpret other state-of-the-art responses to pHRI as simplifications of our approximation. For example, if the robot never updates its estimate of the correct objective function (i.e., the robot never learns from pHRI), then our online approximation reduces to impedance control. Alternatively, if we treat the intended trajectory induced by the human’s correction as the robot’s trajectory (but do not update the robot’s objective), then our approximation reduces to deforming the desired trajectory (Losey and O’Malley, 2018). We compared our online approximation to both of these simplifications—impedance control and deformations—as well as to a more complete QMPD solution. During offline simulations we found that the performance loss between our learning method and the QMPD policy was negligible, but our method outperformed impedance control and trajectory deformations. During user studies with a 7-DoF robot, our learning approach resulted in decreased interaction time, effort, and cumulative trajectory cost when compared to an impedance controller. We also found that users believed the learning robot better understood their preferences, resulted in less interaction effort, and was more a collaborative partner than the impedance robot.

Next, we performed a user study to compare our One-at-a-Time and All-at-Once learning strategies. Here the robot could reason over multiple features during two tasks: one task required correcting a single feature, and the other task required correcting multiple features of the robot’s objective. For the multiple feature task learning about one feature at a time was objectively superior: it led to a better final learning outcome, took a shorter path to the optimum, and had fewer incorrect inferences and human undoing along the way. But the results were not as clear for the single feature task: One-at-a-Time reduced unintended learning on the weights that were initially correct, but it hindered learning for the initially incorrect weights. Overall, study participants subjectively preferred One-at-a-Time to All-at-Once: they thought One-at-a-Time was better at learning the intended aspects of their corrections and required less undoing.

Based on these results, we hypothesize that the superior objective performance of One-at-a-Time was due to the increased complexity of the teaching task. It appears that only learning a single aspect at a time is more useful when the teaching task becomes more complex and requires that the human alter multiple parts of the robot’s objective. When the teaching task is simple, however, and only requires one aspect of the objective to change, it is not yet clear whether One-at-a-Time is a better learning strategy.

Throughout our paper we assumed that the robot had access to the necessary task-related features. Moreover, during our user studies the robot’s objective contained only two or three total features, and these features were intuitive to the human (e.g., “distance-to-person”). In practice objective functions will have larger features sets and may include task-related features that are non-intuitive to the human: additional work is needed to investigate how well our learning strategies perform in these cases.

Finally, solutions that can handle dynamical aspects—like preferences about the timing of the robot’s trajectory—would require a different approach for inferring the intended human trajectory. Here it may actually be necessary to return from the trajectory space to the policy space.

In this work we present an online, in-task response to pHRI that treats human interactions as intentional. We first formulate the problem of responding to pHRI as a partially observable dynamical system, where solving this system defines the optimal way for the robot to react. Unfortunately, this formalism is not directly applicable because we require online solutions in high-dimensional and continuous state, action, and belief spaces. We therefore derive an approximate solution for real-time learning and control. During offline simulations we compared our approximate learning method to a complete solution and state-of-the-art baselines, which are actually simplifications of our approach. We perform two separate user studies on a 7-DoF robot arm to determine (a) whether learning from pHRI is useful and (b) how the robot should learn from physical human interactions. While these simulations and user studies indicate the benefits of our approach, we recognize that this work is only a first step towards leveraging the implicit communication present during human-robot interactions.