Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition

To get a learned agent in Sec. III-C, enabling human-agent joint learning, we follow the Denoising Diffusion Probabilistic Model (DDPM) [36] training paradigm. Here we first briefly introduce the DDPM algorithm. The forward process of the Diffusion Model can be regarded as adding Gaussian noise to the data $x^{0}$ according to a variance schedule $\beta_{1:K}$ by

where $\epsilon\sim\mathcal{N}(\mathbf{0,I}),\alpha_{k}=1-\beta_{k}$. DDPM models the output generation as a denoising process (Stochastic Langevin Dynamics). A line of works [37, 38, 8, 39] use diffusion model to generate the action for agents: given $x^{K}$ sampled from Gaussian noise $\mathcal{N}(\mathbf{0,I})$, it utilizes a parameterized diffusion process to model how $x^{K}$ is denoised in order to get noise-free action $x^{0}$ by

where $p_{\theta}(x^{k-1}|x^{k})=\mathcal{N}(\mu_{\theta}(x^{k},k),\Sigma(x^{k},k))$ is usually referred as reverse process. [40] shows that $p_{\theta}(x^{t-1}|x^{k})$ becomes tractable when conditioned on $x_{0}$ and Eq. 2 can be reformulated as minimizing the error in the noise prediction. [36] simplify the training loss function as

where step $k$ is sampled uniformly as $k\in[1,K]$, $\boldsymbol{\epsilon}_{\theta}$ is the noise prediction model. During the inference phase, we can generate $x_{0}$ by recursively sample $\boldsymbol{z}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I})$:

Similar to [8, 41], with the collected trajectory $\{(s_{i},a_{i})\}_{i=0}^{T}$, we aim to train an agent to imitate the trajectory, accomplishing a specific task $\mathcal{T}$. Therefore, we utilize DDPM to capture the conditional distribution of $p(a|s)$ and the training loss in Eq. 3 shall be modified as

Our system initially captures the raw sensory signal $\mathcal{I}$. Human hand pose $\mathcal{P}^{h}$ can be obtained from the captured signal using off-the-shelf 3D hand pose estimation [10, 11, 13]. The pose $\mathcal{P}^{h}$ consists of the positions of the human hand’s keypoints. Then, employing an inverse kinematic function $f_{\textit{IK}}$, we compute the action of the robot $a\in\mathbb{R}^{m}$, such that $a=f_{\textit{IK}}(\mathcal{P}^{h}_{t},\mathcal{P}^{h}_{t+1})$, where it is calculated upon the change in the hand pose. Given this teleoperation system, the human operator will move the hand to produce a sequence of hand poses $\{\mathcal{P}^{h}_{i}\}_{i=0}^{T}$ to teleoperate the robot with an action sequence $\{a_{i}\}_{i=0}^{T}$ to achieve the task $\mathcal{T}$. The human collected demonstration trajectory $\{(s_{i},a_{i})\}_{i=0}^{T}$, where $s\in\mathbb{R}^{n}$ is the robot state, could be used for downstream tasks.

After collecting data via teleoperation mentioned in Sec. III-B, we train a diffusion-model-based assistive agent to learn how to assist humans in collecting data in a shared control manner.

At an abstract level, the diffusion-model-based assist agent, noted as $f(\cdot|\cdot)$, is provided with the state $s$, denoising step number $k$, and a noise action $a^{k}$, which could be an imperfect action gathered from the teleoperation system or sampled from a Gaussian distribution, to predict the desired action

During data collection, the proposed system offers the option to control the robot in a shared control mode rather than directly applying the collected action $a^{h}$ from the teleoperation system. This leads to a reduced human workload during the data collection process. The classical shared autonomy method is achieved through the equation [29]:

where $a^{r}$ is generated by the learned agent. However, considering that the agent operates as a diffusion policy (Fig. 8), we blend the action from the human with the forward and reverse processes. Given action $a^{h}$, a forward process diffuses the action as follows: $a^{k}=a^{h}+\epsilon^{k}.$ Subsequently, a reverse process denoises the action $a^{k}$:

By applying action $a$, the control of the robot is shared between the human and the diffusion-model-based assistive agent. We can adjust the control ratio $\gamma=k/K$ between the human operator and the diffusion-model-based assistive agent by varying $k$. When $\gamma=0$, the action $a^{s}$ represents the teleoperation action $a^{h}$, which is the dexterous robot directly controlled by a human operator. As $\gamma$ approaches $1.0$, the action $a^{s}$ transitions to full autonomy $a^{r}$. A higher $\gamma$ value indicates a higher level of autonomy, allowing the learning-based agent more control rights to stabilize and direct the dexterous hand.

In this section, we show how to integrate data collection and manipulation learning into a unified framework that progressively reduces human effort and enhances robot autonomy.

We adopt six multi-stage manipulation tasks (Fig. 7). Pick-and-Place aims at picking an object on the table and placing it into a container. Articulated-Manipulation’s objective for the dexterous hand is to grasp and unscrew a door handle to open it, while for the gripper, it is to grab a drawer handle and pull the drawer open. Push-cube requires the robot to push the cube to the target position. Tool-Use aims at picking a hammer and using it to drive a nail into a board.

Our proposed system leverages shared control between human operators and learned agents to enhance the efficiency of data collection. To learn how the assistant agent could improve the data collection process, we conducted a user study.

In the user study, 10 human operators participate, collecting data under two modes: one where control is shared between the operator and the learned agent (w/ Ours), and the other where control is directly by the operator alone (w/o Ours). Each participant is instructed to collect as much data as possible within three minutes under two different modes for three dexterous hand tasks. Three metrics are evaluated: Success Rate (Percent) indicates the percentage of attempts where data collection was successful. Horizon Length (Steps per Sample) measures the length of each collected trajectory, with a lower horizon length indicating smoother data collection. Collection Speed (Samples per Hour) refers to the number of successful trajectories that can be collected in one hour.

In Tab. I, by sharing control between humans and learned agents, our system shows improvements in both success rate and collection speed, while the average horizon length of the collected trajectories is reduced. This suggests that our system enhances the efficiency of data collection by facilitating a process that is easier to succeed, faster, and more fluid in terms of trajectory smoothness. To ensure the fairness of the experiment wasn’t compromised, we equally divided the user group into two parts, Group 1 first collected data directly by themselves (w/o Ours) and then collected data with an assistive agent (w/ Ours), while the Group 2 reversed the order, first (w/ Ours) mode and then (w/o Ours) mode.

To gain deeper insight into how the learned agent assists the human operator, we visualize several keyframes from the data collection process of three dexterous hand tasks. From Fig. 11, it is evident that human operators are not required to provide too precise control with the assistive agent facilitating shared control over the dexterous hand. Instead, they only need to convey high-level intentions, such as the direction of hand movement or finger grasp motions. In multi-stage tasks, like picking up a hammer and then using it to drive a nail, operators only need to provide a trigger action to guide the agent to transition from one sub-stage to the next. As a result, less effort and attention are required, making the data collection easier to execute successfully and speeding it up.

When the learned agent shares control with users, the system effectively corrects imperfect human control signals to accomplish specific tasks. Given the challenge of directly measuring the level of imperfection in user signals and the correction ability of our system, we simulate human input using a baseline agent trained with Behavior Cloning (BC) as a proxy for user control.

In Fig. LABEL:fig:simulateduser, additional data collected under our framework effectively contributes to training improvements. The graph illustrates that with limited data availability, the agent can assist the simulated operator more effectively. As the agent gains access to and trains on more data, its ability to correct actions improves. These results indicate that our system gradually reduces the demand for the operator’s attention and effort, thereby enhancing the overall efficiency of data collection process.

Furthermore, once sufficient data is collected and the assistive agent is trained, it can transition into full autonomy mode by setting $\gamma=1$ and denoising actions from the noise sampled from Gaussian distributions. Across three different dexterous manipulation tasks, we can achieve success rates of 0.76, 0.78, and 0.89, indicating that the assistive agent can effectively transform into an automated dexterous manipulation agent.

From our experiments, we have observed that the assistive agent significantly aids human operators in managing fine control, especially in scenarios where accurate observation by humans is challenging. For instance, tasks such as grasping an egg or moving a hammer present visual challenges. It can be difficult to visually confirm whether the egg is securely grasped or if there’s a risk of it being dropped. This uncertainty makes it hard for human operators to react promptly to sudden changes. However, within our proposed joint learning framework, human operators are primarily required to focus on high-level intentions and task planning during data collection, while the assistive agent manages the detailed low-level actions. This division of labor significantly reduces the burden on human operators by clearly separating strategic planning from execution tasks, streamlining the collaboration between humans and machines.

In this section, we illustrate that collecting data under shared control does not compromise the quality of the data. We gather dexterous hand and gripper manipulation demonstrations via the proposed system in two modes: fully controlling the robots by a human ($\mathcal{H}$) and sharing control ($\mathcal{S}$) between the human operator and the learned assistive agent. And utilize these data to train different kinds of agents, like BC, BC-RNN [7], and Diffusion Policy (DP) [8].

In Tab. II, compared to directly collecting human demonstrations from the expert human operator, who can achieve success rates and efficiency comparable to those with agent assistance, the data collected by sharing control between the human and the assistive agent can achieve comparable or even surprisingly better results with BC and BC-RNN. Their results are comparable with DP, possibly as DP can better fit the tasks, which is in line with [8].

In Tab. III, we compare the effects of using different sets of data to train BC and DP. We can find that utilizing more data collected under the shared control mode leads to comparable performance on the tool-use and push-cube tasks. This verifies that the new data contributes significantly to policy learning and can achieve a similar effect compared to the data from human experts but at a much lower cost. These results indicate that the data collected under the proposed paradigm have sufficient quality and efficiency for downstream tasks.

To better evaluate our system, we further conduct real-world experiments. Three tasks are adopted: Pick-and-Place, Articulated-Manipulation, and Push-cube in Fig. 14. Following the same rules as Sec. IV-B, four human volunteers are invited to participate in the user study to collect data under two modes: one where control is shared between the human operator and the learned agent (w/ Ours), and the other where control is directly by the human operator alone (w/o Ours). Our proposed system achieves significant improvements in success rate and collection speed by sharing control between human operators and learned agents, as demonstrated in Tab. IV. Additionally, data gathered under our proposed joint learning shared control mode yield performance on the three tasks that are comparable to those pure human datasets using BC and DP, further substantiated by the results presented in Tab. V.

We have developed a questionnaire comprising shown in Tab. VI to capture various dimensions of user experience and ergonomics, and we invited 10 volunteers to rate our system based on their feedback.

This questionnaire assesses ease of use and overall satisfaction. The reliability of our questionnaire is supported by strong Cronbach’s alpha values: $\alpha=0.769$ for the satisfaction section and $\alpha=0.852$ for the user-friendly section, indicating internal consistency.