Contrast, Imitate, Adapt: Learning Robotic Skills from Raw Human Videos

IAAformer is composed of three modules, which are shown in Fig. 4. Interaction-Aware Transformer Encoder is used to model the hand-object interaction with self- and cross-attention mechanisms while Temporal Transformer Encoder focuses on modeling global dependencies across the frames, yielding effective spatiotemporal and contextual cues within the input sequence. A Projection Head is used to merge frame-level features and output the 128-dimensional embedding $z_{t}^{h}$.

Given a frame $V_{t}$ and its adjacent frames in the receptive field, the action priors of each frame (including 2D keypoints $P_{2D}$, 3D keypoints $P_{3D}^{{}^{\prime}}$ of the human wrist, rotation $\theta_{wrist}^{{}^{\prime}}$ and the contact state $c$) are extracted as well as object information ($B_{obj},I_{obj}$) as described in Section III-A. Unlike previous works that only model image features [36, 25] or 3D human skeletons [28], IAAformer takes both action priors and object information as inputs.

We use self-supervised losses to learn the interaction-aware representations. The training process is shown in Fig. 5. Given two videos $V^{h_{i}},V^{h_{j}}$ of the same task, where the length is $M$ and $N$, respectively. Each frame is sent to AEnc and OEnc and encoded respectively as $z_{action}$ and $z_{obj}$, which forms $z_{m}$ by the subsequent modules of IAAformer.

Since the frame-level temporal correspondences between two videos are unknown, we utilize temporal cycle-consistency (TCC) loss [25] to mitigate this issue. First, for any frame feature $z_{m}^{i}$ in $V^{h_{i}}$, we find the nearest neighbor in $V^{h_{j}}$, denoted as $\hat{z_{n}^{j}}=argmax_{n\in[1:N]}sim(z_{m}^{i},z_{n}^{j})$. Then, we determine the nearest neighbor of $\hat{z_{n}^{j}}$ in $V^{h_{i}}$, i.e. $z_{k}^{i}$. The matching is cycle-consistent only if $k=m$. Therefore, the matching problem is transformed into the classification problem.

To enable gradient backpropagation, the softmax function is used to calculate the soft nearest neighbor for $z_{m}^{i}$, denoted as Equ. 3. $\alpha_{m,n}$ in Equ. 4 is the probability that the $m$-th frame of video $V^{h_{i}}$ matches the $n$-th frame of video $V^{h_{j}}$. Let $sim(u,v)=u^{T}v/\|u\|\|v\|$ denote the cosine similarity between $u$ and $v$. We add a temperature parameter $\lambda_{temp}$ to increase the similarity of similar features. The classification logits are $\hat{y_{m}}=softmax(sim(\hat{z_{n}^{j}},z^{i}_{m=[1:M]}))$. And the ground truth $y_{m}$ is a one-hot vector while the $m$-th dimension is set to 1. Finally, the TCC loss is optimized by Equ. 5.

However, when trained only on successful interaction videos, TCC loss cannot enable IAAformer to effectively capture the task-relevant structural information in the images. During task execution, minor trajectory deviations may lead to completely different execution results. For instance, for Open drawer task, even if the end effector of the robot is only one centimeter away from the drawer handle, the drawer may not be successfully opened. Therefore, the learned representations should also be sensitive to interactions in diverse failure cases.

To this end, we propose a hindsight interaction contrastive loss to improve interaction awareness by hindsight augmentation. Inspired by advances in Hindsight Experience Replay [40] and our variant works [11, 1] which find successful experiences from unsuccessful interactive data, the insight of our hindsight interaction contrastive loss is to obtain failure experiences from successful human demonstrations.

As is shown in Fig. 5, hindsight augmentation consists of action augmentation and object augmentation. For each frame of one video $V^{h_{i}}$, action augmentation adds appropriate Gaussian noise to 3D body joint rotation $\theta_{body}$ obtained in Section III-A, thereby obtaining the augmented 2D and 3D body keypoints after passing through the SMPL-X layer [33]. Therefore, $z_{1:M}^{i,a\_aug}$ is obtained by inputting ${z_{action}^{i}}^{\prime}$ and ${z_{obj}^{i}}$ to IAAformer. In addition, we use a simpler operation based on a certain probability, which uses action features extracted from other videos as ${z_{action}^{i}}^{\prime}$. Similarly, object Augmentation replaces $z_{obj}^{i}$ of $V^{i}$ with $z_{obj}^{j}$ of $V^{j}$ to obtain $z_{1:M}^{i,o\_aug}$. To obtain the object features with $M$ timesteps, we only use the first frame of $z_{obj}^{j}$ and repeat $M$ times.

After hindsight augmentation, our hindsight interaction contrastive loss is used to learn representations by minimizing the distance between positive pairs and maximizing the distance between negative samples. Specifically, we formulate our contrastive loss $L_{hic}$ for the positive pair $\{z_{m}^{i},z_{n}^{j}\}$ as follows:

where $sum_{neg}$ indicates the features in two sequences except the positive pair $\{z_{m}^{i},z_{n}^{j}\}$ as negative samples, and $sum_{a\_aug}$ and $sum_{o\_aug}$ indicate the hindsight augmented features as negative samples.

In addition, we propose a temporal monotonic loss $L_{mon}$ to ensure the monotonicity of the matching. To allow gradient backpropagation, we define a differentiable matching index $MI(z_{m}^{i},z^{j}_{1:N})$ for the nearest neighbor of $z_{m}^{i}$ in another feature sequence $z_{1:N}^{j}$ as follows:

where $\alpha_{m,k}$ is defined in Equ. 4. Then we formulate $L_{mon}$ as follows:

where $\sigma$ is ReLU activation [41]. Therefore, the total loss of IAAformer is the weighted sum of the three above losses, denoted as follows:

where $\gamma_{tcc}=0.3$, $\gamma_{hic}=0.6$ and $\gamma_{mon}=0.01$, which are determined through the experiments described in Section IV-C.

GAN inversion technique [44] is utilized to find the latent variable in the latent space of the pre-trained GAN with a generation instance. We use optimization-based GAN inversion to obtain the latent variable $v$ by gradient backpropagation. While such methods in computer vision take an amount of iterative time to optimize from high-dimensional images, our TrajGAN spend much less time in GAN inversion since the low dimensions of the output trajectories.

The schematic diagram of our GAN inversion is shown in Fig. 6 through blue lines. Taken as input the latent variable $v$, a generator $G(\cdot)$ of the pre-trained TrajGAN outputs the trajectory $\tau\in\mathbf{R}^{steps\times 7}$, as $\tau=G(v)$. Since the optimal trajectory cannot be available, we utilize the position of the target object to constrain the contact points $C(\tau)$ in the generated trajectory. We obtain the object position in test scenarios through RGBD camera calibration, which we denoted as the environmental variable $VE$. Therefore, we compute the optimization objective as follows:

We set the completion of the GAN inversion when $\|VE-C(\tau)\|_{2}<=\eta$ ($\eta=0.02$) and optimization epochs $<1000$. Due to optimization on low-dimensional trajectories, the GAN inversion only requires about 20 seconds, and the convergence speed may not be affected by different initial latent variables.

Another use of GAN inversion is to measure the success confidence of tasks in testing scenarios, which can be used to reject the execution of tasks with particularly low confidence in advance. The confidence $\theta_{success}$ is computed as follows:

When $\theta_{success}<0.99$, it indicates that TrajGAN cannot generate appropriate trajectories. The reason here is that the position of the current environment variable $VE$ is too far from the distribution of the training data.

The initialized trajectory still may not result in a high success rate in task execution, due to deviations between the contact position of the generated trajectory and the object, the inappropriate posture of the end effector, camera calibration errors, etc. Therefore, it is necessary to learn a policy to further refine the trajectory by interacting with the environment.

The key issue in interacting with real environments is the trade-off between interaction safety and exploration efficiency. We assume that the initialized rough trajectory is reasonable, and therefore the interaction trajectory should not be too far from it to cause danger, nor too close to it to cause inefficient exploration. To this end, we propose to discover semantic directions of trajectories in an unsupervised manner, which are utilized for reasonable and controllable editing of trajectories. Many works prove that GANs spontaneously have some directions in the latent space that represent semantic attributes of generations, such as the color, position and shape of objects [11]. In our case, the semantic direction may be the translation or rotation of the trajectory in a certain direction. By correctly finding these semantic directions, we can use them to make meaningful edits to the generated trajectory and control the intensity of the edits.

In TrajGAN, each layer learns a transformation from input to output. Focusing on the first four fully connected layers $G_{fc}$, the affine transformation on the latent variables $v\in\mathbb{R}^{8}$ can be expressed as follows:

where $y\in\mathbb{R}^{C_{y}}$ is the $C_{y}$-dimensional feature. $A\in\mathbb{R}^{C_{y}\times 8}$ and $b\in\mathbb{R}^{C_{y}}$ denote the weight and bias of $G_{fc}$ respectively. Like prior works [11, 45], the editing model on the latent variables is formulated as follows:

where $\beta$ denotes the editing scale. We can see that given any latent variable $v$, we can utilize $\beta An$ to adjust the intensity of editing. Therefore, the weight parameter $A$ should contain rich knowledge about the semantic attributes of the trajectory. The problem of discovering semantic directions can be factorized by solving the following optimization problem:

where $N^{*}=\{n_{1},n_{2},...,n_{K}\}$ denotes the top-$K$ important semantics. Lagrange multipliers ${\{\lambda_{k}\}^{K}_{k=1}}$ are introduced to solve this problem as follows:

By taking the partial derivative of $n_{k}$ in Equ. 18, we obtain

Therefore, we can calculate the eigenvectors of the matrix $A^{T}A$ and select the eigenvectors with the $K$ largest eigenvalues as the semantic directions $\{n_{k}\}_{k=1}^{K}$. With several proper semantic directions, agents can edit trajectories in a controllable and reasonable manner to explore in the environment.

Reinforcement learning algorithms suffer from low sampling efficiency in vast action spaces and complex environments. To this end, we introduce Cross Entropy Method [30] based on Trajectory Semantic sampling (TS-CEM) as an efficient gradient-free optimization method.

Given the task and the initial latent variable $v_{init}$, we sample a semantic direction $n_{k}$ from $k\sim N\{\mu_{k},\sigma_{k}\}$ and an editing scale $\beta$ from $N(\mu_{\beta},\sigma_{\beta})$. And we obtain the corresponding trajectory $\tau_{new}=G_{\theta}(v_{init}+\beta n_{k})$. Therefore, we execute R=20 trajectories in the real world and record the corresponding videos $V^{r}$. To iteratively improve the policy without human supervision, we utilize IAAformer to compute the alignment distances of randomly sampled human demonstrations $V^{h}$ and robot execution sequences $V^{r}$ as reward functions. The reward is computed as follows:

where $z^{r}_{t}$ denotes the frame feature in the robot’s execution video and $\hat{z}^{h}_{t}$ denotes the corresponding nearest neighbor frame in a randomly selected human demonstration video. Based on such rewards, we rank these trajectories and consider the trajectories of the top five highest rewards as elite trajectories, which can be used to compute the distribution parameters $\mu_{k},\sigma_{k},\mu_{\beta},\sigma_{\beta}$ of the semantic direction and the editing scale in the next iteration.

To perform experiments in various scenarios, we use a Realman RM65 robotic arm mounted horizontally on a mobile chassis with an Inspire five-finger dexterous hand as our humanoid agent, as is shown in Fig. 7. We use a Cartesian position controller to translate the end effector of the robotic arm and an in-built orientation controller to control the rotation. We simplify the degree of freedom of the hand into a 1-dimensional binary on-off state. An Intel Realsense D435 camera is utilized, which is mounted either on the robot head or externally fixed, depending on the task. In addition, we use 6 NVIDIA V100 GPUs to train all the networks. The full training takes about 5 hours.

To validate our framework CIA, we select six tasks that can be abundantly found in existing human daily behavior video datasets. The tasks include Grasping, Opening refrigerator, Closing refrigerator, Opening drawer, Closing drawer, and Wiping shown in Fig. 7a. Note that the human videos of these tasks lack corresponding robot actions, and they vary in terms of viewpoints, execution embodiments, and scenario layouts. The objects used in the testing scenarios are shown in Fig. 7b.

We choose 2853 videos in Dex-YCB [21] for Grasping of diverse objects in diverse viewpoints. Each other task has 365-950 videos from VLOG [22] and Charades [23], where various people record these everyday tasks with diverse objects in various environments. To select appropriate videos, we compute the temporal alignment distance between each video of the same task and the other 50 videos, and eliminate videos whose average temporal alignment distance is more than the threshold 0.5.

We use the success rate to measure the performance of the methods. Following the temporal alignment literature [25, 26, 28], we utilize three metrics for the evaluation of our IAAformer. After training the IAAformer, we freeze the parameters and evaluate in the trained representation space.