Self-Supervised Motion Retargeting with Safety Guarantee

Data-driven motion retargeting methods have been widely used due to its flexibility and scalability [2, 3, 4, 5]. Owing to the merits of such methods, [7] was able to generate convincing motions of non-humanoid characters (i.e., a lamp and a penguin) from human MoCap data. Many data-driven techniques have relied on Gaussian process latent variable models (GPLVM) to construct shared latent spaces between the two motion domains [2, 3]. More recently, [8] proposed associate latent encoding (ALE), which uses two different variational auto-encoders (VAEs) with a single shared latent space. A similar idea was extended in [9] by incorporating negative examples to aid safety constraints. However, these methods do not consider the different expressivity of humans and robots. That is, multiple human poses can be mapped into a single robot configuration due to physical constraints. In this regard, we present the projection-invariant mapping while constructing the shared latent space.

Self-supervised learning has found diverse applications in robotics such as learning invariant representations of visual inputs [10, 11, 12], object pose estimation [13], and motion retargeting [14, 15]. The necessity of self-supervision arises naturally in robotics, where every piece of data often requires the execution of a real system. Collecting large-scale data for robotics poses significant challenges as robot execution can be slow and producing high-quality labels can be expensive. Self-supervision allows one to sidestep this issue by generating cheap labels (but with weak signals) for vast amounts of data. For instance, [16] creates a large dataset of Unmanned Aerial Vehicle (UAV) crashes, and labels an input image as positive if the UAV is far away from a collision and negative otherwise.

Moreover, self-supervised learning can be used to train representations that are invariant to insignificant changes, such as change of viewpoint and lighting in image inputs [10]. As for self-supervised learning methods for motion retargeting, which is the problem of our interest, [14] and [15] train motion retargeting networks with unpaired data using a cycle-consistency loss [17] in either the input or the latent space. While both methods do impose a soft penalty for violating joint limits during the training procedure, they do not guarantee feasibility as the joint limits are not strictly enforced, and self-collision is not taken into consideration.

We first introduce basic notations. We denote by $\mathbf{x}$ and $\mathbf{q}$ the human skeleton representation and robot joint configuration, respectively. Specifically, $\mathbf{x}\in\mathbb{X}\subset\mathbb{R}^{21}$ is a $21$-dimensional vector consisting of seven unit vectors: hip-to-neck, both left and right neck-to-shoulder, shoulder-to-elbow, and elbow-to-hand unit vectors. A robot configuration $\mathbf{q}\in\mathbb{Q}\subset\mathbb{R}^{N_{\text{DoF}}}$ is a sequence of $N_{\text{DoF}}$ revolute joint angles²²2 This could limit the motion retargeting results in that orientation information is omitted. However, this makes the proposed method compatible with the output of pose estimation algorithms for RGB videos. . Moreover, we denote relaxed skeletons and robot configurations that do not necessarily satisfy the robot’s physical constraints by $\tilde{\mathbf{x}}$ and $\tilde{\mathbf{q}}$, and feasible skeletons and configurations by $\bar{\mathbf{x}}$ and $\tilde{\mathbf{q}}$. Throughout this paper, we use robot poses and joint configurations interchangeably as they have one-to-one correspondences. Our goal of motion retargeting is to find a mapping from $\mathbb{X}$ to $\mathbb{Q}$ (i.e., human skeleton space to robot configuration space),

To handle the diverse human poses that map into a single robot pose, we leverage self-supervised learning for constructing the shared latent space. Specifically, the following normalized temperature-scaled cross entropy (NT-Xent) loss will be used:

$\displaystyle\ell(i,j)=-\log\frac{\exp(\cos(\mathbf{z}_{i}^{\prime},\mathbf{z}_{j}^{\prime})/\tau)}{\sum_{m\neq i}\exp(\cos(\mathbf{z}_{i}^{\prime},\mathbf{z}_{m}^{\prime})/\tau)}$

$\displaystyle\mathcal{L}_{\text{nt-xent}}=\frac{1}{2N}\sum_{m=1}^{N}\ell(2m-1,2m)+\ell(2m,2m-1)$

where $\mathbf{z}_{i}^{\prime}$ is the $i$-th latent vector on latent space mapped by an encoder network and $\tau$ is a temperature parameter which we set to be $1$. In image classification domains, $\mathbf{z}_{i}^{\prime}$ and $\mathbf{z}_{j}^{\prime}$ correspond to the feature vectors of two images that are augmented from a single image with different augmentation methods [19]. In our setting, $\mathbf{z}_{i}^{\prime}$ and $\mathbf{z}_{j}^{\prime}$ correspond to latent representations of skeletons that map to the same robot configuration. We present two sampling processes in Sections IV-B and IV-C that allows us to construct $\mathbf{z}_{i}^{\prime}$ and $\mathbf{z}_{j}^{\prime}$ pairs which are encoded from a feasible (retargetable) and a relaxed (possibly not retargetable) skeletons, respectively. Figure 1 illustrates the overview of the two sampling processes.

Our proposed method still requires some amount of paired data, so we use an optimization-based motion retargeting method [20] to create a small paired dataset of human skeletons and corresponding robot configurations. However, unlike the approach taken in [20], which simply computes pairs of human poses ($\widetilde{\mathbf{x}}$) and robot poses ($\overline{\mathbf{x}}$), we carefully design an additional sampling routine so that the resulting tuple can be used for contrastive learning.

First, we randomly select a human pose from the motion capture dataset ($\widetilde{\mathbf{x}}$) and run optimization-based motion retargeting without considering the joint limits and self-collision, which gives us a relaxed robot pose ($\widetilde{\mathbf{q}}$). Note that this process takes considerable amount of time due to solving inverse kinematics as a sub-routine. Then, we compute the feasible robot pose ($\overline{\mathbf{q}}$) from $\widetilde{\mathbf{q}}$ using the method presented in [6]. This process can be seen as projecting $\widetilde{\mathbf{q}}$ to $\overline{\mathbf{q}}$ (i.e., $\texttt{Proj}_{\mathbb{Q}}:\widetilde{\mathbf{q}}\mapsto\overline{\mathbf{q}}$). Finally, from $\overline{\mathbf{q}}$, we compute the skeleton representation by solving forward kinematics and concatenating appropriate unit vectors described in Section IV-A (i.e., $\overline{\mathbf{x}}=\texttt{FK}(\overline{\mathbf{q}})$). The collected tuple ($\widetilde{\mathbf{x}}$, $\overline{\mathbf{x}}$, $\widetilde{\mathbf{q}}$, $\overline{\mathbf{q}}$) is used for both learning the projection-invariant encoding and constructing the shared latent space between the two domains for motion retargeting.

Here, we present a data sampling procedure which requires only the robot’s kinematic information. This has two major advantages. 1) it alleviates the cost of collecting a large number of expensive paired data obtained via solving inverse kinematics, and 2) it allows the learning-based motion retargeting algorithm to not be limited to the motion capture data used in Section IV-B.

We first sample $\tilde{\mathbf{q}}\in\widetilde{\mathbb{Q}}$ where $\widetilde{\mathbb{Q}}$ is a relaxed robot configuration space. In particular, this relaxed configuration space is constructed by relaxing the joint limits with a relaxation constant $\alpha$. If the joint angle range of $i$-th joint is $[l_{i},\,u_{i}]$, then we define the $\alpha$-relaxed joint range as $[l_{i}-\frac{\alpha}{2}(u_{i}-l_{i}),\,u_{i}+\frac{\alpha}{2}(u_{i}-l_{i})]$. The relaxed configuration is converted to the corresponding robot pose by solving forward kinematics where the skeleton features (i.e., $7$ unit vectors) are extracted from the robot pose. We denote by $\tilde{\mathbf{x}}=\texttt{FK}(\tilde{\mathbf{q}})$ the skeleton features given by the relaxed configuration.

The main purpose of sampling $\tilde{\mathbf{q}}$ from $\widetilde{\mathbb{Q}}$ is to collect $\tilde{\mathbf{x}}$ that is less restricted to the physical constraints of the robot model. In other words, if we simply use samples from the robot configuration with exact joint ranges, the data is likely to not cover the human skeleton space sufficiently. Yet, if we directly use the collected relaxed pairs $(\tilde{\mathbf{x}},\,\tilde{\mathbf{q}})$ for training the learning-based motion retargeting, it may cause catastrophic results in that $\tilde{\mathbf{q}}$ may not satisfy the kinematic constraints.

To resolve this issue, we use the same projection mapping from Section IV-B to map the relaxed configuration $\tilde{\mathbf{q}}$ to the feasible robot configuration $\bar{\mathbf{q}}$ that is guaranteed to satisfy joint range limits and self-collision constraints (i.e., $\bar{\mathbf{q}}=\texttt{Proj}_{\mathbb{Q}}(\tilde{\mathbf{q}})$). We also compute the corresponding skeleton feature from the feasible configurations (i.e., $\bar{\mathbf{x}}=\texttt{FK}(\bar{\mathbf{q}})$). To summarize, we sample the following tuples for training our learning-based motion retargeting method:

	$\displaystyle\tilde{\mathbf{q}}$	$\displaystyle\sim P_{\widetilde{\mathbb{Q}}}$
	$\displaystyle\tilde{\mathbf{x}}$	$\displaystyle=\texttt{FK}(\tilde{\mathbf{q}})$
	$\displaystyle\bar{\mathbf{q}}$	$\displaystyle=\texttt{Proj}_{\mathbb{Q}}(\tilde{\mathbf{q}})$
	$\displaystyle\bar{\mathbf{x}}$	$\displaystyle=\texttt{FK}(\bar{\mathbf{q}})$

With slight abuse of notation and denoting function composition by $\circ$, we can represent $\bar{\mathbf{x}}=\texttt{Proj}_{\mathbb{X}}(\tilde{\mathbf{x}})$, where $\texttt{Proj}_{\mathbb{X}}=\texttt{FK}\circ\texttt{Proj}_{\mathbb{Q}}\circ\texttt{IK}$ even though we are not actually solving inverse kinematics in the sampling process. Figure 2 illustrates the proposed projection-invariant mapping.

We also leverage a mix-up style data augementaton [21]. More precisely, we generate auxiliary relaxed robot skeletons $\widetilde{\mathbf{x}}^{\prime}=\beta\widetilde{\mathbf{x}}+(1-\beta)\overline{\mathbf{x}}$, where $\beta$ is uniformly sampled from the $[0,1]$ interval. This has the effect of smoothing the decision boundary of the motion retargeting inputs.

We found that 200K pairs of skeleton features and corresponding robot joint angles generated from the sampling method in Section IV-C were not sufficient enough to properly perform motion retargeting.

This stems from the fact that the coverage of randomly sampled data becomes sparse at an exponential rate in the ambient dimension, which is typically referred to as the “curse of dimensionality”. One can try increasing the number of samples, but the corresponding increase in query time would render it impractical for real-time retargeting when using exact nearest neighbor search. An avenue for future research is using sublinear-time algorithms for retrieving approximate nearest neighbors.

Our method remedies the insufficient coverage of random data by using a small number of human motion capture data from Section IV-B to efficiently cover the low-dimensional manifold of human skeletons. However, we note that our method has the potential to be fully self-supervised, in the sense that no data obtained by solving inverse kinematics is required, if our random configuration sampling in Section IV-C can be improved to efficiently cover the space of human skeletons.

We first conduct a synthetic experiment to conceptually highlight the benefit of using S³LE for constructing the shared latent space. The goal is to find a mapping from Relaxed $X$ to Feasible $Y$, where multiple points in Relaxed $X$ correspond to a single point in Feasible $Y$. This resembles our motion retargeting setup since multiple human skeletons can map into a single robot configuration due to the limited expressivity of humanoid robots.

We set Relaxed $X\subset\mathbb{R}^{2}$ and Feasible $X\subset\mathbb{R}^{2}$ to be $[0,2]\times[0,2]$ and $[0,1]\times[0,1]$, respectively, and define the projection as $\texttt{Proj}_{X}:(x_{1},x_{2})\mapsto(\min(x_{1},1),\min(x_{2},1))$. Note that there exists a one-to-one mapping from Feasible $X$ to Feasible $Y$ as shown in Figure 3. We randomly sample $10,000$ points from Relaxed $X$ (which contains Feasible $X$ as a subset) to optimize a projection-invariant mapping and $20$ pairs as the glue data for constructing the latent space shared by $X$ and $Y$. The mappings learned by S³LE, which is trained with the NT-Xent loss, and a baseline method, trained without the NT-Xent loss, are illustrated in the rightmost side of Figure 3. We can see that our method learns a better mapping from Relaxed $X$ to Feasible $Y$. This example shows that training with a contrastive loss improves the performance when the mapping is not one-to-one as the projection invariant region shown with red dotted circles are better mapped with the proposed method.

For the target robot platform, Compliant Humanoid Platform (COMAN) is used in a simulational environment where we only consider the revolute joints of its upper body and a subset of $10$ motions ($1,057$ pairs) from CMU MoCap DB is used to collect a list of paired tuples ($\widetilde{\mathbf{x}}$, $\overline{\mathbf{x}}$, $\widetilde{\mathbf{q}}$, $\overline{\mathbf{q}}$).

We compare S³LE with two baseline methods, associate latent embedding (ALE) [8] and locally weighted latent learning (LWL²) [6]. Both methods, ALE and LWL², leverage a shared latent space, but LWL² additionally guarantees feasibility of the retargeted pose using nonparametric regression. S³LE also guarantees feasibility in the same way, i.e., by using nonparametric regression in the latent space. Furthermore, the mixup-style augmentation in S³LE yields more training data for constructing the shared latent space.

We first select $10$ motions from the CMU MoCap database and run optimization-based motion retargeting to get $1,008$ pairs of skeletons and corresponding robot configurations. We also sample $100,000$ pairs from the proposed sampling process in Section IV-C, $30,000$ from robot-domain specific data, and $100,000$ from MoCap-domain specific data. All neural networks (encoders, decoders, and discriminators) have three hidden layers with $512$ units. We use the tanh activation function and learning rate of $10^{-3}$.

We conduct comparative experiments using a subset of CMU MoCap DB [22] that was not observed while training as test set. In particular, we chose $27$ expressive motions, compute the corresponding poses of COMAN using an optimization-based motion retargeting [20], and use them as the ground truth. To evaluate the performance of each retargeting method, we compute the average cosine distance of seven unit vectors (hip-to-neck, both left and right neck-to-shoulder, shoulder-to-elbow, and elbow-to-hand) between the retargeted and ground truth skeletons. The average cosine distances of the methods are shown in Figure 4. In most cases, S³LE outperforms the baselines in terms of accuracy measured by the cosine distance. Note that the poses of both S³LE and LWL² are guaranteed to be collision-free while the average collision rate of ALE is $24.9\%$. Figure 5 shows snapshots of input skeleton poses and retargeted motions. S³LE accurately retargets the motion when the input skeleton is lifting both hands up, whereas the baseline methods fail to do so. We hypothesize that this is because the training data from optimization-based motion retargeting failed to cover such poses while the data sampled using the COMAN’s kinematic information (Section IV-C) contained such poses.

We further show that the proposed method can be used to generate natural and stylistic robotic motions from RGB videos. We use FrankMoCap [23], the state-of-the-art $3$-D pose estimation algorithm, to collect sequences of human $3$-D skeleton poses from YouTube videos. Since there is no ground truth, we simply show snapshots of reference RGB images and corresponding COMAN poses in Figure 6 for qualitative evaluation. More videos can be found in the supplementary materials.