Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction

Contemporary research in motion data modelling predominantly explore learned data-driven methods, using neural networks to capture, model, and establish correlations between the motion features of various skeletal configurations [49, 53, 39]. Motion features alone have demonstrated considerable efficacy in cross-skeleton imitation using a variety of strategies, including task-driven objectives [43, 19], diffusion-based generation [47, 25, 50], and latent feature consistency [2, 1, 44, 3, 7]. Latent consistency in particular aims to learn inter-skeleton correlations without the need for paired datasets, which reduces the barrier of entry for practical applications.

Likewise, motion interpolation strategies have evolved through the use of learning based approaches. The inherent numerical precision of motion data has traditionally posed a challenge for the adoption of the learning approaches, due to their approximate nature [27, 21]. This challenge is further amplified by the temporally sparse distribution of keyframes. Recently, a number of recurrent neural network based methods [17, 18, 51, 41] and transformer-based networks [12, 33, 35] have demonstrated encouraging interpolation performance for the transition between keyframes. Additionally, various other motion modelling methods are capable for keyframe based interpolation, albeit with limited intermediate frame representation capabilities [43, 19, 25].

A major constraint inherent in all data-driven motion modelling methods is their dependency on datasets featuring specific skeletal configurations. To address this, skeleton-free training strategies have been explored for elementary tasks involving 3D characters. These include vertex-based pose transfer [30, 45, 8, 7], and point cloud-based human shape reconstruction [23, 46]. Notably, in these methods, 3D mesh and point cloud coordinates serve as general 3D data representations. Concurrently, other research has significantly minimized the required volume of training data, bringing it down to single example sequences. This reduction has been achieved through the use of patch matching techniques [28] and imitation learning within physics-based simulations [37, 29]. Reinforcement learning has facilitated the generation of goal-driven motion without pre-existing datasets [36, 48, 31]. However, these simulation-based methods have typically been either exceedingly complex to implement in practice, or yield results that are too inflexible and/or prone to errors for animation workflows.

Conventional approaches in motion re-targeting have been a cornerstone of 3D animation for decades, primarily dedicated to converting motion capture data into 3D skeletal motions [4, 15]. Typically, raw motion capture data is optically recorded, yielding primarily spatial information. In contrast, motion data is largely rotational in nature [5], often represented in SO(3) space. Therefore, a principal objective of motion re-targeting solutions has been to effectively bridge these two distinct types of representations. In [14], extensive manually adjustable constraints were introduced for flexible re-targeting between skeletons of identical topology (i.e., same hierarchy, different bone lengths and rest poses). Key practices in this approach, including end-effector matching, joint contacts, and inter-skeleton point correlations, are still relevant to current animation pipelines. Point correlations using Inverse Kinematics (IK) have also been proposed as an intermediate representation to facilitate the adaptation between topologically distinct skeletons [34, 9], albeit with hierarchical pairing limitations.

In response, morphologically-independent motion control systems were proposed using IK joint chains and joint-wise constraints [26, 20]. Such systems often expect highly standardised motion skeleton features, and are generally impractical for re-targeting motions produced for different control schemes. Additionally, the reliance on Euler angles for extensive classical re-targeting constraints poses a variety of issues regarding rotational freedom [11], as well as compatibility with animation workflows [32], which predominantly use quaternion-based systems.

Despite these advancements, the challenge of arbitrary skeleton re-targeting from motion datasets still remains largely unresolved, limiting any widespread application of data-driven pipelines. Our paper aims to effectively address this pervasive limitation.

We first declare a set of notations for representing skeletons. A skeleton $S$ is defined by a tree of bones $\mathcal{B}$. A bone $\mathbf{v}_{n}\in\mathcal{B}$ is defined by its bone parent $\mathbf{v}_{m}\in\mathcal{B}$, and a head position $\mathbf{H}_{n}\in\mathbb{R}^{3}$ which denotes the bone’s base offset from its parent head $\mathbf{H}_{m}$. In addition, a tail position $\mathbf{T}_{n}\in\mathbb{R}^{3}$ helps to define the end point of the bone’s line segment representation, starting from its head position, of $\mathbf{v}_{n}$. The root bone $\mathbf{v}_{0}$ has no parent, and thus no head position $\mathbf{H}_{0}$. The set of $\mathbf{H}_{n}$ and $\mathbf{T}_{n}$ for each bone $\mathbf{v}_{n}\in\mathcal{B}$ make up the rest position of the skeleton.

Next, we define the properties of motion data. At any given frame $t$, a skeleton’s pose position can be represented as global 3D positions $\mathbf{p}_{n,t}\in\mathbb{R}^{3}$ and global quaternion rotations $\mathbf{q}_{n,t}\in\mathbb{R}^{4}$ for each bone $\mathbf{v}_{n}\in\mathcal{B}$. For a given motion sequence $M$, only the root bone is given a 3D position in pose position, i.e. $\mathbf{p}_{0,t}$, as all other pose positions are derived using Forwards Kinematics (FK [11, 24]):

where $QR$ is the quaternion rotation function, and $\mathbf{v}_{m}$ is the parent joint. In summary, a motion sequence $M$ of $|M|$ frames is defined by:

Our approach is based on the hypothesis that 3D motion sequences can be visually represented by a more universal format: 3D point cloud sequences. By geometrically sampling the volume around a skeleton, point cloud data becomes a non-hierarchical representation of its pose(s), obfuscating the original skeletal configuration. Due to this decoupling between the skeleton configuration and motion representation, a successful motion data reconstruction from point cloud data would be the first step to enabling learnable cross-skeleton motion features. Formally, for a given source skeleton $S_{A}$ with an associated motion sequence $M_{A}$, and any humanoid skeletal configuration $S_{B}$, the point cloud obfuscation and reconstruction module is to learn a function that can adequately perform $F_{S_{A}\rightarrow S_{B}}(M_{A})=\hat{M}_{B}$, where $\hat{M}_{B}$ is a visually similar motion adaptation of $M_{A}$ on skeleton $S_{B}$, which is an estimation of $M_{B}$.

Point clouds lack absolute roll axis information for each bone. As such, we introduce a First-frame Offset Quaternion (FOQ) transform to incorporate the available relative roll data for motion modelling. We formulate all quaternions of a motion sequence relative to their values from the initial frame. In detail, for any quaternion sequence $Q=\{\mathbf{q}_{0},\mathbf{q}_{1},\mathbf{q}_{2},...\}$, we can simply obtain its FOQ transform as $\{\mathbf{q}_{0}\times\mathbf{q}_{0}^{-1},\mathbf{q}_{1}\times\mathbf{q}_{0}^{-1},\mathbf{q}_{2}\times\mathbf{q}_{0}^{-1},...\}$, where $\mathbf{q}_{0}^{-1}$ is the quaternion conjugate of $\mathbf{q}_{0}$. An FOQ sequence can easily be converted back to a quaternion sequence as long as any $\mathbf{q}_{t}\in Q$ is known.

Crucially, FOQ is a roll-invariant rotation representation, that is, FOQ remains identical regardless of a bone’s absolute roll position in the original data. Trivially, to adjust the rotational roll of a bone (e.g., $\mathbf{v}_{n}$), we perform the quaternion multiplication for each frame (including the initial frame): $\mathbf{q}_{n,t}\times\mathbf{r}$, where $\mathbf{r}$ is a roll quaternion. As a quaternion on the roll axis of $\mathbf{v}_{n}$, $\mathbf{r}$ is constrained by $\mathbf{r}=\alpha+\beta(x\mathbf{i}+y\mathbf{j}+z\mathbf{k})$, where $x$,$y$,$z$ are the 3D XYZ values of $\mathbf{T}_{n}$, and $\alpha,\beta$ are scalars that control the roll magnitude. Note that $\mathbf{r}$ is constant throughout the sequence. We can then deduce that for any constant $\mathbf{r}$,

Therefore, FOQ is indeed roll-invariant.

To leverage our point cloud-based representation learning for cross-skeleton interpolation, a transformer model - CITL [33] is adapted, allowing for an efficient interpolation pipeline with existing datasets to alternative skeletons. We introduce two key modifications to CITL and its training strategy, focusing on enhancing motion quality. First, quaternion predictions are substituted with FOQ predictions. This is inspired by the RNN-based method $\text{TG}_{complete}$ [18], which predicts quaternion offsets from the preceding frame rather than direct quaternion values, thereby achieving greater generalizability and superior quality in interpolation.

For our model’s CITL-based architecture, we substitute the original sinusoidal positional encoding scheme with learned relative positions [42]. While the original implementation relies on the continual nature of sinusoidal functions, we observe that relative positional embeddings with zero initialisation converges upon similarly continuous behaviour, and significantly lowers the complexity of positional attention. The lessened focus on positional relations allows the model to synthesise deeper behaviours within the pose space, improving overall generalisability. As a side effect, the model also accepts arbitrary length inputs beyond the training data length.

Notably, the intended rest pose of a skeletal configuration is often geometrically sub-optimal for point cloud matching, and can manifest in multiple forms. To address this, we devise a rest position augmentation strategy, leveraging a per-bone Inverse Kinematics (IK) system. RPA broadens the range of intended rest poses that the learned interpolation model can accommodate. In detail, during each phase of the Forward Kinematics (FK) process, we augment the bone tails with a random additional offset. We re-align the augmented global tail position as closely as possible to its original location using a quaternion rotation with the maximum possible real component. Algorithm 1 describes the method by which this quaternion can be derived. Since the real component describes, in cosine, the amount of rotation around an axis, this minimizes numerical disturbance from RPA while altering the visual representation to the desired extent.

To thoroughly evaluate the effectiveness of the proposed method, we conducted extensive experiments on the following widely used motion capture datasets:

• •

  LaFAN1 [18] contains long, high quality motions in controllable video game character styles. The dataset was recorded on a large motion capture stage, and manually cleaned to production standards. The native skeleton of LaFAN1 contains 6 spinal bones, and 4 bones for each limb.

• •

  Human3.6M [22, 6] comprises 3.6 million motion frames. The motions captured in Human3.6M tend to be less dynamic, as they were recorded on a relatively compact 4x3m stage. Its native skeleton includes 5 spinal bones, 4 bones for each limb, and an additional finger bone for each hand.

• •

  CMU Mocap [10] is a diverse motion dataset encompassing 113 overlapping categories. It features a mix of static and dynamic motions, captured on a spacious 3x8m stage. The CMU skeleton is identical to that of Human3.6M, with the exception of the pelvis, which is divided into two symmetrical hip bones and a lower back bone.

For PC-MRL, we designated the CMU MoCap skeleton as $S_{B}$. Its native dataset is exclusively utilized for testing purposes; while LaFAN1 and Human3.6M are used simultaneously for training.

In our experiments, point cloud obfuscation adopted a 256-point setting for the sampling process with $\sigma=0.05$ and each point was characterized by a 64-dimensional vector. To maintain consistency, we ensure that all bone offsets and motion positions are defined in metre units. For skeletal reconstruction, 4 Point Transformer [52] layers was utilized to construct the neural network. Within each layer, the feature size was doubled, and the size of the point set was downsampled by a factor of 4 through a farthest-first traversal approach. This processing across layers resulted in a singular feature vector. In the final step, two linear layers with ReLU activations were utilized to transform the resulted feature vector in a frame-wise manner, producing the output skeletal representation. During training, we set $K=8$ for the KNN loss to optimize the network. Additionally, the use of unit quaternion values in practice relies on modulus division for constraint, a process that can potentially lead to exploding gradients. To address this, we introduced an objective function $\mathcal{L}_{q}$ that measures the $\ell_{2}$ norms’ difference between the raw quaternion output and the expected unit quaternion. All experiments were trained and evaluated on a single NVIDIA GeForce RTX 3090 GPU.

To demonstrate the efficacy of our method, we produce and compare our method against the conventional LERP method, as well as three state-of-the-art interpolation models trained on the original CMU MoCap dataset,. Specifically, we measure the performance of the RNN-based $\text{TG}_{\text{complete}}$ model [18], the BERT-based motion interpolation adaptation [12], and a transformer-based encoder-decoder approach [33]. We additionally provide results on a variant of our method trained without our RPA strategy. The experimental configuration for each state-of-the-art model is identical to their original implementations. To measure interpolation accuracy, we employ the standard $\ell_{2}$ positional distance (L2P), $\ell_{2}$ quaternion difference (L2Q), and NPSS [16] for visual similarity evaluation [18].

The results listed in Table 1 clearly demonstrate our method’s ability to approach the accuracy levels exhibited by directly supervised state-of-the-art methods. PC-MRL consistently outperforms both the LERP standard and RNN-based $\text{TG}_{\text{complete}}$ model, particularly in longer keyframe interval scenarios where the quality and depth of learned motion features is most critical and impactful. Likewise, the inclusion of RPA also tends to improve the long interval performance of PC-MRL, particularly when precise movements (i.e. golf) or deeper motion features (i.e. locomotion) are expected. Given this, we can observe that RPA definitively improves the overall consistency of the PC-MRL method in scenarios that may be unseen or less effectively represented by our point cloud-to-motion system.

Native supervision, i.e. with CITL, unsurprisingly produces the most accurate interpolation model, with a notable exception of swimming motions, where our PC-MRL supervision produces even stronger results. We believe the abundance of low crawl motions, a similar motion to swimming in LaFAN1, provides more meaningful supervision over the original CMU dataset, which conversely has few similar motions once swimming motions are filtered out. On the other end of the spectrum, i.e. for high data scenarios such as walking and running locomotion from Fig. 5, all models including our PC-MRL approach observe the strongest improvements for learned methods over the naive LERP method. Both cases strongly support our method’s capability to supervise highly intricate motion patterns, despite the incompleteness of the point cloud representation and reliance on relative rotations and geometric optimisation assumptions.

We further conduct an experiment on motion re-targeting to directly benchmark our method against the sole existing state-of-the-art method for unpaired motion re-targeting with unrestricted skeletons, i.e. primal skeletons (PS) [1]. For direct motion re-targeting evaluation, we utilized the KNN-Loss and end-effector loss as metrics, owing to their effective measurement of visual similarity and non-sensitivity towards absolute roll axis correctness. Though absolute rolls are generally necessary for a complete motion re-targeting solution, this aspect is out of scope for our project as our main goal is motion interpolation.

Table 2 indicates the superior performance of our method over the state-of-the-art PS method in terms of visual similarity. At low $\sigma$ values, the higher relative $\mathcal{L}_{\text{KNN}}$ of the primal skeleton method underscores its geometric deviations from the original motion. In contrast, our method demonstrates significantly better visual adherence to the original geometry. As $\sigma$ increases, our method approaches near-perfect alignment with the original motion. Figure 6 visually indicates this, showing our method follows the original motion closely, while the latent consistency-based primal skeleton method struggles to generalise and decode the skeleton’s internal structures without directly supervised objectives, such as the end-effector positions provided during training. In addition, Fig 7 demonstrates that, like existing re-targeting approaches, our method is able to produce adequate results on disproportionate skeletons.

Since our approach is largely focused on enabling non-native motion interpolation supervision, we additionally compare our PC-MRL method against a CITL model supervised by PS. Due to the suboptimal re-targeting performance of PS, models supervised by PS exhibit expectedly poor accuracy, as shown in Table 3.