Hierarchical Neural Implicit Pose Network for Animation and
Motion Retargeting

Constructing poseable 3D character models from observations is a long-standing problem in computer graphics. A number of representations are available for driving the deformation of a 3D shape. For articulated character animation, linear blend skinning (LBS) [30] is a widely popular approach. In LBS, vertices of a 3D mesh are assigned to the bones and joints of a controlling ”skeleton” using vertex-level skinning weights, resulting in a 3D ”rig”. These weights are used to propagate bone transformations to the underlying surface geometry. There is a wealth of work on automatically creating LBS-style rigs for a character mesh [3, 23, 26, 28, 53], however this representation is far from perfect. Simple LBS formula is known to introduce artifacts and sometimes wildly unrealistic deformations (known as the ”candy-wrapper” effects), prompting research on alternative formulations  [25], and on incorporating pose-specific examples to correct the final shape [29]. Such pose-specific guidance is still used today, and most production rigs are intricate works requiring hours of highly skilled labor. The need for pose-specific control over deformation and the labour-intensive nature of explicit 3D rigs makes neural implicit representations for articulated characters a promising direction for new solutions in this space.

Implicit representations, such as Signed Distance Fields (SDF), have found use in representing and tracking deforming shapes [17, 39]. Most of these grid-based representations faced a trade-off between quality of representation and a prohibitive growth in memory requirements. Various approaches resorted to space-partitioning data structures or hashing functions to address this limitation [19, 40]. Neural implicit representations [4, 42, 37, 9, 10] have surpassed these approaches in terms of both quality and scalability.

Within animation, the promise of neural implicit representations is representing rich pose-dependent geometry of an articulated character by learning from observations alone. Our work follows this motivating principle and requires only points and normals sampled from the surface of the deforming geometry and the corresponding skeletal poses for training. While several recent works also aim to represent articulated 3D humans using a deep implicit representation (See Tab. 1), almost none follow the same low-supervision regime as our method. NASA [18] and imGHUM [1] both devise a hierarchical or part-based models, similar to ours, but are supervised by difficult to obtain data: imGHUM requires a parametric template mesh and is trained using existing GHUM parameterization [52], and NASA training relies on skinning weights, which are not only expensive to generate but might also constrain the model to the space of LBS rigs. Like NASA, LEAP [38] relies on skinning weights, and NPMs [41] need a canonical pose and dense correspondences for every subject. NPMs are also not directly posable, requiring test-time optimization to find a pose encoding for any new pose. In addition, imGHUM [1] and LEAP [38] are closely tied to existing parametric models of the human body, namely GHUM [52] and SMPL [34], respectively, which makes them difficult to extend to other classes of articulated characters. Additionally, LEAP [38] and more recent work like SCANimate [46] rely on LBS within their model, inheriting the artifacts of traditional skinning. This was also noted in the follow-up work Neural-GIF [49]. Out of prior work focused on articulated neural implicit models (Table 1), only SNARF [8] follows our philosophy of low-supervision regime.

While SNARF [8] demonstrates superior results compared to NASA [18], especially on out-of-domain poses, and is trained in a low-supervision regime, it introduces an expensive root-finding step at inference time and cannot handle multiple subjects with a single network or generalize across identities. The key insight behind SNARF is learning a single pose-independent forward skinning transform to represent the bulk of the deformation, unlike approaches like [38, 24], which learn a pose-specific backward skinning function in order to look up occupancy values in the canonical space. Learning pose-specific transform makes these methods vulnerable to over-fitting to the training poses and introduces discontinuities when the same point in the posed space (e.g. touching hands) maps to very disparate locations in the canonical space. However, this key insight is also the cause of SNARF’s limitations, such as the root-finding step. In our work, we show that a careful set of design decisions can extend simpler feed-forward hierarchical approaches [18, 1] to superior generalization to novel poses as well as multi-subject training, all in a low-supervision regime and with an ability to generalize beyond humans, like SNARF. As the result, our work is an important step in this new and exciting line of research.

Other recent works address modeling moving human characters in slightly different regime, for example by learning a controllable neural radiance fields from carefully calibrated multi-camera footage [31] or by focusing on a mesh representation [14]. In addition, an extensive line of research on implicit representations focuses on the problem of reconstructing and completing 3D shapes from partial or 2D inputs [11, 13, 22, 44, 45]. We omit a detailed discussion of these works to maintain focus on articulated neural implicit representations of 3D shape through occupancy or signed distance functions, the subject of our study.

Given a dataset of oriented point clouds (i.e., points and normals) for different subjects that share the same underlying skeleton (e.g., human), as well as the 3D pose of that skeleton for each point cloud, our goal is to train a coordinate-based model, conditioned on pose and subject, that generates a valid SDF with a zero-crossing at the given surface points. In effect, we aim to learn an articulated SDF that can be animated by novel skeletal poses and can represent a number of subjects with a single trained model.

Overview: Given a skeletal pose $\theta$ and subject encoding $\beta$, our model HIPNet predicts SDF values at input query points $x\in R^{3}$. Like some prior modern approaches [18, 1], our model is composed of part-based MLPs. We rely on a consistent skeletal structure of the training dataset and structure our model hierarchically, with a separate MLP sub-network for each of $N$ joints in the skeleton. The output of every sub-network $k$ is a signed distance value $F_{k}(x,.)$ that goes to zero at the surface with a gradient equal to the surface normal. Unlike [18] and [1], we hierarchically propagate the intermediate predictions based on the skeleton’s connectivity graph. The output of a sub-network is therefore concatenated with the input and fed to its children down the graph, allowing child nodes more information to refine the region of overlap with the parent. In addition, we train a final aggregation network to fuse the separate predictions of all sub-networks into the final SDF value $F(x,.)$. The overall architecture of our system is illustrated in Fig. 2.

Pose Conditioning: We next discuss how our networks are conditioned on pose. First, the input pose parameters $\theta_{j}$ (joint transformations) from a sequence $\{\theta_{j}\}$ are encoded into a latent representation $\phi_{j}$. Our encoder $\Phi$ is trained jointly with the model in an end-to-end fashion. Thus, $\Phi$ learns to project an input pose into a subspace of the plausible poses that the skeleton can deform into. Note that we ensure that pose parameters for all subjects are first transformed into a canonical global frame. Given that all the subjects share the same skeleton, and therefore are of comparable sizes, this has no negative effect on our model’s representational power. The entire pose embedding $\phi_{j}$ is passed to all of the sub-networks and the aggregation network as input, thus making all predictions conditioned on the global pose. This can help disambiguate shape for challenging poses with interaction between different parts of the skeletal graph. Unlike NASA’s deformable model [18], we employ the hierarchy of the skeleton for propagation and we do not try to learn to cluster the subject into deformable components.

Subject Conditioning: For multi-subject training, our networks are further conditioned on a learned subject embedding $\beta_{i}\in\mathbb{R}^{d}$. The subject codes $\beta_{i}$ are directly optimized during training. This follows the generative latent optimization (GLO) framework [7], where the model is effectively an autodecoder [42]. Similarly to the pose embedding, $\beta_{i}$ conditions all of the sub-networks and the aggregation network. Note that in order to evaluate on datasets with only one subject and to fairly compare to single-subject baselines, we dropped the subject codes from all the inputs to all our networks.

To train HIPNet, our method requires oriented 3D point clouds along with subject id and 3D pose parameters of the skeleton for each point cloud. We denote the pose parameters transformed to the canonical global frame with $\theta_{ij}$ for subject $i$ in frame $j$, and their encoding as $\phi_{ij}$.

We additionally process the input point clouds offline for approximate vertex-joint assignments. To this end, we process each point cloud individually, finding a proxy surface point on for every joint, and then estimating the approximate Geodesic distance between all points and the joints [16, 33]. Each point is then assigned to the closest joint. See Figure 3 for an illustration. This assignment is only needed at training time for an additional loss term and is not needed during evaluation. Please refer to Section 3.4 for details.

Weak Supervision: Unlike prior and concurrent work that requires uniform SDF or occupancy point samples to train an implicit model [42, 18, 41, 8], our method needs point samples only on the zero-crossing of the SDF, or the surface of the moving body. Such data is much more readily available, and in principle could be the result of raw capture.

In a manner similar to IGR [21], we utilize geometric initialization [2] to initialize the surface to a sphere and train our model to output a solution to the Eikonal equation [15]:

Loss Function: For each input point cloud $\mathcal{P}_{ij}$ with normals $n(x)\ \forall x\in\mathcal{P}_{ij}$, we train our model $F:\mathbb{R}^{3}\rightarrow\mathbb{R}$ with network parameters $w$ using the following loss function:

The data terms in the previous equation are the surface loss $L_{\text{surface}}$ and the normal loss $L_{\text{normal}}$, and are defined as:

The Eikonal regularization term $L_{\text{eikonal}}$ is defined as:

where we sample points uniformly during training within a canonical bounding box as well as near the surface to enforce the unit-norm constraint on the gradient of the SDF.

Sub-network Loss: The surface and normal losses denoted $L^{(k)}_{\text{surface}}$ and $L^{(k)}_{\text{normal}}$, respectively, are added to specialize the $N$ sub-networks to points close to their corresponding joints. The assignment of points to joints is estimated using an approximate geodesic distance on the input point clouds directly [16, 33]. In practice, we saw significant improvement over our metrics by encouraging this sub-network specialization. We ablate this choice in our experiments.

Estimating the geodesic distance on the input point clouds is only needed during training. During evaluation we simply feed our input points through the entire network. We observe that the sub-network that tends to have the highest impact on the final SDF value is the one corresponding to the joint closest to the input point.

In all our experiments, we set $\lambda_{\text{eikonal}}=0.1$ and $\lambda_{\text{normal}}=0.1$. The surface terms have no associated weight.

The subject codes are randomly initialized from $\mathcal{N}(0,10^{-4})\in\mathbb{R}^{256}$, a zero-mean Gaussian with $10^{-4}$ standard deviation, where the dimensionality of the subject latent space is $256$.

All our MLPs are of size $256\times 8$ with a single skip connection at layer 4, similar to [42]. We train on individual points clouds sampled at 25k points per batch. We sample another 25k points uniformly within the bounding box and near the surface for the Eikonal regularization term.

We train HIPNet using ADAM [27] with a fixed learning rate of $10^{-4}$. Our training runs for 50 epochs on 4 Tesla V100 GPUs for a little over 3 days.

We trained our model on various motion capture datasets. We used various datasets of minimally clothed humans from the AMASS motion capture archive [36], in addition to CAPE [35], the recent clothed human dataset.

• •

  Transitions is a single subject dataset with 110 motion sequences. We randomly selected 80 sequences for training and the remaining 30 for testing.

• •

  CMU¹¹1The homepage of the original dataset is at http://mocap.cs.cmu.edu. is a dataset of multiple subjects. Initially released as the Human Interaction subset of the CMU Motion Capture dataset, it was re-exported and added to AMASS. CMU contains over 100 subjects, but we dropped subjects with less than 5 motion sequences, leaving 89 subjects for our evaluation.

• •

  DFAUST [6] is another dataset that has 10 unique subjects performing varying motion sequences.

• •

  CAPE [35] is a dataset of real scans with 10 male and 5 female subjects in various clothing combinations. While the total number of subjects is 15, there are 41 unique subject-clothing combinations overall.

For the multi subject datasets, we select 75% of sequences for training and 25% for each subject so no subjects are unseen during inference, similar to recent work [38, 41]. Testing on unseen subjects is explored later in this section. We also randomly sample frames within each motion sequence when dealing with larger datasets. We randomly sampled 3% of the frames within each sequence for CMU and 50% of the frames in each sequence for CAPE. Overall with both train and test sets, there are over 80k, 80k, 30k, and 70k samples used in the Transitions, CMU, DFAUST, and CAPE datasets respectively.

For all these experiments we randomly sample 25k surface points alongside their normals on the input meshes. While all of these datasets rely on SMPL [34] for the consistent topology and the skinning weights, our approach is completely agnostic to both.

Metrics: We report the mean Intersection over Union (mIoU) and the Chamfer-L1 distance [20]. The mIoU is calculated using points randomly sampled within a tight bounding box around the subject which is stretched along the diagonal by 10% [18]. We also report the mIoU for near surface points for our method which has sampled by taking surface points from the ground truth mesh and then adding isotropic noise with $\mathcal{N}(0.0,0.03)$, a zero-mean Gaussian with $0.03$ standard deviation.

The Chamfer-L1 distance is the mean of an accuracy and a completeness metric. The accuracy metric is the mean distance of points on the reconstructed mesh to their closest point on the ground truth mesh, and the completeness metric is defined in the same way in the opposite direction. Similar to prior work [37, 20], we randomly sample 100k points from both meshes and use $1/10$ times the maximal edge length of the mesh’s bounding box as unit $1$ to normalize our distance estimation.

Baselines: We compare to the reported mIoU for various recent baselines, NASA [18], NPM [41], and SNARF [8]. We also reproduced NASA within our setup. In that setup, we reduced the number of surface and non-surface point samples to 25k from the original 100k, and we switched the surface point sampling strategy to triangle area-weighted uniform sampling instead of Poisson-Disc sampling.

Quantitative Results: We first compare our model, HIPNet, to NASA using the same point sampling strategy, the same number of points per mesh, and the same dataset split. We report the uniform mean IoU in Tab. 2. As shown, both our single and multi subject models outperform the deformable NASA model on all four datasets. Notably, NASA significantly underperforms on CAPE where subjects are the most diverse. It is worth noting that NASA requires skinning weights as additional input while our model (both variants) does not need any input beyond the point cloud and the 3D pose.

We next report our uniform and near-surface mean IoU on DFAUST and CAPE in Tab. 3 and Tab. 4, respectively. With the uniform IoU setting, all methods performed competitively. The surprising exception is NPM [41], which wile very competitive on CAPE, underperformed significantly on DFAUST. Our numbers are state-of-the-art on both datasets and in all metrics, outperforming all baselines.

Finally, we report the Chamfer-L1 distance on all four datasets in Tab. 5. We report the Chamfer-L1 distance scaled per-mesh such that $1/10$ times the maximal edge length of the mesh’s bounding box as unit $1$ [33, 37]. Our multi-subject model consistently outperforms the single-subject model on this metric.

Qualitative Results: A gallery of our extracted meshes for 5 unique subjects from DFAUST performing the same set of test-time poses is rendered in Fig. 4. By varying the subject code and maintaining the same pose we effectively perform motion retargeting for in-distribution subjects.

We next performed various ablations on DFAUST to evaluate the impact of various design choices on the reported metrics. We report our findings in Tab. 6 to motivate our final network design. The baseline model does not use the sub-network loss and aggregates the sub-network predictions in one step (i.e., a flat hierarchy). The biggest impact is clearly due to the sub-network specialization, significantly contributing to the final performance. The hierarchical aggregation and the latent codes further improve the performance as well.

Animation: Unlike various prior works, our model does not require a traditional morphable model during training. Our shape space is entirely learnt and is not based on a pre-trained shape space of SMPL [34] or GHUM [52]. This makes our model suitable for downstream tasks in animation and motion retargeting, where only the skeleton and 3D pose might be available. This also makes it suitable to animate characters or animals that cannot envelope a human skeleton.

Generating New Subjects: Following the Generative Latent Optimization (GLO) framework [7], we can fit a multivariate Gaussian with a full covariance matrix to our subject latent codes. We can then sample from that distribution new characters to showcase the diversity of the data and the representational power of the model. We showcase some samples from our model in Fig. 5.

Test Time Optimization: Similar to [41], we can apply test-time optimization to fit a subject code to an input point cloud, even for an out-of-distribution subject. This approach parallels traditional online fitting of shape space parameters, studied in earlier body and hand personalization work [5, 48, 50, 47]. We showcase this application in Fig. 6. The code is initialized to the mean of the subject codes and optimized for 100 iterations with a learning rate of $10^{-3}$. The mIoU and quality of the predicted mesh increase with the number of input points. More crucially, with only 1k points on the front of the mesh we are competitive with 10k points uniformly sampled everywhere. This specific settings simulates points coming from a front-facing depth sensor and is of practical importance.

One major limitation of our work stems from the inability of implicit models to represent open and thin surfaces. This is negatively impacting results on datasets with clothed humans, and generally thin extremities like fingers, tails, and so on. There is a recent interest in investigating the right representation for these surfaces [12].

Another question that is generally under-investigated in the field is the impact of modeling muscles on surface deformations. Our work in a way overcomes the limitations of LBS [30] and PSD [29] by directly learning to deform from the data. However, modeling the physics of muscles will likely yield a much more data-efficient approach that relies equally on priors from the physical world.