Invertible Neural Skinning

To animate our subject from their canonical to deformed pose we use Linear Blend Skinning (LBS), which involves deforming the canonical surface according to a convex combination of rigid bone transforms. Specifically, we use the differentiable LBS formulation from SNARF [11] and summarize it below.

Canonical Weight Field. We define a learnable weight field in canonical space parameterized by a neural network, $\mathbf{w}_{lbs}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{n_{b}}$. For a given point in canonical space, this weight field predicts the blend weights corresponding to each bone:

To make weights ($w_{i}$) convex for LBS, they are constrained to be always non-negative and sum to 1 using softmax.

LBS. Given the above weight field and the relative body pose as bone transforms $\boldsymbol{\theta}^{t}=[\boldsymbol{B}_{1},\dots,\boldsymbol{B}_{n_{b}}]$, we can forward warp any point $\mathbf{q}_{c}$ of our canonical space to deformed space using Linear Blend Skinning as follows:

where $\mathbf{q}^{t}_{d}$ represents the corresponding point in deformed space where $\mathbf{q}_{c}$ lands after LBS.

Searching Canonical Correspondences. While training on raw scans, we are only provided with points in deformed space. To find their possible correspondences in canonical space, we solve for the roots of Equation 3 using an iterative solver while keeping $\mathbf{w}_{lbs}$ constant. Specifically, we use Broyden’s Method [7] to find a set of $\{\mathbf{q}_{c}^{1},...,\mathbf{q}_{c}^{K}\}$ point correspondences for each deformed point $\mathbf{q}^{t}_{d}$ by initializing the root-finding algorithm at $K$ different points in the canonical space.

Differentiable Skinning. The above formulation is end-to-end differentiable as it is possible to compute the gradients of the weight field $\mathbf{w}_{lbs}$ with respect to input point $\mathbf{q}^{t}_{d}$ via implicit differentiation as shown in SNARF [11], Section 3.4. In this work, we also extend these derivations to compute the gradient of correspondences $\mathbf{q}_{c}^{i}$ w.r.t. input points. For more details, please refer to Appendix D.

Why is LBS insufficient? The above differentiable formulation suffers from the same limitations of traditional LBS, such as being unable to represent the clothed surfaces, and introducing volume loss, as shown in Figure 6, b). This is especially problematic when learning from real-world data of clothed humans in various poses.

Invertible networks [16, 15, 26] are bijective functions composed of modular components called coupling layers, which preserve 1-1 correspondences between their input and output. In this section, we describe the construction of our proposed pose-conditioned coupling layer, which is chained together to construct a PIN.

2D Coupling Layer (Figure 3). A coupling layer operates by splitting its input into two parts using a fixed breaking pattern. After splitting, the first part of the input is transformed by applying a sequence of invertible operations, such as translation and rotation. The parameters for these operations can be produced by any arbitrary function that is jointly conditioned on the second part of the input and an external conditioning, such as pose in our case.

Formally, as we operate in 3D space, let the input point be defined as $[x,y,z]$, and the input splits are $[x,y]$ and $[z]$. Then the 2D coupling layer $\mathbf{G}^{xy}([x,y,z],\theta^{t})$ defines an invertible transformation as follows:

where $\mathbf{R}_{xy}\in\mathbb{R}^{2\times 2}$, and $[t_{x},t_{y}]\in\mathbb{R}^{2}$ is a rotation matrix and translation vector produced by any arbitrary function that takes as input only the bone pose $\theta^{t}$ and the coordinate $z$. The inverse $\mathbf{G}^{-1}_{xy}([x,y,z],\theta^{t})$ of the coupling layer can be computed easily by:

We describe computation of operation parameters $\mathbf{R}_{xy}$ and $[t_{x},t_{y}]$ next.

Pose Embedding. We encode every bone transform in pose $\theta^{t}$ using a MLP $\mathbf{m}_{b}$ which takes a 6D input of concatenated 3D translation and rotation (as Euler angles). To obtain pose embedding, we concatenate the outputs of each bone $\mathbf{e}_{\theta}$ as follows:

Space Embedding. We use SIREN [54], a learned and periodic positional encoding, to map the spatial coordinates denoting it as

We find that this helps to better represent high-frequency surface details such as cloth wrinkles.

Space and Pose Aware Conditioning (Figure 2). We observe that when the relative pose $\boldsymbol{\theta}^{t}$ between deformed and canonical spaces is zero (i.e. $\boldsymbol{B}_{i}=[\mathbf{I}|\mathbf{0}]$, all bone transforms have identity rotation and zero translation), the coupling layer should not introduce any space-varying (i.e. $z$-conditioned) changes.

To enforce this, we take the Hadamard product of the space and pose embeddings, and subsequently concatenate them obtaining

We visualize the construction of our space and pose aware conditioning $\mathbf{e}_{sp}$ in Figure 2.

Rotation and Translation Maps. Finally, to produce parameters for coupling operations, we use two MLPs $\mathbf{m}_{t}$ and $\mathbf{m}_{r}$, which take as input the above conditioning vector:

Note that the output of $\mathbf{m}_{r}$ only predicts the angle of rotation $\gamma_{xy}$ in radians (a single value). The axis of rotation passes through the origin of the split input space, i.e. $\mathbf{XY}$ space here. We convert $\gamma_{xy}$ into a rotation matrix $\mathbf{R}_{xy}$.

1D Coupling Layer. Unlike the 2D coupling layer described above, we cannot use the rotation operator in 1D, and in this case, we only use translation. For layer $\mathbf{G}^{x}([x,y,z],\theta^{t})$ with split pattern as $[x]$ and $[y,z]$ the coupling operation becomes:

where $t_{x}\in\mathbb{R}^{1}$ produced in a similar fashion as a 2D coupling layer using a translation map $\mathbf{m}_{t}:\mathbb{R}^{2d}\rightarrow\mathbb{R}^{1}$ with single scalar output instead of 2D translation. The space embedding of Equation 8, in the 1D case, takes both coordinates as input $\mathbf{e}_{xy}=\boldsymbol{\Phi}([x,y]):\mathbb{R}^{2}\rightarrow\mathbb{R}^{d}$.

Pose-conditioned Invertible Network (PIN). Finally, we compose our PIN by chaining together multiple 1D and 2D pose-conditioned coupling layers as:

where $\mathbf{p}$ represents point in 3D space, $\mathbf{G}_{i}$ represents a coupling layer, and $\boldsymbol{\theta}^{t}$ represents pose. Inverting PIN is simply equivalent to sequentially inverting each coupling layer in the reverse order:

Since the PIN is invertible by construction, it preserves exact correspondences between its input and output spaces:

We visualize a single coupling layer of PINin Figure 3.

Overview (Figure 4). Our overall posing pipeline INS is comprised of three previously described components chained together:

• •

  $\mathbf{H}_{c}$: A Pose-conditioned Invertible Network (PIN) $\mathbf{H}_{c}$ that operates after canonical space and before LBS.

• •

  $\mathbf{H}_{d}$: A Pose-conditioned Invertible Network (PIN) that operates before deformed space and after LBS.

• •

  Differentiable LBS network as described in Section 3.1 operating between above PINs.

Next, we discuss how we formulate an invertible mapping that preserves correspondences between deformed and canonical spaces.

Deformed to Canonical (Training). For any point $\mathbf{p}^{t}_{d}$ in deformed space, we first process it using PIN $\mathbf{H}_{d}$, and obtain $\mathbf{q}^{t}_{d}$. Next, we use Broyden’s algorithm to get correspondences of $\mathbf{q}^{t}_{d}$ in canonical space, let’s say $\{\mathbf{q}^{i}_{c}\}_{i=1}^{K}$. Finally we use a second PIN $\mathbf{H}_{c}$ to map these points $\{\mathbf{p}_{c}\}_{i=1}^{K}$ in the pose-independent canonical space.

To obtain the most suitable canonical correspondence, we take the $\operatorname*{arg\,max}$ over all predicted canonical occupancies

During training, we approximate the $\operatorname*{arg\,max}$ with a $\mathrm{softmax}$ function in order to backpropagate gradients softly through all correspondences following SNARF.

Training Objective. In our dataset, we are given points in deformed space and corresponding ground truth occupancy values of zero or one. We map these deformed points to canonical space and apply binary cross-entropy loss to jointly train all components of the posing network according to

Auxiliary Objectives. Following SNARF, we enforce a prior on the canonical pose by using two additional losses during the first epoch. First, we sample additional points on bones in canonical pose and encourage their occupancies to be one. Second, we encourage the skinning weight of bone joints to be equal. However, no ground truth skinning weights are required during these steps.

Canonical to Deformed (Inference). Once trained, we can animate characters using INS in any given novel pose $\boldsymbol{\theta}^{n}$ in two simple steps. First, running mesh extraction on the canonical occupancy network. Second, reposing the mesh vertices via an inverse pass of our posing pipeline as follow:

Fast Reposing. As our canonical occupancy network $\mathbf{O}$ is independent of $\theta^{n}$ we only have to extract mesh exactly once. And reposing this mesh for a sequence of poses simply becomes equivalent to performing multiple inferences described in Equation 19.

Datasets. Training our method requires sampled points in the deformed space, along with corresponding occupancies and poses. Thus, we benchmark INS on two datasets CAPE [34], which features scanned humans in loose clothing, and DFAUST [35], containing only minimally clothed human scans.

CAPE [34] contains scans of 15 subjects (8 males and 7 females), wearing 8 different types of garments while performing a large number of actions. These actions were recorded using a high-resolution body scanner (3dMD LLC, Atlanta, GA), and the scans were registered using an SMPL model [31]. Similarly to SNARF [11], INS requires training a new model every subject-cloth pair, and exhaustively training on every combination quickly becomes expensive. To manage computational costs, we use a subset of 15 sequences. This subset covers all garment types at least once and most of the subjects, thus capturing variations in both — body shape and clothing.

DFAUST is a subset of the AMASS [35] dataset consisting of 10 subjects who are minimally clothed. Each subject is scanned similarly to CAPE while performing 10 different actions. As the subjects wear minimal clothing, much of their motions can be represented accurately by rigid body transformations. As a consequence of little subject clothing, we observe that DFAUST contains significantly fewer pose-specific deformations. Thus we note that DFAUST dataset is not well-posed to test the true capabilities of INS.

Data Splits. For a given subject in DFAUST or a subject-clothing pair in CAPE, we are provided with multiple temporal sequences, each containing a different action. We divide these sequences in 9:1 ratio into train and test sets. This split is similar to SNARF [11]. More details about training sequences and garment types are provided in Appendix B.

Metrics. Following SNARF [11], we report the mean Intersection-over-Union of points sampled near the mesh surface (IoU surface), and of points sampled uniformly in space (IoU bbox).

SNARF-NC. We use SNARF [11] without pose-conditioning in the canonical occupancy network as our first and primary baseline. For this, we only remove the pose-conditioning used by SNARF such that canonical space becomes pose-independent, i.e. $\mathbf{O(\mathbf{p}_{c})}$. We do not make any other changes. This setting is comparable to INS as it allows for fast posing and preserves correspondences across different poses.

SNARF. We also compare INS to the original SNARF [11] which uses a pose-conditioned occupancy network, i.e. $\mathbf{O(\mathbf{p}_{c},\mathbf{\theta}^{t})}$. However, we point out that the above pose-conditioned occupancy comes at the sacrifice of fast posing, by requiring expensive mesh extraction for each new pose while not preserving correspondences across them. These disadvantages make the direct comparison between INS and SNARF based solely on their performance a little lopsided.

To obtain SNARF and SNARF-NC results, we use the official codebase ²²2https://github.com/xuchen-ethz/snarf released by the authors.

AVG-LBS. In addition to the above strong learned baselines, we provide results on two simpler baselines, which use the SMPL-fitted LBS weights to unpose the meshes (scans) using forward skinning. For this, we simply take an average of all the canonicalized training meshes to generate a final canonical mesh and deform it to any unseen given pose using Foward LBS and SMPL weights.

FIRST-LBS. This baseline is similar to the AVG-LBS baseline described above and uses SMPL-fitted weight for reposing. Instead of using an average across all training meshes, it only uses the first mesh, thus containing lesser pose-conditioned details.

CAPE. We demonstrate the results of INS on clothed human data in Table 1. Given the challenging nature of modeling cloth deformations contained in this dataset, we find that INS surpasses SNARF-NC (without pose-conditioning) on average by +6.24% and +6.41% absolute percentage points in Surface IoU and Bounding Box IoU respectively. Moreover, INS also outperforms vanilla SNARF with pose-conditioning by +0.89% and +1.02% absolute percentage points in Surface IoU and Bounding Box IoU, respectively, while also enjoying the benefits of fast posing, and matched correspondences across various poses. We observe that the simple aggregation baseline of AVG-LBS performs quite closely with SNARF-NC with a performance drop of only 1.88% and 1.66% percentage points between them. However, AVG-LBS benefits from using a strong prior of parametric SMPL model and corresponding fitted weights.

DFAUST. We also report results on much simpler minimally clothed humans from the DFAUST dataset in Table 2. We find that INS outperforms SNARF-NC (without pose-conditioning) on average by +3.37% and +0.63% absolute percentage points in Surface IoU and Bounding Box IoU metrics, respectively. When compared to SNARF with pose conditioning, we find INS lags behind by -1.42% and -0.86% absolute percentage points in Surface IoU and Bounding Box IoU metrics, respectively. Given the minimal clothing and few pose-conditioned non-linear effects in DFAUST, we hypothesize that this performance drop can be attributed to SNARF overfitting easily to this benchmark. We believe this result also reflects the importance of testing on many realistic datasets such as CAPE.

Timing Study. In Figure 5, we compare the times taken by SNARF and INS to repose a clothed character across a sequence of 125 different poses. A single mesh extraction pass with MISE [36] operating on the cube of resolution $128^{3}$, takes nearly $1.5$ seconds. While reposing, SNARF performs this operation for every given pose, whereas INS requires mesh extraction only once. Reposing the extracted mesh INS takes $0.13$ seconds for an inference pass, which is an order of magnitude faster than SNARF.

We perform numerous ablations of our INS setup, the results of which are summarized in Table 3.

Multiplying Pose and Space Embeddings is important. Reformulating the pose conditioning by simple concatenation, i.e. $[\mathbf{e}_{z},\mathbf{e}_{\theta}]$ instead of multiply-then-concatenate, i.e. $[\mathbf{e}_{z},\mathbf{e}_{z}\odot\mathbf{e}_{\theta}]$ leads to a significant performance drop of $\sim$ 11% points in both metrics (Row 2).

$\mathbf{H}_{c}$ contributes much more than $\mathbf{H}_{d}$. The invertible networks do not contribute equally to the performance. Removing the canonical space PIN $\mathbf{H}_{c}$ leads to a sharp drop in IoU Surface by 4.94%, when compared to removing the deformed space INN $\mathbf{H}_{d}$ which drops performance by only 0.17% points (Rows 5,6). We attribute this partly to the fact that editing an LBS-deformed mesh introduces the additional complexity of resolving part-wise correspondences, such as locating the new positions of joints and limbs.

Replacing SIREN positional embedding with MLP hurts. IoU Surface drops by 3.16% if the learned sinusoidal embeddings are replaced by simple MLP layers. This happens as fine surface details such as cloth wrinkles get blurred when using MLP, which is prevented by SIREN.

PINs without rotation perform slightly worse. Removing 2D operations from our PINs leads to a drop in IoU Surface by 0.92%. This happens because twisting deformations in the surface have to be represented by only displacements, which has previously been shown to be difficult to learn [42].

Simply using PINs without LBS performs worse. Entirely removing the differential LBS module and relying solely on PINs to capture the full articulate motion results in a huge drop of $32$% on both metrics (Row 7).

INS can represent finer details compared to SNARF. In Figure 6, we demonstrate a subject in a challenging novel pose from the CAPE dataset. We find that SNARF (leftmost) is unable to capture fine details of cloth wrinkles, while also missing fingers as highlighted by the markers. Whereas SNARF-NC (second left) struggles with LBS artefacts such as volume loss by shrinking the arched back (highlighted), and displaying candy-wrapper effects. Finally, INS (second right) is able to capture much sharper local details around the body joints, such as around the waist and neck.

PINs can represent pose-varying deformations well. In Figure 8, we tease apart the edits made by solely PIN $\mathbf{H}_{c}$ in the canonical space (displayed in the top row) given two unseen target poses (shown in the bottom row). As highlighted in the figure, we find that PIN learns to introduce pose-varying deformations such as raising cloth outlines around the neck and shoulder joint, introducing dress wrinkles at near extremities, and even adjusting limbs such as orienting feet.

Extreme poses from PosePrior. The most loose-fitted sequence in CAPE is of subject 00375 wearing a blazer and trouser, we show it animated in extreme poses from Pose Prior in first two columns of Figure 7. We also visualize other subjects (both clothed and naked) in extreme poses as well. We find that INS produces much more realistic cloth deformations compared to SNARF.