DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces

Let us first assume that point-wise correspondences are known for all pairs of shapes — to understand a simpler setting. In this case, we can specify a deformation as an offset vector for corresponding points. When the points are ordered in a consistent way so that corresponding points across the shapes have the same index in the order, we can consider a linear deformation function as an action $\oplus$ of the affine space:

where $\mathbf{A}\in\mathbb{R}^{3n\times k}$, with $k$ being the dimension of $V$. The deformation for a $v\in V$ is now explicitly defined as adding per-point offsets $\left(\mathbf{A}v\right)$ to the source point cloud, and the same $v$ produces the same offsets for the corresponding points across all the shapes. This action on shapes is free ($x\oplus\vec{0}=x$), transitive if $k$ is smaller or equal to $3n$, yet large enough to capture all possible differences of shapes in $X$, and also is in the additive group of $V$ over $X$. In the context of an autoencoder, this can be understood as constructing a latent shape space to be decoded to a linear subspace $\mathbf{A}$ of the shape point clouds, so that a free vector $v$ over that latent space describes the same point displacements in $3D$ space, regardless of the original shape. Alternatively, one can simply interpret $\mathbf{A}$ as a set of principal axes of the point clouds $x\in\overline{X}$, in the context of PCA.

Now we consider the case when the point-wise correspondences are unknown, and possibly even ill-defined. ¹¹1For a heterogeneous collection of shapes, particularly for human-made objects, the correspondences may not be clearly defined for some pairs, either geometrically or semantically. Hence, in this case, it is not possible to define a canonical linear deformation function for all shapes with a single matrix $\mathbf{A}$. Instead, we propose to predict a matrix $\mathbf{A}$ for each individual shape $x\in X$ using a neural network $\mathcal{F}:X\rightarrow\mathbb{R}^{3n\times k}$. By denoting the output for the input shape $x$ as $\mathbf{A}_{x}$, the action $\oplus$ is now written as follows:

The action is still free and transitive (with a large enough $k\leq 3n$) but is not in the additive group anymore, since now $\mathbf{A}_{x}$ is source-dependent; $\mathbf{A}_{x}$ and $\mathbf{A}_{y}$ for different shapes $x$ and $y$ may be inconsistent. To impose the property the additive action above, we jointly train a shape encoder $\mathcal{E}:X\rightarrow\mathbb{R}^{k}$ along with the above network $\mathcal{F}$ so that the deformation vector $\overrightarrow{xy}$ from shape $x$ to $y$ is given by $\mathcal{E}(y)-\mathcal{E}(x)$, while the matrix $\mathbf{A}_{x}$ is predicted as $\mathcal{F}(x)$. The deformation from the point cloud $x$ to $y$ is now computed as follows:

where the deformation vector is computed in latent space through $\mathcal{E}$, decoded into real space by $\mathbf{A}_{x}$ as a set of point offsets in an $x$-specific way, and added to the original point cloud for $x$.

This utilization of the two networks $\mathcal{E}$ and $\mathcal{F}$ realizes our overall plan: learning (i) a canonical affine deformation space from $\mathcal{E}$; and (ii) an individual linear (affine) deformation space for each shape from $\mathcal{F}$, and connecting these individual shape action spaces by sharing the same source latent space. The individual deformation spaces, defined by the matrices $\mathbf{A}_{x}$, are thus synchronized across the shapes via the connection through the canonical latent deformation space.

Empirically, even without the joint network training, the matrix $\mathbf{A}_{x}$ can be predicted consistently for similar input shapes due to the characteristics of neural networks when learning a smooth function — several recent works have proposed unsupervised methods finding shape correspondences based on this network property (Tulsiani et al., 2017; Groueix et al., 2018; Sung et al., 2018; Zhao et al., 2019; Li et al., 2019; Genova et al., 2019). However, we found that enforcing the property of the additive action regularizes the $\mathbf{A}_{x}$ matrices to become consistent even when the shapes are dissimilar (see Section 4.1 and Figure 5).

The matrix $\mathbf{A}_{x}$ can be interpreted as a dictionary of deformations (point-wise displacements), and in this context, we call the network $\mathcal{F}$ a deformation dictionary predictor in the rest of the paper. The idea of learning dictionaries of functions on shapes is analogous to the Deep Functional Dictionaries work by Sung et al. (2018), but there are two main differences. First, our dictionaries are learned from pairwise deformations, not from the input functions over the shapes (thus, out work is purely self-supervised). Second, we synchronize our dictionaries by explicitly learning a mapping of the dictionaries to a canonical space, instead of solely relying on the smoothness of the learned functions.

Let the input linear deformation function (defined by the deformation handles) denoted as $\mathbf{B}_{x}\left(z+z_{0}\right)$, where $\mathbf{B}_{x}\in\mathbb{R}^{3n\times m_{x}}$, $m_{x}$ is the number of deformation handles, and $z_{0}\in\mathbb{R}^{m_{x}}$ is the default parameters. We also have the learned linear deformation function for the shape: $\mathbf{A}_{x}v+x$ (Equation 2), where $x=\mathbf{B}_{x}z_{0}\in\mathbb{R}^{3n}$ is the initial shape and $\mathbf{A}_{x}\in\mathbb{R}^{3n\times k}$ gives a valid variation space of $x$. The goal of the projection is this: given the user input via the deformation handles $z\in\mathbb{R}^{m_{x}}$, we find $v\in\mathbb{R}^{k}$ (a valid variation) that minimizes the difference of point-wise offsets from the edited shape to the valid shape: $\left\|\mathbf{B}_{x}z-\mathbf{A}_{x}v\right\|_{2}^{2}$. In particular, we consider a shape editing scenario where the user edits the shape through some of the deformation handles and prescribes values for them at each time, and then the system automatically projects the edit to the valid space. Thus, given the equality constraints to the deformation handles $\mathbf{C}z=z_{\text{in}}$, where $\mathbf{C}\in\mathbb{R}^{m_{x}\times m_{x}}$ is a matrix indicating the selected handles and $z_{\text{in}}\in\mathbb{R}^{m_{x}}$ is the sparse user inputs, we find $\widehat{z}\in\mathbb{R}^{m_{x}}$ defined as follows:

When $\mathbf{B}_{x}^{\prime}$ denotes the matrix $\mathbf{B}_{x}$ except for the columns of the selected handles, Equation 4 can be written in a following form:

where $\widehat{z^{\prime}}$ is the best values for the rest of the parameters, and $c=\mathbf{B}\mathbf{C}z_{\text{in}}$. For any point cloud $y\in\mathbb{R}^{3n}$, the projection distance to the linear space of $\mathbf{A}_{x}$ is computed as:

where $\mathbf{A}_{x}^{\dagger}$ is a pseudoinverse of $\mathbf{A}_{x}$ (note that $\mathbf{A}_{x}\mathbf{A}_{x}^{\dagger}\neq I$ since $\mathbf{A}_{x}$ is a thin matrix, i.e., $3n\gg m_{x}$). Hence, Equation 5, again, can be rewritten as follows:

The solution of this linear regression $\widehat{z^{\prime}}$ is defined as:

where $\mathbf{P}_{x}=\left(I-\mathbf{A}_{x}\mathbf{A}_{x}^{\dagger}\right)\mathbf{B}_{x}^{\prime}$, $\mathbf{Q}_{x}=-\left(I-\mathbf{A}_{x}\mathbf{A}_{x}^{\dagger}\right)$, and $\mathbf{P}_{x}^{\dagger}$ is pseudoinverse of $\mathbf{P}_{x}$. Note that, when the user edits through one deformation at each time, $\mathbf{P}_{x}^{\dagger}$ and $\mathbf{Q}_{x}$ can be precomputed for each handle.

Some deformation handles may have inequality constraints; e.g., scale parameters must be positive numbers. The least squares problem in Equation 6 can also be quickly solved with inequality constraints using standard techniques such as interior-point methods or active-set methods.

At training time, we can also project the output deformed shape of the networks to the given deformation handle space to make the deformation describable with the given handles (the effect of this option is investigated in Section 4.5):

We remark that our framework can easily exploit any type of linear deformation handles, even when they are inconsistent and their numbers are different across the shapes.

While both our network and the others are trained to learn deformation from one shape to the other, we not only learn how to fit the input to the target but also discover the plausible variation space of the input shape, which is represented as a linear space with the deformation dictionary $\mathbf{A}_{x}$. Hence, in shape editing, when the deformation handles are given as linear deformation functions, the user’s input can be easily projected back to the learned variation space as described in Section 3.2 and also demonstrated in Section 4.1. This is an additional capability of our method compared to the other methods. Moreover, our deformation dictionary $\mathbf{A}_{x}$ captures strong correlations among the given deformation handles with additional regularization loss (see Equation 12), and thus provides more intuitive interpretation of the learned deformation space, as discussed in Section 4.1.

We also found that the factorization of shape offset representation into a source-dependent dictionary $\mathcal{F}(x)$ and a latent vector for a pair of shapes $\left(\mathcal{E}(y)-\mathcal{E}(x)\right)$ gives a better performance in the fitting. See Section 4.2 for quantitative evaluations. While Neural Cages (Yifan et al., 2020) have a similar factorization to ours, the others (3DN (Wang et al., 2019a) and Cycle Consistency (Groueix et al., 2019)) immediately combine the information of the pair to the source shape in the network without separately manipulating the source shape beforehand.

To attain the affine space properties in the latent space, in our framework, the deformation from one shape to the other is represented as the subtraction of two latent codes $\left(\mathcal{E}(y)-\mathcal{E}(x)\right)$, as described in Section 3.1. In all the other methods, however, the deformation is learned by first concatenating two latent codes and processing it in the next layers. In Section 4.2, we conduct an ablation study, changing the computation of the latent deformation vector in our framework, similarly with the other methods. The results demonstrate that our architecture enforcing the affine space properties produces more plausible shapes in the deformation transfer compared with the other methods and the architecture in the ablation study.

We experiment with five categories from the ShapeNet repository (Chang et al., 2015), namely Airplane, Car, Chair, Sofa, and Table. Most of the 3D models in ShapeNet are composed of smaller parts that have simpler geometry. Thus, we compute oriented bounding boxes for parts using the Trimesh library [Dawson-Haggerty et al.] and take translation and anisotropic scales along each local coordinate as our deformation handles. Figure 4 shows the number of shapes in each category (in parentheses), the distribution of the number of components in each shape (first row), and a sample object with part bounding boxes (second row). Before computing the bounding boxes, we run a preprocessing to merge small components to their closest neighbors and also combine overlapping components, as illustrated in the work of Sung et al. (2017). However, we do not group symmetric parts in order to have more freedom in editing each part. (Symmetries will be learned by our networks, as illustrated in Section 4.1 and  4.4.) After preprocessing, we take models with a number of component parts in the range of $[2,16]$. We do not normalize the scale of each shape individually to see natural shape variations (e.g., making a chair to a bench without decreasing the height), but all shapes are in the range of $[-0.5,0.5]$ in each axis.

We train the networks for each category. We sample $2K$ points randomly over each shape and feed the networks with random pairs of the source and target shapes. Training, validation, and test sets are randomly split with the 85-5-10 ratio. We set the dimension of latent space (i.e., the number of dictionary elements) $k$ to $64$ and the weight of sparsity regularization loss $w_{S2,1}$ to $10$.

Since we aim to deal with uncurated datasets where no manual annotations are present, we do not use any part labels in ShapeNet (unlike those used by Mo et al. (2019a)) or any correspondence supervision in network training. Also, such part labels are often insufficient to infer correspondences across the deformation handles.

We illustrate the learned deformation dictionary $\mathbf{A}_{x}$ in Figure 5. Each column visualizes shapes deformed along some elements in the dictionaries. Thanks to the sparsity regularization loss (Equation 12), the elements show local and natural deformation modes and correlations among the given deformation handles. For example, the first and third columns of chairs translate and scale the seat along up and front directions, respectively, while preserving connectivities with the other parts. The last column also elongates the back part upward. Similar local deformations are also observed in sofas. Interestingly, the second column scales only swivel leg (red circle), and it does not affect the shape with a different type of leg. Similarly, in the second column of tables, the element translates the shelf in the middle along the up direction and does not make any changes if the shape does not include the shelf. The supplementary video shows how the shapes vary along the dictionary elements in animations.

We also describe how our framework can be employed for interactive shape co-editing and demonstrate qualitative results. We consider a scenario that the user modifies a source shape through one of the given deformations handles at each step. Given the user input, the system (i) first automatically snaps the edited shape to the plausible shape space while fixing the modified parameter as described in Section 3.2. When solving the least square in Equation 6, we use a constraint that all scale parameters should be greater than zero. Then, (ii) the projected deformation is transferred to the desired new source shape in the same category.

Figure 1 and 6 show some examples of the deformation projection and transfer. In Figure 6, given a source shape (first column), we randomly select one of the deformation handles and perturb the parameter (second column). If the selected handle is a translation along one of the local coordinates of a box (blue arrow in the figure), we vary the parameter in the range of [$-0.5+t$, $0.5+t$], where $t$ is the given number. If the select handle is a scaling (red arrow in the figure), we pick a random number in the range of [$0.5s$, $1.5s$], where $s$ is the default scale. Then, the modified shape is projected to the learned plausible shape space (third and fourth column). Given another source shape (fifth column), the projected deformation is transferred (sixth and seventh columns) by taking the latent deformation vector and applying it to the new source shape with the learned shape-dependent action. The results show that the projection adjusts the rest of the parameters in a way to maintain the shape structure such as symmetry and part connectivity. For instance, right-wing of the airplane (first row) and a leg of the chair (third row) and the table (last row) are properly attached to the other parts after the projection while preserving symmetry. Also, the deformation on the source shape is naturally transferred to the new source shape even when the geometry or the deformation handles are different. For example, the tables in the last row have different part structure, but the translation of legs is naturally adopted to the new shape as decreasing the width. Refer to the supplemental video for more examples.

We analyze the effect of sparsity regularization loss in Equation 12 by varying its weight. Figure 7 (a) shows the mean fitting error (Chamfer distance) of $1k$ random chair pairs when varying the weight from $0.1$ to $10$. While the fitting error increases with a larger weight, the errors with large weights are smaller than the ones of baseline deformation methods (see Section 4.2). In Figure 7 (b) and (c), we show how many non-zero elements and columns we obtain in the projected dictionaries $\mathbf{B}_{x}^{\dagger}\mathcal{F}(x)$ when increasing the weight and also varying the threshold of zero. The numbers of non-zero elements and columns dramatically decrease, indicating that the network discovers more strongly correlated deformation handles, while still learning all possible deformations.

For the quantitative evaluations, we use the same preprocessed dataset of ShapeNet with Groueix et al. (2019). This dataset contains five categories: Airplane, Car, Chair, Lamp, and Table. The difference of this dataset with the one used in Section 4.1 is that it normalizes each point cloud to be fit in a uniform cube, which size is $2$ for each axis. All networks including ours and baselines are trained as described in Section 4.1, by taking $2k$ random sample points in each shape and feeding random source-target shape pairs in each category. To maximize the fitting capability, in this experiment, we set the dimension of the latent space in our framework, $k$, to 512 and disable sparsity regularization loss ($w_{S2,1}=0$). Also, in our framework, we do not use the projection to the deformation handle space described in Equation 8 during the network training, and the effect of the projection is analyzed in Section 4.5.

We compare our method with non-rigid ICP by Huang et al. (2017) and three recent neural-network-based deformation methods mentioned in Section 3.4: 3DN (Wang et al., 2019a), Cycle Consistency (CC) (Groueix et al., 2019), and Neural Cages (NC) (Yifan et al., 2020). We take point clouds as input to all networks and do not use the differentiable mesh sampling operator introduced in 3DN (Wang et al., 2019a), which can be easily plugged into any of the networks. All the baseline neural-net-based methods have a part in their architecture to compute deformation (i.e. shape difference) from the given source and target shapes and apply it to the source shape. Thus, we implement the deformation transfer of the other methods as applying the given source-target pair information to the other source shape, similarly with our method.

We test the impact of enforcing affine space properties by modifying the part of computing latent deformation vector in our networks. As mentioned in Section 3.4, instead of taking the difference of two latent codes $\left(\mathcal{E}(y)-\mathcal{E}(x)\right)$, we concatenate latent codes of source and target shapes and process it to produce the deformation vector ³³3Within the PointNet (Qi et al., 2017) classification architecture, we concatenate global features of the shapes produced from the max-pooling layer and then pass it through the same next MLP layers. The final output replaces $\left(\mathcal{E}(y)-\mathcal{E}(x)\right)$ in our original architecture..

We compare the methods in two aspects: fitting accuracy of deformations and plausibility of deformation transfer results.

For the fitting accuracy, we perform target-driven deformation, deforming a source shape to fit it to a target shape, with random $1,000$ pairs of source and target test shapes in each category. Then, we measure the average Chamfer distance between the target and the deformed source shape (Fitting CD). Also, inspired by Groueix et al. (2019), we evaluate the fittings by finding the closest points from the deformed source shape to the target and measuring mean Intersection over Union (mIoU) of semantic parts provided by Yi et al. (2016). This metric demonstrates how beneficial each method at unsupervised co-segmentation or few-shot segmentation.

For the plausibility of deformation transfer results, the quantitative evaluation is challenging since it is not precisely determined without involving human perception. In our evaluation, we follow the ideas of Achlioptas et al. (2018) evaluating generative models. Achlioptas et al. introduce two data-driven evaluation metrics: Minimal Matching Distance (MMD-CD) and Coverage (Cov-CD). Minimal Matching Distance is the average of the Chamfer distance from each generated shape to its the closest shape in the reference dataset, indicating how likely the output shape looks like a real shape. Coverage is the proportion of the shapes in the reference dataset that are closest from each generated shape, showing how many variations are covered by the generated shapes. For these two, we randomly choose $200$ pairs of source and new source shapes, and for each of the $10$ target shapes, we transfer the source-to-target deformation to the new source shape. Then, we measure the metrics by taking the training set as the reference dataset; the averages for all target shapes are reported.

We report all results in Table 1. Bold is the best result, and underscore is the second-best result. Note that our DeformSyncNet is the only method that shows outstanding performance both in the fitting accuracy and in the plausibility of deformation transfer results. For the Fitting CD and mIoU, DeformSyncNet gives better accuracy in most of the categories compared with the other methods. The network in the ablation study, concatenating two latent codes, is the only one that shows the better fitting accuracy than DeformSyncNet in terms of Fitting CD, but its performances in other metrics are poor, particularly in MMD-CD and Cov-CD. For the MMD-CD and Cov-CD, our DeformSyncNet also outperforms the other methods. The only competitor is Neural Cages (Yifan et al., 2020), but it gives inferior accuracy of the fitting, as shown in Fitting CD.

Figure 8 illustrates some results of quantitative evaluation. Our DeformSyncNet shows the best fitting results in most of the cases and also captures well the difference between the source and target shapes in the transfer. For instance, the source and target chairs in the fifth row have differences in the width and height of the back part, and these differences are correctly transferred to the new shape, which has a different global structure. Also, in the sixth row, ours combines chair arms of the source shape to seat in the deformation, and this change is properly transferred to the new shape. The other methods tend not to transfer the difference but to do target-oriented deformation, making the new source shape close to the target shape.

One of the beneficial properties of affine space is the path invariance, producing the same deformation regardless of the pathway from the starting point to the destination. This property is desirable in the downstream applications since it allows the user to explore the latent shape variation space freely without specifying the order of steps. We test this property by taking a random parallelogram in the latent space. A triplet of shapes $x$, $y$, and $z$ are given, and deformations are transferred in two ways: $\overrightarrow{xy}$ to $z$ and $\overrightarrow{xz}$ to $y$ (see inset). Then, we verify whether these two results are the same by measuring Chamfer distance.

The quantitative results comparing with other methods are reported in Table 2. Our DeformSyncNet gives much smaller differences between two pathways of deformations compared with the other methods. Cycle Consistency (Groueix et al., 2019) using regularization losses for the consistency provides the second-best results in all categories, but its fitting distance is much larger than ours. Figure 9 illustrates some of the qualitative results.

The dataset of Schulz et al. (2017) contains 74 parametric models, each of which has its own deformation parametrization. For each model, we uniformly sample $2k$ points and generate $1k$ variants with random parameters; $128$ shapes out of them are used as the test set. The main difference of this dataset with ShapeNet in Section 4.2 is that point correspondences are known across the shapes since they are generated from the same parametric models. The point correspondences are not used as supervision in any experiment, but will be used as ground-truth in evaluations. During training, we do not project the deformed shape to the given deformation parameter space (see Equation 8).

As discussed in Section 3.3, we empirically verify the effect of our network design enforcing consistency in the deformation, compared with Cycle Consistency in Groueix et al. (2019) leveraging dedicated loss functions for the consistency and also the other baseline methods. We train the networks for the shapes of each parametric model and measure the two-way consistency error for a point cloud pair ($x$, $y$) as follows:

where $x_{i}\in\mathbb{R}^{3}$ and $\left(d_{i}(x\rightarrow y)-x\right)\in\mathbb{R}^{3}$ are $i$-th point of the point cloud $x$ and its new position after deformation toward $y$, respectively, $n$ is the number of points, and here we assume that all point clouds are ordered consistently based on the given point correspondences. We computed the mean of this two-way consistency error for 50 pairs, which are the first 50 of the largest Chamfer distances among $1k$ randomly generated pairs. Figure 10 (a) shows a comparison between the results of our method and the baseline methods. Each dot indicates the consistency error for a single parametric model, and the x-axis and y-axis are for our DeformSyncNet and the other methods, respectively. Our DeformSyncNet gives smaller two-way consistency errors than the other methods including Cycle Consistency in most cases: 70, 58, and 71 cases out of the 74 models compared with 3DN, Cycle Consistency, and Neural Cages, respectively.

Using the given deformation parametrization, we also quantitatively evaluate the performance of deformation transfer. If we choose all of the source, target, and destination (the new shapes to transfer deformation) shapes from the same parametric model, the ground truth of the deformation transfer can be computed by transferring the difference of parameters from source to target to the destination shape. Given the 50 test pairs per model above, we randomly pick another shape as the destination and compute Chamfer distance between the predicted and the ground truth shape. Figure 10 (b) illustrates the mean Chamfer distance; x-axis is DeformSyncNet, and y-axis is the other methods. In most cases, DeformSyncNet gives a magnitude smaller distances compared with the other methods. The numbers of cases out of the 74 models when DeformSyncNet outperforms 3DN, Cycle Consistency, and Neural Cages are 74, 69, and 73, respectively.

We further assess the quality of deformation transfer results by taking the destination shape from a different parametric model (but in the same category) with the source and target shapes. Since we cannot compute the ground truth in this case, we conducted a user-study on Amazon Mechanical Turk. Among the 74 parametric models, we grouped 8 Airplane, Chair, and Table models per category and trained all the networks for each group, while still taking the source and target shape from the same parametric model. But in the test time, the destination shape is randomly selected from the other model in the same group — note that we take the source and target shapes from the same parametric model so that the participants can clearly identify the difference. From the same 50 test pairs per model above, in total 1,200 ($50\times 8$ models $\times 3$ groups), we randomly select $10\%$ ($120$ pairs), assign the random destination shape, and create user study questions as shown in Figure 11. For each question, we ask human subjects (Turkers) to select at least one among four shown objects: the outputs of 3DN, Cycle Consistency, Neural Cages, and our DeformSyncNet. The associations between the methods and the objects are hidden to the Turkers, and the order of the objects is randomized. The Turkers are also encouraged to choose multiple objects if they think more than one options are equally good. In total, we collected 1,200 responses from a pool of 100 different participants (each question was answered by 10 distinct Turkers).

In the results, our DeformSyncNet was selected in $54.7\%$ of all 1,200 responses, whereas 3DN, Cycle Consistency, Neural Cages were selected in $30.9\%$, $4.0\%$, and $46.6\%$, respectively. (Note that we allow multiple choices, and thus the sum of percentages is greater than 100.) The performance gap between DeformSyncNet and Neural Cages (the second best) is statistically significant as the McNemar’s contingency test (McNemar, 1947) between the two empirical (binomial) distributions has a p-value of $5e-4$ for the null hypothesis of that the two distributions have an equal marginal. We also remark that for each of the 10 responses collected for each question, DeformSyncNet was the most preferred one among all methods in 59 out of 120 questions, and also it tied up with the other most preferred one(s) in 17 questions.