Meta Deformation Network: Meta Functionals for Shape Correspondence

Dynamic Neural Networks are a design in which one network is trained to predict the parameters of another network in test-time in order to give the second network higher adaptability. The possibility of using dynamic networks on 2-D and more recently 3-D data has been researched by several of authors. [17], [9], [1], [3] applied dynamic convolutional kernels to 2-D images to input-aware single-image super-resolution, short-range weather prediction, one-shot character classification and tracking, few-shot video frame prediction, respectively. These tasks demanded highly adaptive behaviors from the neural networks, and they show improvements when using dynamic neural networks over conventional methods that used static-parameter convolutional kernels. [10] was one of the first works to apply the idea of dynamically generated parameters to the 3-D domain, where it achieved state-of-the-art results on single-image 3D reconstruction on ShapeNet-core [4] compared to conventional methods in [20], [6], [16]. We are motivated by these works to further investigate the potential of dynamic neural networks. We build on previous studies by examining the effectiveness of dynamic neural networks for 3-D shape correspondence in which both the input shapes and output correspondence are of a 3-dimensional nature.

The registration of and correspondence among 3-D structures with non-rigid geometrical variations is a problem that has been extensively studied and one in which machine-learned based methods have had great success. MPI-FAUST [2] is a widely used benchmark for finding correspondence between pairs of 3-D scans of humans with realistic noises and artifacts as a result of scanning real-world humans. There are various approaches to the task. [10] used a Siamese structure to learn compact descriptors of input shapes and using functional maps [13] to solve for inter-shape correspondences. [21] fitting query meshes to a pre-designed part-based model (”stitched puppet”) and aligning the models for the two query scans to get correspondence. [7] took a simpler approach, by choosing a set of points to be a template and getting inter-shape correspondences by computing the query shapes’ individual correspondences to the common template, effectively factoring the problem of computing correspondence between two unknown shapes into two problems each involving the correspondence between a known template and an unknown shape. This method proved to give the best correspondence accuracy on FAUST and is the inspiration for the training procedure of our method, the Meta Deformation Network. [7] obtains correspondence between a template and a query shape by using a multi-layer perceptron $g$ to deform the points on templates into the shape of the query and obtaining correspondence by Euclidean proximity between the deformed points of the template and actual points on the query shape. However, $g$ holds fixed parameters and takes a latent vector concatenated with template points as input. By contrast, our Meta Deformation Network has a decoder $g$ that holds dynamic parameters, which adds more flexibility and adaptability of the deformation with respect to different query shapes.

A graphical description of the Meta Deformation Network is shown in fig. 1. We used a simplified version of the PointNet [15] encoder following [7], which we will denote as $f$, to output a feature vector $\mathbf{E}$ of size 1024 from a set of points from the query shape $\mathbf{S_{q}}$. L independent MLPs, each denoted $h_{i}$, then $\mathbf{E}$ takes as input outputs $\hat{\theta_{g}}_{i}$, the predicted optimal parameters for layer $i\in\{1,...,L\}$ of the meta decoder $g$. If layer $i$ of $g$ has $K_{in}$ input neurons and $K_{out}$ output neurons, then $\hat{\theta_{g}}_{i}$ is a vector of size $(K_{in}*K_{out}+K_{out}+K_{out})$ since we define the forward-pass for each layer of g as

where $*$ denotes element-wise multiplication, and $x_{out}$ is the pre-activation output. Note that we place a scaling term $s$ in addition to the typical feed-forward calculation of an MLP with the intuition that it facilitates the learning of $h$ (reasoning behind this intuition is given in 3.5).

With $\hat{\theta}_{i}$ computed, the meta decoder takes as input a point $\mathbf{p_{t}}\in\mathbf{S_{t}}$ outputs a 3-D residual vector $\Delta(\mathbf{p_{t}})$ that will shift $\mathbf{p_{t}}$ to the location of the corresponding point on the query shape $\mathbf{S_{q}}$. Note that the input layer to $g$ is only a vector of 3 elements, which leads to a major speed up over a decoder that uses the LVC input, which take in a vector of size $1024+3$ if using an $\mathbf{E}$. The difference in time is greater as the resolution of the template increases because the deformation computation is repeated $N$ times to get the respective $\Delta(\mathbf{p_{t}})$ for all $\mathbf{p_{t}}\in\mathbf{S_{t}}$.

In all experiments, we pick $g$’s structure $L=6$ and $K_{out}=64$ for every layer of $g$ except the last, which has $K_{out}=3$ since it outputs a 3 dimensional translation vector $\Delta{\mathbf{p_{t}}}$ for each input point $\mathbf{p_{t}}$.

We used a simplified variant of the PointNet feature encoder [15]. The encoder, denoted $f$, applies a shared 3-layered MLP of output sizes $(64,128,1024)$ to each point $\mathbf{p_{q}}\in\mathbf{S_{q}}$ to obtain a local feature of size $(N\times 1024)$ where $N$ is the number of points chosen from the template and is equal to $6890$ in all experiments. It then applies max-pooling along the first dimension to obtain a primitive global feature vector of size $(1024)$, which is refined by a 2-layered MLP of output sizes $(1024,1024)$ to become the final global feature embedding $\mathbf{E}$. Formally,

We pick $N=6980$ for all of our experiments.

Having defined operations performed by each layer of $g$, we define a set of 1-layered MLPs (or more accurately, SLPs), which we will denote as $h_{i}$, that maps the global feature embedding $\mathbf{E}$ to $\hat{\theta}_{i}$. Formally,

So $h_{i}$ has an input size of $1024$ and an output size that matches the number of parameters needed for the $i^{th}$ layer of $g$ (see the calculation above eq. 1 in 3.1).

Following [5], instead of choosing a template prior to training and making it permanently fixed, we initialize our template with a subsampled scan (so that $N=6890$) from the MPI-FAUST training dataset and learns a translation vector $v_{i}$ for each original point on the template $\mathbf{q_{t_{i}}}$. So $\mathbf{p_{t_{i}}}=\mathbf{q_{t_{i}}}+v_{i},i\in\{1,...,N\}$. This allows the network to learn an template that results in the best deformation correspondence during training. The chosen template initialization is depicted in fig 2. In test-time, we hold the template fixed by fixing the learned translation vectors $\{v_{i}\}$. Note that in test-time, if a high-resolution template is desired, we will no longer be able to use the learned translation vectors $\{v_{i}\}_{i=1}^{N}$ because each $v_{i}$ is learned specifically for $\mathbf{p_{t}}_{i}$. We will not have a dense set of translation vectors for every point in $\{p^{\prime}_{t_{j}}\}_{j=1}^{N^{\prime}}$ if we wish to use a high-resolution version (i.e. $N^{\prime}>N$) of the initial template in test-time.

We formulate each layer of $g$ as a customized version of a fully-connected layer (from eq. 1),

The full network $g$ outputs a residual vector for every input point $\mathbf{p_{t}}\in\mathbf{S_{t}}$ which will take it to the location of the corresponding point $\mathbf{p_{q}}\in\mathbf{S_{q}}$. Formally,

Optimizing correspondence among query shapes is equivalent to minimizing the distance between $\hat{\mathbf{p}}_{q}$ and $\mathbf{p}_{q}$ for every query shape. Thus, the formulations above leads to a straightforward supervised training loss with just a single term when training for correspondence.

Since $\hat{\theta}_{g}$ is predicted, we only need to update the parameters of $f$ and $h$ in train-time. Let $\mathbb{S}_{q}$ be the set of all query shapes in the training dataset, this gives the following supervised optimization problem, which is the same equation used in [7]:

Note that in the supervised case we assume having knowledge of the ground-truth location of the point $\mathbf{p_{q,i}}\in\mathbf{S_{q}}$ in correspondence with $\mathbf{p_{t,i}}\in\mathbf{S_{t}}$ for all $i\in{1,...,N}$. However, we do not need any explicit information on the correspondence among the query shapes themselves as it is implicitly given by the correspondence with a common template.

We adopt augmented SURREAL dataset by created T. Groueix et al. [7] using SMPL [11] and SURREAL [18], containing 229,984 meshes of synthetic humans in various poses for training and 200 for validation. Samples of the training data are shown in fig. 3. We train the model using ADAM with at a learning rate of $2*10^{-5}$ for $25$ epochs and then $2*10^{-5}$ for $15$ epochs. This takes around $14$ hours on a machine with a GTX 1080 Ti and a 6-core Intel CPU.

As in [7], we use a procedure during testing to improve the deformation before the optimization of the latent vector $\mathbf{E}$. Before calculating the correspondence, we rotate each query shape $\mathbf{S_{q}}$ by selecting different pitch and yaw angles so that the deformation of the template given the rotated $\mathbf{S_{q}}$ has the smallest Chamfer distance to the rotated $\mathbf{S_{q}}$. (When returning the correspondence output, we apply the inverse rotation matrices to recover the pre-rotation predicted locations).

where $R_{z,\beta}$ stands for the counterclockwise rotation matrix around the $z$ axis by angle $\beta$ (yaw), $R_{y,\alpha}$ stands for the counterclockwise rotation matrix around the $y$ axis (pitch) by angle $\alpha$, and $(R_{z,\beta}R_{y,\alpha})\mathbf{S_{q}}$ denotes the set of points after applying the pitch and yaw rotation matrices to every point in $\mathbf{S_{q}}$. In practice, we try every combination of $(\alpha,\beta)$ where $alpha\in[-\pi/2,\pi/2]$, $\beta\in[-\pi/4,\pi/4]$. The intervals are discrete with strides $\pi/100$ and $\pi/50$ respectively for $\alpha$ and $\beta$. This leads to a total of $100*25$ tries during which we remember the pair $(\alpha^{*},\beta^{*})$ that gives the best deformation and replace $\mathbf{S_{q}}$ with $(R_{z,\beta^{*}}R_{y,\alpha^{*}})\mathbf{S_{q}}$.

As 6.1 illustrates, using $f(\mathbf{S_{q}})$ directly as the feature embedding $\mathbf{E}$ produces suboptimal deformations. To mitigate this problem, during testing only, we use $f(\mathbf{S_{q}})$ as an initialization for $\mathbf{E}$ and optimizes over $\mathbf{E}$ for 3000 iterations using ADAM at a learning rate of $5*10^{-5}$ to implicitly find the $\hat{\theta}_{g}$ that minimizes the Chamfer distance between the deformation $\{g_{\hat{\theta}_{g}}(\mathbf{p_{t}}):\mathbf{p_{t}}\in\mathbf{S_{t}}\}$ and the query shape $\{\mathbf{p_{q}}:\mathbf{p_{q}}\in\mathbf{S_{q}}\}$. We do not perform this step in training as we reduces training speed and we want to train the encoder to predict feature embeddings directly. The Chamfer loss given $\mathbf{E}$ and $\mathbf{S_{q}}$ is defined as:

We use the algorithm presented in [7] to optimize over $\mathbf{E}$ and find the correspondence between a pair of 3-D shapes, shown here in algorithm 1. Note that although we only compute correspondence between two shapes for the FAUST-Inter challenge, the same algorithm can easily be extended to predict correspondence across multiple shapes with an execution time linear to the number of query shapes by matching each shape to the common template.

In a Latent Vector Concatenation decoder construction, a fixed-parameter decoder $g$ takes as input the coordinates of a template point $\mathbf{p_{t}}\in\mathbf{S_{t}}$ concatenated with the feature embedding $\mathbf{E_{q}}$ representing the query shape $\mathbf{S_{q}}$ and outputs the predicted coordinates $\mathbf{\hat{p}_{q}}$ of the corresponding point in the query shape. Since we predict pairwise correspondence based on the query shapes’ individual correspondences to the common template, the quality of the template deformation was crucial to the correspondence accuracy. In LVC, because the decoder has fixed parameters and needs to accurately deform all points on the template into corresponding points on various query shapes when given only $\mathbf{p_{t}}$ and $\mathbf{E_{q}}$ as input, its structure is often complex in terms of the number of trainable parameters, which could leads to problems like lengthy training times and the overfitting. Denoting the feature encoder as $f$ and the decoder as $g$, this leads to the formulation:

Where $\theta_{f}$ and $\theta_{g}$ are jointly optimized during training.

Our approach simplifies the structure of the decoder by formulating it as a concise dynamic multi-layer perceptron, that is, a MLP whose parameters are dynamically and uniquely determined by another neural network for each input. Denoting the encoder f and the dynamic decoder g, we have:

Because $\hat{\theta}_{g}$ is determined by f, we only need to optimize $\theta_{f}$ during training. Note that the decoder computes a location residual instead of the coordinates of the actual deformed point $\mathbf{\hat{p}_{q}}$ that corresponds to $\mathbf{p_{t}}$. We found that predicting the residual was an easier task to learn for the network.

Although they perform different tasks, the neural networks in [7], [15], and [14] have all employed $g$ as an LVC decoder that is an MLP with non-dynamic parameters. E. Mitchell pointed out in [12] that this formulation is equivalent to having a MLP that has fixed input and weights but a variable bias. Consider a case in which $g$ needs to deform a template into query shapes, then the LVC deformation decoder $g$ would take in of $\mathbf{p_{t}}\in\mathbf{S_{t}}$ concatenated with a feature embedding representing the query shape $\mathbf{E}=f_{\theta_{f}}(\mathbf{S_{q}})$. Such is the case in [7]. Let vector $x$ be the concatenation of $\mathbf{p_{t}}$ and $\mathbf{E}$, that is, $x=(\mathbf{p_{t}};\mathbf{E})$, then the first layer of the LVC MLP decoder will perform the following operation:

where $a$ is the pre-activation output, $W$ is the fixed weights matrix, and $b$ is the fixed bias. We can rewrite $x$ as the sum of two vectors $y$ and $z$ where $y=(\mathbf{p_{t,x}},\mathbf{p_{t,y}},\mathbf{p_{t,z}},0,0,...)$ and $z=(0,0,0,\mathbf{E})$

Note that $W,b,x$ are fixed with regards to $\mathbf{S_{q}}$ and only z changes with different $\mathbf{S_{q}}$, so the only term that varies with the query shape is $Wz$. Thus, $Wz$ can be seen as a dynamic bias predicted from query shapes for the first layer, which has fixed weights and input (points on the template were fixed in test-time). Using the concatenation of $\mathbf{p_{t}}$ and $\mathbf{E}$ as input is therefore equivalent to having an MLP that has fixed input, weights, and biases, except that a variable bias term $Wz$ is added to layer 1’s pre-activation output that depends on $\mathbf{S_{q}}$. The low adaptability of the LVC decoder prevents it from producing high-quality deformations with a concise structure.

In the case of the meta decoder, we set all parameters of $g$ to be dynamic. With this approach, the input $x$ is simply a 3-D vector $(\mathbf{p_{t,x}},\mathbf{p_{t,y}},\mathbf{p_{t,z}})$, and the first layer of the decoder MLP performs the operation

where $W,s,b$ are given by $h_{1}(f(\mathbf{S_{q}}))$. The process is similar for subsequent layers. As a result, all parameters of $g$ are dynamic a function of the query shape $\mathbf{S_{q}}$. Defining $g$’s parameters to be a function of $\mathbf{S_{q}}$ offers the decoder more flexibility in computing the optimal translation that deforms the template into diverse query shapes. We also include an element-wise scaling factor $s$ in the formulation of $g$ to facilitate the learning of $h$, which maps $\mathbf{E}$ to $\hat{\theta}_{g}$. To see this, suppose that a particular output neuron of $g$ needs to be changed in order to compute the optimal deformation. Multiplying $s_{1}$ by $\lambda$ has the same effect on the output as scaling all elements in the first row of $W$ by $\lambda$, but is an easier change to predict for $h$ since it requires predicting a single number instead of multiple numbers to capture this change. Overall, the meta decoder has fully adaptive parameters to different query shapes. By contrast, the LVC decoder has only a variable bias in the output of the first layer and fixed parameters everywhere else. The enhanced adaptability of $g$ with respect to query shapes allows it to produce better deformations of the template with fewer parameters and at higher speeds.

We show the deformation of the template into the query shapes before the optimization of $\mathbf{E}$, and compare Meta Deformation Network against the non-meta deformation network developed by T. Deprelle et al. in [5]. The figures are included in fig. 4.

Both approaches give a reasonable unoptimized deformation of a template into the query shape, though both suffer from varying amounts of distortions and imprecision from the actual query shape. The optimization step depicted in 6.2 over $\mathbf{E}$ improves the quality of the deformation.

We also test how sensitive the quality of the template deformation is to different numbers of optimization steps for $\mathbf{E}$. We qualitatively demonstrate that the deformation continues to improve in quality as $\mathbf{E}$ is optimized for more iterations, but the marginal gain diminishes as the number of iterations increases, with the difference between 1,000 and 3,000 iterations hardly noticeable (See fig. 5).

The imprecision of the unoptimized deformation suggests that our ShapeNet encoder $f$ is unable to produce an optimal feature embedding $\mathbf{E}$ that fully captures the characteristics of the query shape; the unoptimized deformation has a more generic structure lacking details of the person’s body shape, which the post-optimization deformation is able to capture. The relatively weak encoding power of our simplified ShapeNet encoder, can be alleviated by optimizing over $\mathbf{E}$ given $f(\mathbf{S_{q}})$ as its initialization. Also note that although the query shape has incomplete data around the person’s feet, our decoder has learned to reconstruct the missing parts from the input.

Though the Meta Deformation Network did not surpass the state-of-the-art performance, it showed an improvement over 3D-CODED w/. Learned 3D Translation Template, an approach that is comparable to ours in that it shares the same simplified PointNet encoder design and also applies a learnable translation matrix to the points of the base template, with the difference being that it its LVC decoder has fixed parameters. This shows quantitatively that having the meta decoder produces more accurate correspondences on FAUST-Inter than does a static-parameter decoder (while also having speedier execution).

We report results from training under a low-resolution template and hot-swapping in a different high-resolution template in inferencing (details for this procedure are given in 6.4). See the accuracy numbers are shown in table 1.

In the training step, we have trained a feature encoder $f$ and a parameter mapper $h$ to enable $g$ to predict deformations from a known template into unseen query shapes. Here, we also test how well the model performs when given a template and query shapes that are both unseen in training. We run an experimental setup where we find the rotation matrices and the feature embedding initialization for $\mathbf{S_{q}}$ using the learned template $\mathbf{S_{t}}$ but swaps in a different, higher-resolution template $\mathbf{S_{t}}^{\prime}$ for the optimization step and subsequent correspondence calculation. The learned template $\mathbf{S_{t}}$ contains 6,890 vertices while the unseen template $\mathbf{S_{t}}^{\prime}$ contains slightly over 176,000 vertices. A visualization is given in fig. 6. The visual results is consistent with table 2, with little difference in error rate between using a learned template and a high-resolution unseen template.

Results from table 2 suggest that $h$ is able to transfer the learned predictions for $\hat{\theta}_{g}$ when $g$ operates on a different template. As a result, the meta decoder adapts wells operating on a template that is different from the one used in training and takes advantage of the higher resolution of the new template to produce more accurate correspondences. The high adaptability of the Meta Decoder Network suggests the possibility of variations of the model that use a dynamic template in addition to the dynamic decoder, which we desire to research in future works.

We compare the training and testing speeds of the Meta Deformation Network against those of 3D-CODED, which has the same encoder structure but uses an LVC decoder. We train both models on the same Extended SURREAL dataset created by [7].

We find that the Meta Deformation Network took $35.8\%$ less time to carry out an iteration of training (table 3) with a batch size set of 32, yet it yields improved accuracy (table 1) on the FAUST Inter challenge than did the model from [5] that also had a translation-based learnable template.

For testing, we evaluate the time a model takes to compute the correspondence for one pair of 3D scans from FAUST Inter. In testing, our model took $4.45\%$ less time to finish. Here the difference was less dramatic than in training, which we think was due to the under-utilization of the computer’s hardware in both methods due to the the latent vector optimization with a batch size of 1. See table 4.