Learning Anchored Unsigned Distance Functions with Gradient Direction Alignment for Single-view Garment Reconstruction

To represent shapes with open surfaces, we adopt unsigned distance fields (UDFs) [13] which assign a non-negative scalar value $s$ to a spatial point $\mathbf{p}$:

$$UDF({\mathbf{p}})=s:{\mathbf{p}}\in{\mathbb{R}^{3}},s\in\mathbb{R}_{0}^{+},$$

(1)

where $s$ represents the unsigned distance from $\mathbf{p}$ to the closest surface and the shape surface is implicitly represented by the zero level-set $UDF(.)=0$. In contrast to signed distance fields (SDFs) [30] or occupancies [27] which divide the 3D space into inside and outside the surface, UDFs allow to naturally represent open surfaces. We can project $\mathbf{p}$ onto the surface by moving $\mathbf{p}$ along the negative gradient direction of UDF:

$${\mathbf{q}}:={\mathbf{p}}-UDF({\mathbf{p}})\cdot{\nabla_{\mathbf{p}}}UDF({\mathbf{p}}).$$

(2)

Dense point clouds are able to be easily computed from the implicit surfaces using fast gradient evaluation for UDFs [13] and naive classical algorithms for meshing [9] can be used to generate the corresponding meshes.

It is worth noting that regressing a continuous distance field is much more difficult than classifying a binary occupancy value. Moreover, different from SDFs, UDFs need gradient direction computation at inference. Besides a point itself, the prediction accuracy for the neighborhood of the point is also required to ensure the correct gradient direction. Thus, discriminative features are more critical for learning a good UDF.

To this end, our AnchorUDF employs both local image features and 3D position features of query points to predict UDF values. The key idea is to leverage a set of anchor points located around 3D surfaces to compute relative position features for query points, which makes the distance function better fit the garment topologies.

Given an input image $I$, a fully convolutional image encoder is first used to compute a feature map of $I$. For a 3D query point $\mathbf{p}$, we project it onto its corresponding position on the image plane to compute its local image features from the feature map by bilinear interpolation at the projected pixel coordinates: ${\Phi_{img}}(\pi({\mathbf{p}}),I)$, where $\pi$ represents the (weak) perspective camera projection.

In order to compute more discriminative 3D position features for the query point $\mathbf{p}$, a set of anchor points $\mathcal{C}$ located around the surface is introduced to encode the 3D position of $\mathbf{p}$. We then discretize the anchor points $\mathcal{C}$ to a voxel grid and feed the grid into a point encoder consisting of a series of 3D convolutional layers to produce a 3D feature tensor of $\mathcal{C}$. By applying trilinear interpolation to the feature tensor according to the 3D coordinates of $\mathbf{p}$, we extract the feature vector $\Phi_{pos}({\mathbf{p}},\mathcal{C})$ as the position features of the query point. Compared to using an absolute depth value to provide the 3D position information [33], our position features reflect relative position relationship between query and anchor points and provides stronger 3D space context to make UDF better capture multiple topologies.

To obtain a set of anchor points which can adequately cover the surface only using a small number of points, we cluster points sampled from the ground-truth surface with k-means and use clustering centers $\tilde{\mathcal{C}}$ as targets to learn a regression network on top of the backbone to predict the anchor points. Here the anchor point prediction and the shape reconstruction are learned jointly, which can be seen as multi-task learning since they share the backbone. We argue that compared to the full shape, these few anchor points (typically a few hundred) are easier to estimate. Therefore, our method can also be regarded as a coarse-to-fine strategy, i.e., a shape profile is first predicted, then further refined to produce the detailed surface.

At last, we concatenate the local image features, the 3D position features and the 3D coordinates of the query point as the input of a decoder to predict UDF values and our AnchorUDF is formulated as:

$${f_{I}}({\mathbf{p}},\mathcal{C};{\mathbf{w}})={f_{dec}}([{\Phi_{img}}(\pi({\mathbf{p}}),I),{\Phi_{pos}}({\mathbf{p}},\mathcal{C}),{\mathbf{p}}]),$$

(3)

where $\mathbf{w}$ denotes network parameters and the decoder $f_{dec}$ is parameterized by multi-layer perceptrons (MLP) with ReLU in its last layer.

To learn our AnchorUDF, for an input image $I$, we generate training examples $\mathcal{P}_{I}$ by sampling points near its corresponding surface and computing their ground-truth UDF values $UDF({\mathbf{p}})$. Then, we minimize the L1 loss between the predicted and ground-truth UDF values on $\mathcal{P}_{I}$ by updating parameters $\mathbf{w}$ of ${f_{I}}({\mathbf{p}},\mathcal{C};{\mathbf{w}})$:

$${L_{UDF}}=\sum\limits_{{\mathbf{p}}\in\mathcal{P}_{I}}{|\min({f_{I}}({\mathbf{p}},\mathcal{C};{\mathbf{w}}),\delta)-\min({UDF({\mathbf{p}})},\delta)|}.$$

(4)

Similar to [13, 30], a small value $\delta$ is used to clamp the maximal regressed distance to concentrate the model capacity on details in the vicinity of the surface.

In order to learn the anchor point predictor, we consider a loss defined by the Chamfer distance [16] between the predicted anchor points $\mathcal{C}$ and the targets $\tilde{\mathcal{C}}$ because the set of anchor points is unordered:

$${L_{AP}}=\sum\limits_{{\mathbf{c}}\in{\mathcal{C}}}{\mathop{\min}\limits_{{{\mathbf{\tilde{c}}}}\in\tilde{\mathcal{C}}}||{\mathbf{c}}-{{\mathbf{\tilde{c}}}}||_{2}^{2}}+\sum\limits_{{{\mathbf{\tilde{c}}}}\in\tilde{\mathcal{C}}}{\mathop{\min}\limits_{{\mathbf{c}}\in{\mathcal{C}}}||{\mathbf{c}}-{{\mathbf{\tilde{c}}}}||_{2}^{2}}.$$

(5)

The standard back-propagation through AnchorUDF can be used to compute the spatial gradients of the learned distance field to project points by Eq. (2).

However, only optimizing the point-wise distance loss (4) cannot guarantee a good estimation for the true gradient directions of UDF. As illustrated in Fig. 3 (for the sake of brevity, here $I$ and $\mathcal{C}$ in ${f_{I}}({\mathbf{p}},\mathcal{C};{\mathbf{w}})$ are omitted), consider two points ${\mathbf{p}}_{1}$ and ${\mathbf{p}}_{2}$ on the neighborhood of a point ${\mathbf{p}}_{0}$, when the ground truth UDF values of ${\mathbf{p}}_{1}$ and ${\mathbf{p}}_{2}$ are close, because the distance loss does not penalize the relationship between points, even if the loss is small, it is also possible that $f({{\mathbf{p}}_{1}};{\mathbf{w}})>f({{\mathbf{p}}_{2}};{\mathbf{w}})$. In this case, the predicted gradient direction at ${\mathbf{p}}_{0}$ will be significantly different from the ground truth one. Therefore, during training we explicitly constrain the spatial gradient of AnchorUDF to align its direction with the true gradient direction by the following loss:

$${L_{GDA}}=\sum\limits_{{\mathbf{p}}\in\mathcal{P}}{1-\cos({\nabla_{\mathbf{p}}}f({\mathbf{p}};{\mathbf{w}}),{\nabla_{\mathbf{p}}}UDF({\mathbf{p}}))}.$$

(6)

In practice, the direction of ${\nabla_{\mathbf{p}}}UDF({\mathbf{p}})$ can be computed by $({\mathbf{p}}-{\mathbf{q}})/||{\mathbf{p}}-{\mathbf{q}}||_{2}$, where ${\mathbf{q}}$ is the point closest to ${\mathbf{p}}$ on the surface.

Finally, the overall objective function is

$$L={L_{UDF}}+{\lambda_{1}}{L_{AP}}+{\lambda_{2}}{L_{GDA}},$$

(7)

where $\lambda_{1}$ and $\lambda_{2}$ are loss weights.

We validate the effectiveness of main components in our proposed AnchorUDF by both qualitative and quantitative evaluations. We first investigate the impact of the number of anchor points on MGN dataset. Table 1 lists Chamfer and P2S errors obtained by using 100, 300, 600 and 900 anchor points. It can be seen that using more anchor points achieves better results because stronger space context can be referenced by query points. However, when the point number exceeds a certain value, e.g., 900, the performance degrades. The reason is that too many points to regress increases the difficulty of learning the point prediction network and reduces its stability when testing, which further affects the subsequent shape reconstruction. Fig. 4 shows that compared to 100 anchor points, using 600 points produces finer shapes, especially for the side view.

We further assess the importance of the proposed anchored 3D position features by applying different 3D position features to our reconstruction framework, including the depth value, 3D point coordinates and our anchor point based position features. As reported in Table 2, our position feature obtains the best results on both MGN and Deep Fashion3D. The depth value and 3D coordinates only provide absolute and local position information of query points, which is not enough to predict UDF with good neighborhood consistency. In contrast, anchor points located around the surface provide the information about shape profile which enables relative position computation between query points and the surface for more global and discriminative position features. Fig. 5 illustrates some qualitative results. As one can see, the depth value fails to project some points which are far away from the surface, and 3D point coordinates perform a little better but still cannot obtain accurate surfaces at edges of the shapes.

To evaluate the influence of the proposed gradient direction alignment, Table 3 reports the results with and without $L_{GDA}$ in Eq. (6). It can be observed that optimizing $L_{GDA}$ further reduces Chamfer and P2S errors. Visual comparison is shown in Fig. 6. By explicitly aligning spatial gradient direction of the distance field during training, we can obtain sharper geometric details for the visible region and less artifacts for the self-occluded region.

We also visualize the anchor points predicted by our model to see if these points actually indicate shape profiles. As illustrated in Fig. 7, the predicted anchor points are evenly distributed around the input garments for different garment topologies and thus are able to provide holistic 3D shape information for query points.

We compare our method against related single-view 3D reconstruction methods with different shape representations, including BCNet [22], Pixel2Mesh [36], PIFu [33] and PIFuHD [34]. As listed in Table 4, our AnchorUDF obtains the lowest reconstruction errors on both MGN and Deep Fashion3D datasets. Fig. 8 qualitatively compares our method with BCNet [22] which represents garment shapes using template meshes on MGN dataset. Although such template-based method can recover garment shapes with open surfaces, it tends to produce overly smooth surfaces and loses lots of geometric details. Fig. 9 presents visual comparison on Deep Fshion3D dataset. Pixel2Mesh [36], a mesh-based method whose shape is initialized from an ellipsoid, only recovers coarse shapes and cannot handle large deformations of garments. PIFu [33] using implicit function representation is able to generate detailed surfaces. However, it can only reconstruct closed surfaces, thus has difficulty handling garment reconstruction with multiple open boundaries. In contrast, our method can not only retain fine-scale geometric details but also faithfully capture open garment surfaces.

We further incorporate the HD module introduced in PIFuHD [34] into our AnchorUDF (AnchorUDF-HD) to evaluate the scalability of our method on high-resolution input images ($1024\times 1024$). To make a fairer comparison, we try to directly use UDF as the implicit functions in PIFu (PIFu-UDF) and PIFuHD (PIFuHD-UDF) so that they can also represent open surfaces. Note that here we do not use extra normal maps as input because our main purpose is to verify if our method can be further improved under the HD framework. Table 5 shows quantitative comparison on Deep Fashion3D. We can see that by taking high-resolution images as input, AnchorUDF-HD effectively reduces the reconstruction errors, especially P2S, indicating more accurate shape details are recovered. PIFuHD-UDF performs better than PIFu-UDF but still worse than our AnchorUDF, which shows that direct combination with UDF cannot work well and further confirms the necessity of the proposed components in AnchorUDF. Qualitative evaluation is shown in Fig. 10. We can see that AnchorUDF-HD reconstructs more accurate shapes and richer details. Note that although PIFuHD-UDF uses 3D embedding provided by the coarse level as the input of distance function instead of the depth value, it sill cannot project points appropriately onto the surface. More qualitative results can be found in the supplemental materials.