Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape

In early years, some methods use CNN to directly estimate 3D Morphable Model parameters [10, 25, 12] or its variants [26, 27, 28, 29, 30], providing both dense face alignment and 3D face reconstruction results. However, the performances of these methods are restricted by the limitation of linear 3D space [27, 31, 32, 33, 34, 35]. It is also challenging to estimate the required face transformations, including perspective projection and 3D thin plate spline transformation. Recently, vertex regression networks [16, 9], which bypass the limitation of the linear model, have achieved state-of-the-art performance on their respective tasks. VRN [16] proposes to regress 3D faces stored in the volume, but the redundant volumetric representation loses the semantic meanings of 3D vertices. PRNet [9] designs a UV position map, a 2D image recording the 3D facial point cloud while maintaining the semantic meaning at each UV position. Cheng et al. [19] encode an image with a CNN and decode the 3D geometry with a lightweight Graph Convolutional Networks (GCN) for efficiency. Although breaking through the limitations of 3DMM, their reconstruction results are still model-like as their ground truth still comes from 3DMM fitting, e.g., 300W-LP [10].

Another way to bypass the limitation of 3DMM is to encode the 3D face model by a neural network, which can be learned on the unlabeled images in an analysis-by-synthesis manner. Tran et al. [11] achieve a certain breakthrough by learning a nonlinear model from unlabeled images in a self-supervised manner. They first estimate the projection, shape, and texture parameters with an encoder and then adopt two CNN decoders as the nonlinear 3DMM to map the parameters to a 3D textured face. With an analytically-differentiable rendering layer, the network can be trained in a self-supervised way by comparing the reconstructed images with the original images. Zhou et al. [18] follow a similar path but decode the shape and texture directly on the mesh with graph convolution for efficiency. Gao et al. [36] utilize a CNN encoder and a GCN decoder to learn 3D faces from unconstrained photos with photometric and adversarial losses. In addition to learning a single model accounting for all shape variations, some works construct facial details independently. Chen et al. [37] un-warp image pixels and facial texture to the UV plane with a fitted 3DMM and predict a displacement map of the coarse shape by minimizing the difference between the reconstructed and the original images. Wang et al. [38] adopt a similar method but predict the moving rate in the direction of the vertex normal instead of the displacement map. The self-supervised nonlinear 3DMMs can cover a larger shape space than linear 3DMMs, but the fine-grained geometry, as a subtle constituting factor of face appearance, is easy to ignored due to the absence of straightforward training signals from the ground-truth shapes.

Considering the lack of high-precision 3D scans, a common strategy for fine-grained reconstruction is Shape from Shading (SfS), which recovers 3D shapes from shading variations in 2D images. However, traditional SfS methods largely depend on the prior geometric knowledge and suffer from the ambiguity problem caused by albedo and lighting estimation. To address this issue, Richardson et al. [25] refine the depth map rendered by a fitted 3DMM with the SfS criterion to capture details, where a trained depth refinement net directly outputs high-quality depth maps without calculating albedo and lighting information. DF²Net [39] further introduces depth supervision from RGB-D images to regularize SfS in a cascaded way. However, these methods only learn the $2.5$D depth map and lack correspondence with the 3D face model. Jiang et al. [40] propose to refine the coarse shape with photometric consistency constraints to generate a more accurate initial shape. Li et al. [41] employ a masked albedo prior to improve albedo estimation and incorporate the ambient occlusion behavior caused by wrinkle crevices. Chen et al.  [42] construct a PCA model for details using collected scans and adopt the analysis-by-synthesis strategy to refine the PCA model. Guo et al. [43] employ a FineNet to reconstruct the face details, whose ground truth is synthesized by inverse rendering. Although the methods capture fine-level geometric details (such as wrinkles), their global shapes still come from the 3DMM fitting. Unlike SfS, our method aims to improve the global visual effect such as facial feature topology and face contour.

In addition to single-view reconstruction, high-quality 3D faces can be better recovered with multiview inputs. Many 3D face datasets, e.g., D3DFACS [44] and FaceScape [15], use the calibrated depth camera arrays to reconstruct high-quality 3D shapes. Besides, recent deep learning based works have shown that multiview images of the same subject benefit shape reconstruction. DECA [45] proposes a detail-consistency loss between different images to disentangle expression-dependent wrinkles from person-specific details. MVF-Net uses the MultiPIE [46] samples collected by $15$ cameras for multiview 3DMM parameters regression. Shang et al. [47] also adopt the MultiPIE to alleviate the pose and depth ambiguity during 3D face reconstruction. Lin et al. [48] take the RGB-D selfie videos captured by iPhone X to reconstruct high-fidelity 3D faces. Although our proposal focuses on 3D reconstruction from a single image, these multiview systems motivate our network structure to observe the input image in several calibrated views.

Given a face image, we aim to simulate its appearances at $5$ constant poses, whose $($pitch, yaw$)$ angles are $(0^{\circ},0^{\circ})$, $(0^{\circ},25^{\circ})$, $(0^{\circ},50^{\circ})$, $(15^{\circ},0^{\circ})$ and $(-25^{\circ},0^{\circ})$, as shown in Fig. 2. The view synthesis can be achieved by transferring the image $\mathbf{I}$ to a 3D object through the strong prior from the fitted 3DMM $\mathbf{V}$. Following face profiling [10], we tile anchors on the background, set their depth to the mean of the 3D face, and triangulate them to a 3D mesh $\mathbf{V}^{I}$, which can be rendered at any required views, as shown in Fig. 3. However, the texture in 3D mesh $\mathbf{V}^{I}$ is not complete. When the target pose is smaller than the original pose, the self-occluded region will be exposed, leading to large artifacts. To fill the occluded region, we also generate a flipped 3D mesh $\mathbf{V}^{I}_{flip}$ from the mirrored image and register it to $\mathbf{V}^{I}$ to complete the face texture, as shown in Fig. 3.

When rendering the 3D mesh at specified views, we find that each pixel is determined by either the original image or the flipped image, depending on the vertex visibility. The visibility score of each vertex is defined in terms of the angle between the vertex normal and the view direction:

where $\mathbf{l}=[0,0,1]$ is the view direction and $\mathcal{N}(\mathbf{v})$ is the normal of vertex $\mathbf{v}$ in the original image. Note that the visibility scores of face vertices are increased by $2$ to ensure that face regions overlap the background. The visibility scores, regarded as the face texture, are then rendered to the target view to obtain the visibility map, as shown in Fig. 3. Finally, the virtual face image at a specified view is constructed by:

where $\mathcal{R}(\cdot)$ renders the 3D mesh $\mathcal{V}(\mathbf{V}^{I},\mathbf{C}_{v})$ to a specified view, indicated by the camera parameter $\mathbf{C}_{v}$, $\lambda$ is the weight map calculated by comparing the visibility maps $\mathbf{vis}$, and $\odot$ is the element-wise production. As shown in Fig. 3, the invisible region is made transparent to indicate that the texture is not real but inpainted by face symmetry, which should guide the network to concentrate more on the visible texture. In the case that the input is the frontal view, we directly stretch the texture as the side-face texture, and the neural network is trained to handle the artifacts.

To fuse the features extracted from multiview inputs, we propose a novel many-to-one hourglass network with a $5$-stream encoder and a $1$-stream decoder, as shown in Fig. 2. The encoder learns each view by a weight-shared CNN, and then concatenates the features as a multiview description. Afterwards, the decoder deconvolves the feature to the UV displacement map [24]. Besides, the intermediate features at symmetric layers are added together to merge high-level and low-level information. However, the particular structure of the many-to-one hourglass network poses a critical challenge for intermediate feature fusion. The features at one position have different semantic meanings, since the encoder has $5$ different image views and the decoder is on the UV plane. Directly adding the features would degrade each other. To address the problem, each feature map in the encoders is warped to the UV plane according to the fitted 3DMM before being sent to the decoder.

Regarding the three properties desired for network design, the VMN fulfills the normalization property by transferring the input face to constant views. It also possesses the lossless property as little information is lost during the construction of virtual multiview input, i.e., the original topology and the external face regions are preserved. For the concentration property, since the intermediate features are all aligned to the UV plane, the receptive field can consistently concentrate on the most related region.

In 3D object retrieval, the light field descriptor [53] is widely computed for silhouette images of 3D shapes. Besides, recent analysis-by-synthesis 3D face reconstruction methods [17, 18, 36] optimize face images generated by a facial appearance model. Inspired by these two achievements, we consider that the images rendered by a 3D face can be utilized to model its visual effect, and explore a Plaster Sculpture Descriptor (PSD) to measure the reconstruction quality directly by how we see it. As shown in Fig. 2, a 3D shape is considered as a plaster sculpture with all-white vertex color and orthogonal light on it. During training, the 3D shape is rendered at $5$ views, whose pitch and yaw angles are $(0^{\circ},0^{\circ})$, $(0^{\circ},90^{\circ})$, $(0^{\circ},-90^{\circ})$, $(30^{\circ},0^{\circ})$ and $(-30^{\circ},0^{\circ})$, and the L2 distances of the rendered images between the output and the ground-truth 3D shapes are formulated as the visual-effect distance:

where $v$ is the view index, $\mathcal{R}(\cdot,\cdot)$ is the renderer whose input is 3D shape and texture, $\mathbf{R}_{v}$ is the rotation matrix corresponding to each view, and $\mathbf{T}^{w}$ is the all-white texture under orthogonal light. If we employ a differentiable renderer [54], the $\mathcal{D}^{psd}$ can be directly regarded as a loss function. As shown in Fig. 4(e), the facial features recovered by PSD are more similar to the ground truth, which is crucial in visual evaluation.

Although plaster sculpture descriptor measures how the shapes are observed by humans, there is an intrinsic problem on the face contour due to the defect of the differentiable renderer. As shown in Fig. 5, on the PSD error map, the differentiable renderer back-propagates the gradients of pixels to the corresponding vertices. Therefore, only the outputted vertices that located within the target face region have the back-propagated signals, and those on the background are not trained, leading to unsatisfactory contour reconstruction. To tackle this issue, we propose a Visual-Guided Distance (VGD) loss which considers the visual-effect distance as the vertex weights rather than the optimization target. The overview of VGD is shown in Fig. 6, whose insight is that, given the ground-truth position of each vertex, we only need to find which vertices dominate the visual effect. Specifically, we calculate the output-to-target and the target-to-output PSD errors simultaneously, on either of which the poorly fitted face contour inevitably brings large values. Then, we employ pixel-to-vertex mapping to retrace the pixel errors to vertices, and add the vertex errors from both the output and the target. Finally, the accumulated vertex errors from all the views are regarded as the vertex weights for the MSE loss:

where $\mathbf{W}^{psd}$ represents the vertex weights, $\mathcal{R}_{\mathbf{S}}^{-1}(\cdot)$ denotes the inverse rendering function that maps pixel errors to the vertices according to the 3D face $\mathbf{S}$, and $\mathcal{D}^{psd}_{v}$ denotes the PSD error map in the $v$th view. As shown in Fig. 4(f), the visual-guided weights enable the network to focus on the visual discriminative regions, such as the face contour and the rough regions.

The training data requires face images and their corresponding topology-uniform 3D shapes. Thus, the first task is registering a template to the depth images, where Iterative Closest Point (ICP) method is commonly adopted. However, plausible registered results are not sound enough to be the supervision of neural network training. Semantic consistency, i.e., each vertex has the same semantic meaning across 3D faces is a critical criterion. As shown in Fig. 7, semantically consistent registration cannot be achieved by finding the closest point in $(x,y,z)$ only. Considering that our registration target is RGB-D data, whose texture contains discriminative features for vertex matching, we introduce two terms on the RGB channels to provide more constraints in the popular optimal nonrigid-ICP loss function [55].

The first term comes from the edge. As shown in Fig. 7, a series of landmarks, which are dense enough to finely enclose the eyes, eyebrows and mouth, are detected to represent the edges on the facial features. The constraint is formulated as:

where $\left[\begin{array}[]{cccc}a^{11}&a^{12}&a^{13}&a^{14}\\ a^{21}&a^{22}&a^{23}&a^{24}\\ \end{array}\right]$ is the first and second rows of the $3\times 4$ affine transformation matrix of each vertex, which should be optimized, $(x^{3D},y^{3D},z^{3D})$ is an edge 3D landmark and $(x^{2D},y^{2D})$ is the corresponding 2D landmark detected on the image.

The second term aims to regularize the face contour. However, it is unreliable to optimize the vertex-to-landmark distances as in Eqn. 9, since there is no strict correspondence between 3D and 2D contour landmarks [56]. To utilize contour landmarks, we perform curve fitting rather than landmark matching to reduce the semantic ambiguity. As shown in Fig. 7, the 2D contour landmarks are sequentially connected to a contour curve. Then, the 3D contour vertices found by landmark marching [56] are fitted to the 2D curve by:

where $(x^{2D}_{c},y^{2D}_{c})$ is the 2D-closest point on the curve $\mathbb{C}$ to the 3D vertex $(x^{3D},y^{3D},z^{3D})$. During ICP registration, Eqn. 9 and Eqn. 18 are employed as additional terms constraining the first two rows of affine transformations, and the third row is optimized separately by traditional ICP constraints [55].

Given the registered face $\mathbf{V}_{regist}$, we further disentangle rigid and non-rigid transformations by:

where $(f,\mathbf{R},\mathbf{t}_{3d})$ are the rigid transformation parameters and the optimized shape is regarded as the ground-truth shape $\mathbf{S}^{*}$. The difference between the ground-truth shape and the 3DMM-fitted shape $\Delta\mathbf{S}=\mathbf{S}^{*}-\mathbf{\overline{S}}-\mathbf{A}_{id}\bm{\alpha}_{id}-\mathbf{A}_{exp}\bm{\alpha}_{exp}$ will be the target of the neural network training.

Currently, most of the public RGB-D images are frontal faces [57, 58, 59], leading to the risk of poor pose robustness. Although RGB-D data can be rendered to other views, the results suffer from large artifacts due to the incomplete depth channel and face texture. To address this issue, a full-view augmentation method is proposed specifically for RGB-D data. Based on the face profiling method [10], we complete the depth channel for the whole image space, where the depth on the face region comes directly from the registered 3D face and the depth on the background is coarsely estimated by some anchors, as shown in Fig. 8. Specifically, these anchors $(x_{i},y_{i})$ are triangulated to a background mesh and their depth values $d_{i}$ are regularized by a depth-channel constraint and a smoothness constraint:

where $Depth(x,y)$ is the depth channel of the RGB-D data, $Mask(x,y)$ indicates whether $(x,y)$ is hollow, and $Connect(i,j)$ indicates whether two anchors are connected by the background mesh. With the complete depth channel (Fig. 8), the RGB-D can be rotated (Fig. 8) and rendered (Fig. 8) to any views. More implemental details are provided in the supplemental materials.

Different from face profiling [10], whose purpose is enlarging medium poses to large poses, our method aims to augment frontal faces. However, the main drawback of the rotate-and-render strategy is that, when rotating from frontal faces, there are serious artifacts on the side face due to the incomplete face texture, as shown in Fig. 8. In this work, we employ the texture and illumination model in 3DMM as a strong prior to refine the artifacts. Specifically, the face texture is modeled by PCA [13]:

where $\mathbf{\overline{T}}$ is the mean texture, $\mathbf{B}$ is the principle axis of the texture and $\bm{\beta}$ is the texture parameter. Given 3D shape $\mathbf{V}$ and texture $\mathbf{T}$, the Phong illumination model is used to produce the color of each vertex [49]:

where $C_{i}$ is the RGB of the $i$th vertex, the diagonal matrix $\mathbf{Amb}$ is the ambient light, the diagonal matrix $\mathbf{Dir}$ is the parallel light from direction $\mathbf{l}$, $\mathbf{n}_{i}$ is the normal direction of the $i$th vertex, $k_{s}$ is the specular reflectance, $\mathbf{ve}$ is the viewing direction, $\nu$ controls the angular distribution of the specular reflection and $\mathbf{r}_{i}=2\cdot\langle\mathbf{n}_{i},\mathbf{l}\rangle\mathbf{n}_{i}-\mathbf{l}$ is the direction of maximum specular reflection. The collection of texture parameters is $\mathbf{p}_{tex}=[\bm{\beta},\mathbf{Amb},\mathbf{Dir},\mathbf{l},k_{s},\nu]$. Given the ground-truth 3D shape $\mathbf{V}_{regist}$, the texture parameters $\mathbf{p}_{tex}$ can be estimated by the analysis-by-synthesis manner:

where $\mathbf{I}(\mathbf{V})$ is the image pixels at the vertex positions. The optimized result $C(\mathbf{p}_{tex})$ is the face texture, as shown in Fig. 8. With the estimated model texture, we render the 3D face with both the image pixels and the model texture, as shown in Fig. 8 and Fig. 8. Then, we inpaint the self-occluded side face with the model texture through Poisson editing, as shown in Fig. 8. It can be seen that the model texture is realistic enough to inpaint the smoothly textured side face.

In addition to pose variations, shape variations covered by the training data are also important since the personalized shape is the main goal. Unfortunately, existing data does not contain sufficient identities due to demanding data collection. In this section, we propose a shape augmentation method to transform the underlying 3D shape of a face image and refine the face appearance accordingly. In specific, we first construct the target shape to transform, whose eye, nose, mouth, and cheek come from different faces in the datasets, as shown in Fig. 9. Second, the base image is tuned into a 3D mesh following the same process in the full-view augmentation, as shown in Fig. 9. Third, we replace the 3D shape and warp the background to adjust the new face. Specifically, the background anchors are adjusted by minimizing the following function:

where $(x^{s}_{i},y^{s}_{i})$ is the anchor position on the source image, $(x^{t}_{i},y^{t}_{i})$ is its target position on the augmented image, $FaceContour(i)$ indicates whether anchor $i$ is located on the face contour (the red points in Fig. 9), and $Connect(i,j)$ indicates whether two anchors are connected by the background mesh. Afterwards, we render the warped 3D mesh and obtain the shape-transformed result, whose ground-truth 3D shape is the target 3D shape, see Fig. 9.

In addition to image pixel warping, shading adjustment is also important according to the shape-from-shading theory [32], which can be achieved by the illumination model the same as Eqn. 22:

where all the parameters except $\mathbf{n}^{t}$ and $\mathbf{r}^{t}$ are from the source image, and $\mathbf{n}^{t}$ and $\mathbf{r}^{t}$, which account for shading, are from the target shape. Finally, we change the facial pixels to $\mathbf{C}^{t}$ and obtain the final result of shape transformation, see Fig. 9. We provide more implemental details in the supplemental materials.

Fine-Grained 3D Face (FG3D) is constructed from three datasets, FRGC, BP4D, and CASIA-3D. FRGC [59] includes $4,950$ samples, each of which has a face image and a 3D scan in full correspondence. BP4D [58] contains $328$ 2D+3D videos from $41$ subjects, where $3,376$ frames are randomly selected in total. CASIA-3D [57] consists of $4,624$ scans of $123$ persons and the non-frontal faces are filtered out.

We divide $90\%$ of subjects as the training set FG3D-train and the remaining $10\%$ of subjects as the testing set FG3D-test. For the training set, we perform shape transformation as in Section 6.3 $4$ times to augment shape variations. Then, the out-of-plane pose variations are augmented by full-view augmentation as in Section 6.2, where the $yaw$ angle is enlarged at steps of $15^{\circ}$ until $50^{\circ}$ and the $pitch$ angle is enlarged by $15^{\circ}$ and $-25^{\circ}$, generating $474k$ samples in total. For the testing set, we only perform full-view augmentation and generate $12k$ samples. Besides, we manually delete the bad registration results in the FG3D-test manually for better evaluation. This dataset is employed for the model training and the performance analysis of our proposal. Some examples are shown in Fig. 10.

Florence [60] is a 3D face dataset containing 53 subjects with 3D meshes acquired from a structured-light scanning system. Based on the protocol in [9], each subject is rendered at pitches of $-20^{\circ}$, $0^{\circ}$, $20^{\circ}$ and yaws of $-45^{\circ}$ to $45^{\circ}$ at the step of $15^{\circ}$, as shown in Fig. 10. Besides, we register each 3D mesh with hand-labeled landmarks and carefully check the registration results. This dataset is used for cross-dataset evaluation to demonstrate the generalization.

NoW [61] contains 2054 face images of $100$ subjects, which are split into an open validation set ($20$ subjects) and a private test set ($80$ subjects). Each subject has face images with different poses and expressions, and one neutral 3D face scan for reference. Under the original protocol, the reconstruction results should be disentangled from the expression to compare with the neutral 3D scan, which is not consistent with our goal. Therefore, we only select the neural-expression images of the validation set, as shown in Fig. 10, for cross evaluation.

Stirling/ESRC [62] is a 3D face dataset with $134$ subjects ($64$ males and $70$ females). Each subject has a 3D face scan in a neutral expression and two corresponding face images in $yaw=\pm 45^{\circ}$, which are captured by the Di3D camera system simultaneously. In the experiments, all subjects are employed in the cross-dataset evaluation. Some examples are shown in Fig. 10.

It is still an open problem to determine the proper measurement of 3D shape accuracy. Errors calculated after projection [9, 10, 63] mostly account for pose accuracy since pose is the dominant factor of vertex positions [10]. To highlight shape accuracy, which is our main purpose, we normalize pose before error calculation. Specifically, we register a face template to the ground-truth 3D face as in Section 6.1, finding the vertex correspondence $(k,k^{t})$, where $k$ is the vertex index on the face template and $k^{t}$ is that on the ground truth. We also note $k\in\mathcal{C}$ if $k$ belongs to the face region and the correspondence $(k,k^{t})$ is reliable (the spatial and normal distances are below thresholds). For each reconstruction result $\mathbf{V}$ and the ground truth $\mathbf{V}^{*}$, we estimate a rigid transformation:

where $(f,\mathbf{R},\mathbf{t}_{3d})$ are the rigid transformation parameters for pose alignment. Based on this rigid alignment, we utilize the widely applied Normalized Mean Error (NME) to measure the reconstruction error:

where $K$ is the number of vertices, $\mathbf{v}_{k}$ is a vertex on the reconstructed face, $\mathbf{v}^{*}_{k^{t}}$ is the target of $\mathbf{v}_{k}$, and $d$ is the 3D outer interocular distance. This error is adopted in the performance analysis on the FG3D-test, where we trust the registration results.

We also employ a Densely Aligned Chamfer Error (DACE) to measure the distances of the closest points:

where $\mathbf{v}^{*}_{k^{nn}}$ is the nearest neighbor of $\mathbf{v}_{k}$ on the raw 3D scan after rigid alignment. We adopt this error in the comparison experiments on Florence, NoW and Stirling/ESRC, where the raw 3D scans are employed as the target and different methods may have different topologies. It is worth noting that, only the vertex in set $\mathcal{C}$, which indicates that its ICP matching is reliable, participates in the error calculation, which filters out the errors on the noisy, hollow and occluded regions.

The initial 3D face is acquired by 3DDFA [64]. The backbone follows the network defined in PRNet [65] and the shortcuts are added to generate an hourglass network. The models are trained by the SGD optimizer with a starting learning rate of $0.1$, which is decayed by $0.1$ at epochs $30$, $35$ and $40$, and the model is trained for $45$ epochs. The training images are cropped by the bounding boxes of the initial 3DMM fitting results [64] and resized to $256\times 256$ without any perturbation. The UV displacement map is also $256\times 256$. In all the following experiments, the FG3D-train is employed as the training set. The testing is conducted on the FG3D-test for performance analysis with Normalized Mean Error (NME), and several other datasets for comparison with the state-of-the-art methods, where Densely Aligned Chamfer Error (DACE) is preferred.

In this section, we thoroughly analyze how the network structure benefits the reconstruction accuracy.

In Section 6, we augment face appearance in both pose and shape to provide adequate data for neural network training. In pose augmentation, we improve face profiling [10] by completing the depth channel and inpainting the side face with a texture model. As shown in Table III, our full-view augmentation outperforms face profiling, especially in large poses, which is attributed to the artifacts exposed on the side face that face profiling cannot repair. Besides, by further incorporating shape transformation, the training data reaches $474$k samples and the error is greatly reduced by $15.7\%$. It can be seen that in high-fidelity reconstruction, where shape is mostly concerned, our shape transformation is a highly effective augmentation strategy. However, adding an overwhelming number of augmented samples may not benefit the performance as the shape-augmented samples are fake images. Therefore, we further add the shape-transformed samples progressively and observe how the performance changes. As shown in Fig. 14, the error reduction converges when approximately $265$k samples ($2.2$ times that of the original samples) are added.

In supervised learning, the loss function directly judges the reconstruction results according to the targets. Considering that the fine-grained geometry cannot be well captured by the traditional Mean Squared Error (MSE), we propose the Plaster Sculpture Descriptor (PSD) to model the visual effect and the Visual-Guided Distance (VGD) to supervise the training. We evaluate the loss functions by both Normalized Mean Error (NME) and Densely Aligned Chamfer Error (DACE), where the latter concerns more about the visual effect.

The results listed in Table IV indicate that: 1) Even with the ground-truth 3D shape as the supervision, intuitively adopting MSE cannot capture personalized shape well. 2) The introduction of PSD improves the accuracy by directly optimizing the visual effect. However, PSD cannot constrain the face contour due to its defect in back-propagation, which is discussed in Section 5.2. 3) The VGD loss further remedies the defects of PSD and captures the face contour well, achieving the best performance. 4) The unsatisfactory results achieved by the combination of PSD and VGD indicate that, it is better to regard PSD as the vertex weights than the loss function. There are also some examples in Fig. 15.

The PSD draws the 3D face as a plaster sculpture with white vertex color under frontal light, which is common in 3D structure demonstration. We also evaluate the performance of PSD by rendering the 3D face in different vertex colors and lighting conditions, finding little difference.