SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

Blanz and Vetter [11] proposed 3DMM to represent a face model by a linear combination of orthogonal bases obtained by PCA. In this way, 3D face reconstruction can be formulated as 3DMM parameters regression problems. Later, Paysan et al. [25] extended the model by adding more scans and decomposing the expression bases from shape bases. A lot of earlier methods regressed the 3DMM parameters by solving a nonlinear optimization function [26, 5, 25, 27, 28]. Thanks to the development of deep learning, some methods [29, 1, 30, 31, 12, 16, 32, 33] started to use deep convolutional neural network (DCNN) architectures to learn 3DMM parameters and largely replaced traditional optimization-based methods with more accurate results and shorter running time. The self-supervised training of DCNN-based methods was implemented by exploiting a differentiable renderer [34, 35, 36, 37, 38, 39, 40], which alleviated the lack of 3D-supervised data and improved the generalization of networks. However, these model-based methods’ reconstruction geometry is constrained by the linear bases with limited representation power.

Some works proposed to break the limitation by using nonlinear models [41, 42, 17, 20, 43, 41, 44]. In [19], a DCNN was used as a nonlinear 3DMM decoder. Ranjan et al. [45] used spectral graph convolutions to learn 3D faces. Zhou et al. [18] presented a nonlinear 3DMM using colored mesh decoding. Guo et al. [46] learn a more powerful nonlinear 3DMM from different data sources: scanned 3D face, RGB-D images, and RGB images.

Some other works directly obtained the full 3D geometry to avoid the restriction of parametric models and difficulty in pose estimation [47, 2, 45]. Jackson et al. [21] proposed to use a volumetric representation of 3D face shape instead of the previously used point cloud or mesh and directly regressed the voxels. Feng et al. [2] mapped the mesh of a face geometry into UV position maps and then trained a light-weighted network that obtains the 3D facial geometry along with its correspondence information.

Some works also combine the 3DMM-based regression and direct 3D geometry regression to improve the reconstruction performance. Chen et al. [48] used a 3DMM-based coarse model and a displacement map in UV space to represent a 3D face, and utilize the input image as supervision to effectively learn the facial details. Huber et al. [12] proposed to estimate an intermediate volumetric geometry and finetune it with 3DMM parameters’ regression.

In the beginning, face alignment works aimed to locate a set of 2D facial landmarks in the image plane. Traditional works were mainly based on Active Appearance Models (AMM) and Active Shape Models (ASM) [49, 50, 51, 52, 53] and considered face alignment as a model parameter optimization problem. Then cascaded regression methods [54, 55] became popular. They iteratively refined the predictions and reached higher accuracy. With the development of deep learning, CNNs were wildly used to regress the landmarks’ positions directly or predict heat maps and largely improved the performance [56, 57, 58, 59]. Since the 2D face alignment methods have limitations on detecting invisible landmarks, the 3D face alignment problem has been widely researched in recent years. There are two major strategies for 3D face alignment: (1) separately detecting 2D landmarks and their depth [60, 61, Bulat2016TwoStageCP], and (2) fitting a certain 3D face model [63, 64] to obtain the full 3D structure to guide the 3D landmark localization.

Since the methods mentioned above can handle only a limited quantity of landmarks, which is far from enough in some applications, 3D dense face alignment [65] started to be researched. It requires methods to offer pixel-wise facial region correspondence between two face images. As the prediction target changes from a sparse set of facial landmarks to a dense set of tens of thousands of points, it is natural to solve this problem by fitting a registered 3D face model.

Most model-based reconstruction approaches can be applied to dense face alignment if the face model is well registered and provides a dense correspondence [65, 17, 12]. Explicit facial pose estimation is needed in these works. Zhu et al. proposed 3DDFA [1], which is a representative work in this task. They used a cascaded CNN framework to regress the 3DMM parameters, including the pose parameters. The rotation angles are represented by a 4D unit quaternion for less difficulty in learning. However, their output faces are sensitive to the fluctuation of every parameter and hard to reach high precision. In  [12], an ICP post-processing is incorporated to refine the regressed pose parameters, but it takes enormous extra computation. Zhou et al. learned a nonlinear 3DMM by directly using graph convolutions on face meshes and reached an extremely fast decoding speed. Nevertheless, the face pose is still obtained by direct regression like 3DDFA.

Some model-free methods that directly predict 3D coordinates of facial points are also applicable to the task of dense face alignment with their face representation registered with fixed semantic meaning [2, 21]. However, it is difficult for these model-free methods to handle severe occlusions and large poses since no prior knowledge or constraint is provided.

Our method’s final output face geometry is represented by a UV position map, which possesses a dense correspondence between the face shape and the input image. Unlike the methods mentioned above, we explicitly deal with object occlusions by leveraging an attention mechanism on network features. We predict a pose-independent face model to avoid the large variance in face shape brought by large poses. Instead of directly regressing the pose parameters, we perform a visibility-aware self-alignment between a pose-dependent face model and a pose-independent face model to estimate the pose of a nonlinear 3DMM. The alignment process is stable as the error caused by a single outlier is apportioned by all of the landmarks. In contrast, the error of every pose parameter is accumulated in parameter-regressing methods. Besides, the alignment process is based on two regressed faces with similar shapes, rather than the matching of landmarks and a fixed face template. Therefore, the change of face shape is taken into account in pose estimation. In this way, our method is more robust and accurate than previous works.

Facial geometry representation. We assume the projection from the 3D face geometry to the 2D image is a weak perspective projection:

where $\mathbf{V}$ is the projected geometry on the 2D plane, $\mathbf{Pr}=\begin{bmatrix}~{}1&~{}0&~{}0\\ ~{}0&~{}1&~{}0\end{bmatrix}$ is the projection matrix, $\mathbf{G}\in\mathbb{R}^{3\times n}$ is the 3D mesh of a specific face with $n$ vertices.

We separate the 3D face geometry into pose, mean shape, and deformation as:

$\mathbf{S}\in\mathbb{R}^{3\times n}$ represents the pose-independent (i.e., pose-normalized) face shape, $\mathbf{\bar{S}}\in\mathbb{R}^{3\times n}$ is the mean shape template provided by [25] and $\mathbf{D}\in\mathbb{R}^{3\times n}$ is the deformation between $\mathbf{S}$ and $\mathbf{\bar{S}}$. The pose parameters consist of the scale factor $f$, the 3D rotation matrix $\mathbf{R}\in\mathbb{R}^{3\times 3}$ and the 3D translation $\mathbf{t}\in\mathbb{R}^{3}$.

Dual face regression and analysis. We propose to jointly regress two face models: a pose-independent face (the face shape) $\mathbf{S}$ and a pose-dependent face $\mathbf{P}$. And then, we use a self-alignment post-process $\phi$ to estimate face pose from $\mathbf{P}$, $\mathbf{S}$ and facial vertices’ visibility information $\mathbf{Vis}$:

The visibility information $\mathbf{Vis}$ is explained in Sec.III-B. Based on the estimated face pose $\phi(\mathbf{P},\mathbf{S},\mathbf{Vis})$ and pose-independent face $\mathbf{S}$, we could reconstruct our final face shape $G$ via transformation defined in Eq.2.

In fact, $\mathbf{G}$ and $\mathbf{P}$ have the same physical meaning, which means the ground truth of them are the same:

The difference is that $\mathbf{P}$ is obtained by direct network regression, while $\mathbf{G}$ is obtained by applying Eq.2. To distinguish them, we term $\mathbf{P}$ as pose-dependent face and $\mathbf{G}$ as face geometry. The learning of pose-dependent face is easy to over-fit to pose and under-fit to shape as the orientation variations bring much greater point-to-point distances than the shape variations, resulting in some implausible face shape under large pose cases. By contrast, the pose-independent face $\mathbf{S}$ does not change with the pose. Thus the network would focus on the shape characteristics and learn more details. $\mathbf{S}$ is disentangled into the mean face shape template $\mathbf{\bar{S}}$ and the deformation $\mathbf{D}$ between the actual shape and the mean shape. Only the zero-centered $\mathbf{D}$ needs to be predicted. It further reduces the fitting difficulty. The mean face shape also serves as prior knowledge to keep the invisible facial parts plausible. By combining the shape of $\mathbf{S}$ with the pose $\phi(\mathbf{P},\mathbf{S},\mathbf{Vis})$ estimated from $\mathbf{P}$ and $\mathbf{S}$, we are able to get a much better $\mathbf{G}$, as demonstrated in experiments.

UV map representation. The face geometry $\mathbf{G}$, pose-independent face $\mathbf{S}$, mean face $\mathbf{\bar{S}}$, deformation $\mathbf{D}$, and pose-dependent face $\mathbf{P}$ are transformed into UV space [2] as UV maps. UV map $\mathbf{U}$ is a 2D representation of 3D vertices on the face mesh model. It can be expressed as

where $(x_{i},y_{i},z_{i})$ is the 3D coordinate of vertex $i$ on the 3D face mesh and $(u_{i},v_{i})$ is the corresponding 2D UV coordinate.

The mapping relationship between the 3D object coordinate $(X_{i},Y_{i},Z_{i})$ and the UV coordinate $(u_{i},v_{i})$ of a facial vertex $i$ can be formulated as

where $\alpha_{1}$, $\alpha_{2}$, $\beta_{2}$, $\beta_{1}$ are scaling and translation constants. The mapping relationship is computed on the mean face mesh from the Basel Face Model (BFM) [25] and applied to all the face meshes. In this way, the points in the UV map are registered to the face mesh model. Fig. 3 illustrates the mapping for a better understanding. This representation guarantees the spatial consistency between face model and UV map, i.e., spatially neighboring points on the face model are neighboring in the UV map. Since 2D UV maps can be processed by sophisticated CNNs, this ensures a great potential of the representation in applications of unconstrained situations. In the remainder of this paper, the face geometry, pose-independent face, mean face, deformation, and pose-dependent face are represented as UV maps. For clarity, the same notation is used for a thing in the two spaces (e.g., $\mathbf{G}$ is used to denote both the face geometry and the UV map of it).

Network architecture. Our self-aligned dual face regression network is an encoder-decoder-based architecture that regresses the deformation $\mathbf{D}$ and infers the pose parameters $f$, $\mathbf{R}$ and $\mathbf{t}$ to reconstruct the 3D face geometry from a single 2D face image. It consists of three sub-networks: encoder, attention side branch, and decoder.

The encoder network contains a sequence of residual blocks that first extract low-level features, which are then fed into the attention side branch. The attention network generates an attention mask $\mathbf{A}$ with the visible face region highlighted. Then encoder network further encodes the attended features into high-level features. The decoder then decodes them into the pose-dependent face $\mathbf{P}$ and the deformation $\mathbf{D}$. The face shape $\mathbf{S}$ is obtained by Eq. 3. The pose parameters $f$, $\mathbf{R}$ and $\mathbf{t}$ are then obtained from $\mathbf{P}$, $\mathbf{S}$ and $\mathbf{A}$ in the self-alignment module. The final face geometry $\mathbf{G}$ is generated by applying Eq. 2.

To deal with the occlusions, we adopt an attention mechanism to extract features mainly from the visible face regions in the input images. This sub-network is a side branch consisting of five convolutional layers. It outputs a soft attention mask assigning high values to the pixels corresponding to the visible regions and low values to the pixels in the occluded regions and background region. The attention is applied to the low-level features to make a trade-off between resolution and accuracy.

Considering the attention mask may be inaccurate in some cases, we do not discard the information of the regions with low attention entirely. We instead highlight the features of the visible face regions according to the attention values. This operation can be formulated as

where $\mathbf{F_{a}}$ denotes the obtained weighted feature map, $\mathbf{F_{l}}$ is the low-level feature map, and $\mathbf{A}$ is the attention mask.

Once we get the attention mask, we are able to estimate the visibility of the vertices in the pose-dependent face through

where the 3D position of vertex $i$ is $(x_{i},y_{i},z_{i})$. The normal direction of vertex $i$ that can be obtained from its adjacent vertices is $(n_{x},n_{y},n_{z})$. The vertex visibility is employed in the self-alignment module and is discussed detailedly in Sec. III-C.

Since no database has the ground truth for face occlusion annotation, we simulate occlusions through data augmentation. Specifically, we project the 3D face geometry to the image plane to generate a binary map that indicates the full face region. Then we overlay patterns with random shapes to the input image and use them as the occlusions as shown in Fig. 4. We use the resulting binary image as the ground truth of the attention mask. In this way, we can obtain the ground truth data for the training of the attention branch. Although occlusions in the real world can be very diverse, our experiments find the proposed method has the ability to predict real-world occlusions as shown in Fig. 5.

This module aims to extract the pose information from the pose-dependent face $\mathbf{P}$ and the shape information from the pose-independent face $\mathbf{S}$ (see Fig. 2 for the two faces $\mathbf{P}$ and $\mathbf{S}$). It is achieved by estimating the similarity transformation matrices between $\mathbf{P}$ and $\mathbf{S}$ (i.e., $f$, $\mathbf{R}$, and $\mathbf{t}$ in Eq. 2). The similarity transformation matrix is estimated using two sets of correspondent landmarks, $\mathbf{K_{S}}$ and $\mathbf{K_{P}}$, extracted from $\mathbf{S}$ and $\mathbf{P}$, respectively. The pixels with the same coordinates in the UV maps of $\mathbf{S}$ and $\mathbf{P}$ are semantically correspondent (e.g., the nose tip corresponds to two locations with the same coordinates in the UV maps of $\mathbf{S}$ and $\mathbf{P}$). Thus, in our method, $\mathbf{K_{S}}$ and $\mathbf{K_{P}}$ are extracted based on the same UV coordinates set. Specifically, we extract the 68 landmarks as the same as [1] for $\mathbf{K_{S}}$ and $\mathbf{K_{P}}$.

We do not simply use all the 68 landmarks. We propose to use more reliable facial vertices, i.e., the visible landmarks, for the alignment. We define a diagonal matrix $\mathbf{W}\in\mathbb{R}^{k\times k}$ that represents the weight of each landmark with the visibility. The weight of the $i$th landmark is formulated as

where $\mathbf{Vis}(i)$ denotes the visibility of the $i$th landmark as mentioned earlier. $eps$ is set to $0.1$ in our implementation to avoid the divide-by-zero error caused by the estimated visibility of all landmarks being zero. The estimation of the similarity transformation goes as the following steps: we first estimate $f$, and then estimate $\mathbf{R}$ and $\mathbf{t}$ using the singular value decomposition method. Specifically, We first compute the two weighted centroids $\mathbf{M_{S}}$ and $\mathbf{M_{P}}$ of the two sets of landmarks $\mathbf{K_{S}}$ and $\mathbf{K_{P}}$ by

Then we obtain $f$ by

After that, we normalize $\mathbf{K_{S}}$ and $\mathbf{K_{P}}$ as bellow:

Performing SVD to $\mathbf{H}=\mathbf{K_{S}^{{}^{\prime}}*\mathbf{W}*\mathbf{K_{P}^{{}^{\prime}\mathrm{T}}}}$, we have $[U,\Sigma,P]=\mathrm{SVD}(\mathbf{H})$, the rotation matrix $\mathbf{R}$ between $\mathbf{K_{S}^{{}^{\prime}}}$ and $\mathbf{K_{P}^{{}^{\prime}}}$ can be obtained by

and $\mathbf{t}$ can be obtained by

The pose estimation from landmarks requires only a small amount of computation and has high accuracy because the estimation is performed on a number of reliable landmarks on the two regressed faces.

The loss of our SADRNet consists of 6 components, the face geometry loss $\mathcal{L}_{G}$, the deformation loss $\mathcal{L}_{D}$, the pose-dependent face loss $\mathcal{L}_{P}$, the attention mask loss $\mathcal{L}_{A}$, the edge length loss $\mathcal{L}_{E}$, and the normal vector loss $\mathcal{L}_{V}$. Since the significance of the vertices in different face regions may differ considerably, we adopt a weight mask $\mathbf{M}$ as [2] to $\mathcal{L}_{D}$, $\mathcal{L}_{P}$, and $\mathcal{L}_{G}$. We adjust the weight ratio of the four sub-regions for better training results. We use 16:12:3:0 for sub-region1 (the landmarks): sub-region2 (the eyes, nose, and mouth): sub-region3 (the cheek, chin and forehead): sub-region4 (the neck).

$\mathcal{L}_{G},\mathcal{L}_{D}$, and $\mathcal{L}_{P}$ can be obtained by the weighted average Euclidean distance between the estimated value and the ground truth, i.e., generalized as

$N$ denotes any one of the face geometry $\mathbf{G}$, the pose-dependent face $\mathbf{P}$, and the shape deformation $\mathbf{D}$. $h$ and $w$ are the height and width of the maps. $\mathbf{N}(u,v)$ denotes the estimated 3D coordinates of the vertex at the UV location $(u,v)$ in the UV maps of $\mathbf{G}$, $\mathbf{P}$, or $\mathbf{D}$. The symbol $\mathbf{\hat{~{}}}$ denotes the corresponding ground truth. $\mathbf{M}(u,v)$ denotes the weight value at the pixel location $(u,v)$ in $\mathbf{M}$.

We use binary cross entropy (BCE) between the predicted attention mask $\mathbf{A}$ and the ground truth $\mathbf{\hat{A}}$ to compute $\mathcal{L}_{A}$.

The edge length loss $\mathcal{L}_{E}$ is defined based on the lengths of the edges in the pose-independent face $\mathbf{S}$ as

$\mathcal{E}$ is the edges set. It is composed of every pair of adjacent vertices $(i,j)$ in the UV map. $(u_{i},v_{i})$ is the UV coordinates of vertex $i$. $\mathbf{E}_{ij}$ is the edge vector.

The normal vector loss $\mathcal{L}_{V}$ is defined as

where $\mathcal{T}$ is the triangle facets set, $\mathbf{n}_{i,j,k}$ is the normal vector of the triangle facet $(i,j,k)$. The edge length loss and the normal vector loss are used to generate better-looking face models.

The entire loss function of SADRNet is given by

where $\beta_{G}$, $\beta_{D}$, $\beta_{P}$, $\beta_{A}$, $\beta_{E}$, $\beta_{V}$ are respectively set to 0.1, 0.5, 1, 0.05, 1, 0.1 in our implementation.

Our network takes cropped-out $256\times 256\times 3$ images as the input, regardless of the effect of the face detector. The network starts with a single convolution layer followed by a low-level feature extractor that consists of 6 residual blocks [66] and outputs a $32\times 32\times 128$ feature map. The attention sub-network contains 5 convolution layers and a sigmoid activation. The high-level feature extractor contains 4 residual blocks and outputs a $8\times 8\times 512$ feature map. The decoder starts with 10 transpose convolution layers that up-sample the feature map to $64\times 64\times 64$, followed by 7 transpose convolution layers to output a UV map of the face shape with size $256\times 256\times 3$ and another 7 layers to output a UV map of the pose-dependent face with size $256\times 256\times 3$.

We follow [1, 2, 12] and use the full 300W-LP [1] as the training set. 300W-LP contains $122,450$ face images generated from 300W [1] by 3D rotation around the y axis and horizontal flipping. Since there is no sample with a larger than 90 degrees yaw angle in 300W-LP, we generate 5,000 samples with yaw angles range from 90 to 105 degrees by 3D rotation to supplement the dataset. Similar to [2], we augment the training data by randomly rotating the image from -90 to 90 degrees, translating in the range of 0 to 10 percent of input size, scaling the image size from 0.9 to 1.1, and scaling the color channel separately from 0.6 to 1.4. We also generate synthetic occlusions as explained in Sec. III-B. We use Adam optimizer with a gradual warm-up strategy [67]. We start from a learning rate of 1e-5 and increase it by a constant amount at each iteration. After 4 epochs, the learning rate reaches 1e-4. Then we use an exponential scheduler that decays the learning rate by 0.85 every epoch. The batch size is set to 16. We train our network for 25 epochs.

The original annotations of 300W-LP are 3DMM parameters, so the ground truth of $\mathbf{D}$ can be generated by computing $\mathbf{D}=\sum_{i}\alpha_{i}\mathbf{A}_{i}$, where $(\alpha_{1},\alpha_{2},\dots)$ are the shape parameters, $(\mathbf{A}_{1},\mathbf{A}_{2},\dots)$ are the shape basis vectors (including the identity and expression basis vectors).

AFLW2000-3D [1] is an in-the-wild dataset with large variations in pose, illumination, expression, and occlusion. There are 2,000 images annotated with 68 3D landmarks and fitted 3DMM parameters to recover the ground truth face model. We evaluate both face alignment and 3D face reconstruction performance on this dataset.

Florence [24] is a publicly available database of 53 subjects. The ground truth annotations are meshes scanned by a structured-light system. Similar to [21, 2, 12], each face mesh is rendered in 20 poses: a pitch angle of -15, 0, 20, or 25 degrees and a yaw angle of -80, -40, 0, 40, and 80 degrees to generate the face images. We evaluate the 3D face reconstruction performance on this dataset.

We employ normalized mean error (NME) as the evaluation metric. It is defined as the normalized mean Euclidean distance between each pair of corresponding points in the predicted result $\mathbf{p}$ and the ground truth $\mathbf{\hat{p}}$:

For a fair comparison, the normalization factor $d$ of the compared methods is computed in the same way. For sparse and dense alignment, the normalization factor of NME is defined as $\sqrt{h*w}$ where $h$ and $w$ are the height and width of the bounding box of all the evaluated points.

We evaluate the performance of 2D and 3D sparse alignment on the point set of 68 landmarks. We evaluate the performance of 2D and 3D dense alignment on the point set of around 45K points selected from the largest common face region of different methods as in  [2]. Following the settings of previous works [2, 1, 18] on 2D sparse alignment, we also test the images with yaw angles in $[0^{\circ},30^{\circ})$ (1,306 samples), $[30^{\circ},60^{\circ})$ (462 samples) and $[60^{\circ},90^{\circ}]$ (232 samples) separately, as well as the balanced subsets of each angle interval.

The quantitative results on AFLW2000-3D are shown in Table I. The results of other methods are from published papers or produced by the official open-source codes. We can see our method is superior to the other methods on most metrics. Especially on 3D and dense alignment tasks, our method has taken a clear lead. SPDT [23] generates rendered large pose training samples and uses a CycleGAN [22] to transform the rendering style. The large pose samples in our training database 300W-LP are mostly obtained by 3D image rotation with large distortion. We argue that they outperform us for sparse alignment in-between 60 and 90 degrees is because of the quality gap of training data.

Visualized results of challenging samples in AFLW2000-3D are demonstrated in Fig. 6, in which various degrees of occlusion and different face orientations are evaluated. We compare our method with two representative methods: MGCNet [40] and PRN [2]. MGCNet is a model-based method that fits the shape and the pose parameters of 3DMM by CNN. PRN is a model-free method that directly infers 3D coordinates of face mesh vertices with the UV position map. Our pose estimation method based on the alignment using visibility is more accurate and robust than previous works.

On the samples where the faces are partially occluded, as shown in Fig. 6c and Fig. 6d, the inaccuracy of the estimated poses of MGCNet leads to misalignment. And under severe occlusion, e.g., Fig. 6e, MGCNet fails to obtain a reasonable result, while our method is still able to precisely estimate the head pose. PRN jointly obtains pose and shape and works well for the visible facial region. However, PRN tends to give results with larger errors for invisible regions (see the misaligned landmarks in Fig. 6b and Fig. 6g). It is worth noting that AFLW2000-3D [1] is semi-automatically annotated and has some inaccurate annotations in some cases. In Fig. 7, we demonstrate some examples from AFLW2000-3D that our predictions have relatively larger NME but are apparently more accurate than the ground truth. This may narrow our method’s superiority margin.

On this task, we use NME normalized by 3D outer interocular distance as the evaluation metric. We follow the settings of  [2] to evaluate our method on AFLW2000-3D [1] and Florence [24]. For AFLW2000-3D, we use the same set of points as we do for dense face alignment. The results under different yaw angles are also reported. As for Florence, we choose the face region containing 19K points. We use iterative closest point (ICP) to align the results to the corresponding ground truth as in  [2]. Table II gives the quantitative comparison. Our method is robust to pose variations and achieves around 13% improvement over the state-of-the-art method on both datasets.

As our method employs a nonlinear shape deformation decoder that is more powerful than the linear bases based representation of MGCNet, our network can reconstruct more shape variations. It is worth mentioning that the output space of PRN is also nonlinear, but its large space, caused by entangled shape and pose, increases the learning difficulty for the high-frequency details. It is shown in Fig. 6 that eyes, lips, and noses of the face models reconstructed by our method have more details than PRN. Our attention-aware mechanism also contributes to the robustness toward the occlusions. In Figs. 6b-e, our reconstructed faces maintain natural appearances while those from PRN are distorted in occluded areas.

On this task, we use MAE (mean absolute error) of the head pose parameters as the evaluation metric:

where $\mathbf{p}$ represents the predicted pose parameters and $\mathbf{\hat{p}}$ represents the ground truth.

In Table III, we compare our proposed self-aligned module with the SOTA head pose estimation methods. The actual output of our method is the transformation matrix. So we convert the rotation matrix to Euler angles to calculate MAE with the ground truth. However, when the yaw angle is close to 90 degrees, the angle conversion suffers a serious gimbal lock problem. Some examples are shown in Fig.8. Their yaw angles’ errors are low, and the faces’ orientations seem nice, but the errors of the pitch and roll angles are extremely high. It will make the quantitative evaluation result worse than the actual performance. Thus, besides the initial evaluation results, we also report a result marked as SADRNet-fix that ignores the samples with more than 20 degrees pitch or roll error and less than 5 degrees yaw error. However, the negatively biased performance of our method (SADRNet) is still equivalent to the best standalone head pose estimation method.

We conduct ablation experiments on AFLW2000-3D to analyze the effectiveness of the following three modules: 1) the attention mechanism, 2) the self-alignment module, and 3) the regression via the shape deformation. The experiments are conducted on three tasks: sparse alignment, dense alignment, and 3D face reconstruction. The results are summarized in Table IV.

Our method has a model size of 60MB (including the mean face template parameters). In Table V we compare the model sizes of the proposed method and the baseline methods. Our model is lighter than all other deep-learning-based methods. The network inference for one face image takes 12ms on a GTX 1080 Ti GPU. The self-alignment post process (i.e., the step generating $\mathbf{G}$ from $\mathbf{S}$ and $\mathbf{P}$) takes 4ms on GPU or 1.5ms on an Intel Xeon E5-2690 CPU @ 2.60GHz. The fastest implementation of our method reconstructs the 3D face model from a cropped image in up to 13.5 ms.

The backbone of our method, (i.e. the framework in Fig.9.a) has a model size of 52MB. The extra size introduced by the attention side branch is 7MB. The parameters of the 7-layer pose-dependent face decoder and the 7-layer pose-independent face decoder add up to 1MB.

We note the following limitations of our work:

• •

  Our network only regresses the shape geometry and pose, but does not reconstruct the facial texture from the input image.

• •

  The learning of the proposed SADRNet is fully supervised and depends on the costly face mesh annotation. Designing a weakly supervised architecture that can utilize additional data modalities (e.g., facial keypoint detection data, silhouette data, segmentation data) may improve the application potential.

• •

  There is still much room for improvement in the reconstruction of high-frequency facial details (i.e. the pores, blemishes, and wrinkles) in our work. We believe the improvements can be made in two aspects. First, the training dataset we currently use is labeled with the parameters of 3DMM. The ground truth face models lack high-frequency details. Using datasets with high-fidelity ground truth models may help the detail synthesis. Considering that such training data is expensive, a self-supervised architecture that utilizes the facial details in the input image as supervision may be another feasible way. Second, although we have regressed the face shape deformation separately in the framework, the high-frequency details are relatively minor compared to the deformation and easy to ignore in learning. Further decomposing the shape and details or iteratively updating the face shape with cascaded regressors can more adequately supervise the learning of high-frequency details.

We leave these limitations as our future work.