Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network

The proposed BALI fields have a composite structure. They consist of a scalar component for confidence (boundary heatmaps as shown in Fig. 3 (b)) and a vector component that points to the closest landmark in this boundary (as shown in Fig. 3 (c) and (d)). The proposed BALI fields need to estimate the confidence $B^{ij}_{c}$ and a vector $(B_{u}^{ij},B_{v}^{ij})$ at every output location $(i,j)$, which can be expressed as follows:

where $(B_{u}^{ij},B_{v}^{ij})$ is calculated within a square region with edge length $2R+1$ and centered at the ground-truth landmark location $(\hat{u},\hat{v})$. $B_{c}^{ij}$ denotes the boundary heatmap which is constructed according to MMDN [15]. To be specific, for each boundary, landmarks on this boundary are firstly interpolated to get a dense boundary line. Then, a Gaussian distribution is used to construct a ground-truth boundary heatmap by transforming distance map $Dist$ that is obtained with a binary map and distance transform function. The boundary heatmap is constructed as follows:

where $\xi$ is a small constant, $\sigma$ denotes the standard deviation of the corresponding Gaussian distribution and $\hat{B}$ denotes the ground-truth boundary heatmap. So far, the BALI fields have been constructed, however, how to utilize the BALI fields to detect more accurate landmarks is still an unsolved problem. In this paper, we address the above problem with the help of landmark heatmaps, i.e., we generate the landmark heatmap and the BALI fields at the same time. Therefore, by optimizing landmark heatmaps and boundary heatmaps (contained in the proposed BALI fields) in a multi-task way, the facial boundary constraints can be introduced to generate more effective landmark heatmaps. Then, we further use the field constraints to obtain more precise landmark coordinates. To be specific, we firstly compute the coarse landmark coordinates by using $\arg\mathop{\max}\limits_{i,j}\left(H\right)=\left({i^{\prime},j^{\prime}}\right)$, where $H$ denotes the predicted landmark heatmap. Then, we crop a small square region from the landmark heatmap with edge length $2r+1$ center at $(i^{\prime},j^{\prime})$, which is denoted by $H^{\prime}$. Finally, the soft-argmax operation is utilized on $H^{\prime}$ to calculate the final landmark coordinates. The whole process can be formulated as follows:

where $(\bar{u},\bar{v})$ denotes the final predicted landmark coordinates. Therefore, by designing Eq. (5), both boundary and field constraints can be introduced to detect more accurate landmarks.

The proposed Boundary-Aware Landmark Intensity (BALI) fields can introduce both boundary and field constraints that would be helpful for detecting more precise landmarks, however, how to generate more accurate and effective BALI fields and landmark heatmaps are still open questions. The heatmap regression-based facial landmark detection methods [30, 31, 12, 13, 15] can generate effective landmark heatmaps and have achieved state-of-the-art performance as they can effectively encode the part constraints and context information. However, these methods suffer from performance degradation when facing large poses and heavy occlusions, because 1) the large pose or occlusions will mislead models’ learning of robust features and accurate facial shape constraints. 2) backpropagated gradients diminish in strength as they are propagated through a very deep network. Therefore, in this paper, a Self-Calibrated Pose Attention (SCPA) model is proposed to address the above problem by introducing a self-calibrated mechanism and a pose attention mask. The self-calibrated mechanism is able to provide intermediate supervision and address the gradient vanishing problem by optimizing the $L_{2}$-loss of the generated heatmaps between paired images, and the pose attention mask can drive the network to focus on part of interest by using the newly generated heatmaps as attention for learning more representative and discriminative features. Therefore, the SCPA model is able to produce more accurate BALI fields and landmark heatmaps and help achieve more robust and precise facial landmark detection. The network structure of the proposed SCPA model is shown in Fig. 2.

The proposed BALI field is able to achieve highly precise landmark detection by modeling both boundary and field constraints. Then, the SCPA model can learn more representative and discriminative features by introducing the self-calibrated mechanism and the pose attention mask. Finally, by integrating the SCPA model and BALI fields into a Self-Calibrated Pose Attention Network (SCPAN), we can generate more accurate and effective landmark heatmaps and boundary-aware landmark intensity fields for achieving more robust and precise facial landmark detection. The overall network structure of the proposed SCPAN is shown in Fig. 2.

The proposed SCPAN outputs landmark heatmaps and BALI fields, therefore, the loss between the generated landmark heatmaps and BALI fields and the ground-truths should be used as part of the objective function. Moreover, the loss between the paired images and the loss between the predicted landmark coordinates and the ground-truth coordinates should be a part of the final objective function. Therefore, the objective function can be formulated as follows:

Original loss. Generally, the Mean Square Error (MSE) loss is selected for optimizing models [38, 30, 39, 40, 41] to generate landmark heatmaps. However, the MSE loss treats each pixel in the heatmap equally, which easily leads to blurred heatmaps and reduces the detection accuracy. Fortunately, MMDN [15] and MMHN [13] have shown that the Jensen-Shannon divergence loss can pay more attention to the foreground area of heatmaps instead of treating the whole heatmap equally, thus accurately measuring the difference between two distributions. Hence, the Jensen-Shannon Divergence loss is also selected as the objective function to calculate the distribution differences between the generated heatmaps and the ground-truths. The Jensen-Shannon Divergence loss is expressed as follows:

where $i$ and $j$ denote the indexes of a pixel in the heatmap and $KL$ means the Kullback-Leibler divergence. $p_{G}$ and $p_{\hat{G}}$ denote the probability distributions of the generated and ground-truth heatmaps. Base on the above Jensen-Shannon Divergence loss, the original loss can be reformulated as follows:

where $t$ denotes the stage, and $T=4$. $SE(B_{u,v})$ means cropping the corresponding area (i.e., a square region with edge length $2r+1$ center at the ground-truth landmark location) from $B_{u}$ and $B_{v}$.

Self-calibrated loss. The proposed self-calibrated mechanism could provide intermediate supervision by optimizing the loss of paired images. We also use the Jensen-Shannon Divergence loss as the objective function, and the self-calibrated loss can be reformulated as follows:

That means in each SCPA model, the self-calibrated loss is introduced to provide intermediate supervision and produce more effective pose attention masks for learning more discriminative representations.

Coordinate loss. The loss between the predicted landmarks and the ground-truths can also be used to optimize the proposed SCPAN, which further help obtain more accurate landmarks. The coordinate loss can be formulated as follows:

where $\phi$ denote the landmark index. The coordinate loss is able to help better integrate the boundary and field constraints and produce more effective landmark heatmaps and BALI fields for detecting more accurate landmarks.

Overall loss. By combining $\mathbb{L}_{org}$, $\mathbb{L}_{scl}$ and $\mathbb{L}_{coor}$, we can obtain the overall loss, which can be formulated as follows:

With the above overall loss function, the original label information can be better used to improve the detection accuray.

SCPAN is able to detect more accurate landmarks for faces with large poses and heavy occlusions by incorporating the proposed BALI fields and SCPA model. However, its performance still depends on large-scale training samples, i.e., high-resolution face images and their landmark annotations. Although it is easy to collect high-resolution face images, annotating them is more expensive and tedious. The proposed self-calibrated mechanism can provide a self-learned objective function that enforces intermediate supervision by utilizing unlabeled face data. In this way, more facial prior knowledge can be learned to enhance the detection accuracy of the proposed SCPAN (i.e., semi-SCPAN). Suppose there are $N$ labeled face images and $M$ unlabeled ones, the objective function of semi-SCPAN can be formulated as follows:

where ${{\mathbb{L}^{\prime}}_{scl}}$ denotes the self-calibrated loss corresponding to $M$ unlabeled images and ${{\mathbb{L}^{\prime}}_{scpan}}$ represents the final objective function of semi-SCPAN that is designed by utilizing both labeled and unlabeled data. With the above new objective function, the SCPAN is able to use unlabeled face images to further boost the performance of facial landmark detection. We also present the main steps of the proposed Semi-Supervised SCPAN in Alogrithm I.

300W (68 landmarks): The training set of 300W is composed of the training set of AFW [42], LFPW [43] and Helen [44], which contains 3148 face pictures. The testing set of 300W includes IBUG and the testing set of LFPW and Helen, which can be further divided as follows: 1) Challenging subset (i.e., IBUG dataset [18]). It contains 135 more general “in the wild” images, and experiments for this dataset are more challenging. 2) Common subset (554 images, of which 224 images are from LFPW test set and 330 images from Helen test set). 3) Fullset (689 images, composed of the challenging subset and common subset). Moreover, following LUVLI [12] and KDN [45], we perform cross dataset evaluation on 300W dataset, i.e., We first train the SCPAN on 300W-LP dataset [46], and then fine-tune on the trainset (3837 samples). We evaluate the SCPAN on 300W testing set, which contains 600 images.

Menpo 2D (68 landmarks): It consists of images from AFLW and FDDB [47], which are re-annotated following 68 landmark annotation scheme. It has two subsets, frontal faces (6679 samples which have 68 annotations) and profile faces (300 samples which have 39 landmark annotations). We use the frontal set for cross dataset evaluation.

COFW (68 landmarks): It is another very challenging dataset on occlusion issues which is published by Burgos-Artizzu et al. [20]. It contains 1345 training images of which 845 images are from LFPW and the others are with heavy occlusions. The testing set includes 507 face images with large variations on the head pose, facial expression and occlusion. We use the testing set for cross dataset evaluation.

AFLW (19 landmarks): It contains 25993 face images which has extremely large variations of jaw angles ranging from ${\rm{-}}{120^{\circ}}$ to ${\rm{+}}{120^{\circ}}$ and pitch angles ranging from ${\rm{-}}{90^{\circ}}$ to ${\rm{+}}{90^{\circ}}$. Moreover, it also contains very complicated occlusions. AFLW-full selects 24386 images from the whole AFLW dataset and further divides them into two parts: 20000 for training and 4386 for testing. Moreover, 1165 images (i.e., AFLW-frontal) are selected from AFLW-full testing set to evaluate the alignment algorithm on frontal faces.

WFLW (98 landmarks): It has 98 landmark annotations and images in WFLW are collected from more complicated scenarios. The training set contains 7500 images and its testing set includes 2500 images. WFLW also has other attribute annotations, including occlusion, pose, makeup, lighting, blur, and expression, which can help more comprehensively evaluate existing alignment algorithms.

300VW (68 landmarks): Following Shen et al. [22], we use 50 videos to train our SCPAN and test it on the rest 64 videos. The testing set is divided into three parts: well-lit (Scenario1, various head poses, occlusions such as glasses and beards), mild unconstrained (Scenario2, different illuminations, dark rooms, overexposed shots and arbitrary expressions) and challenging (Scenario3, illumination conditions, occlusions, make-ups, expressions and head poses) according to the difficulties.

Evaluation Metrics. Facial landmark detection results are envaluated with Normalized Mean Error ($\rm NME_{box}$ [48, 49], $\rm NME_{diag}$ [50, 14] and $\rm NME_{io}$ [18, 51]), Area Under the Curve (AUC) [12, 13] and Failure Rate (FR) [52, 51]. The $\rm{NME}$ is defined as follows:

where $\phi$ denotes the landmark index, $(\bar{u},\bar{v})$ and $(\hat{u},\hat{v})$ represent the predicted and ground-truth landmark coordinates, respectively. $d$ denotes the normalization term, which can be set to the interpupil distances, the interocular distance, the distance between the outer corners of the two eyes, the geometric mean of the width and height of the ground-truth bounding box and the diagonal of the tight bounding box for $\rm NME_{ip}$, $\rm NME_{io}$, $\rm NME_{box}$ and $\rm NME_{diag}$, respectively. To compute AUC, we firstly plot the cumulative distribution of the fraction of test images whose NME (%) is less than or equal to the value on the horizontal axis, then the AUC is computed as the area under that curve. FR means the percentage of images in the test set whose NME is larger than a certain threshold.

Implementation Details. In our experiments, all the training and testing images are cropped and resized to 256x256 according to the provided bounding boxes. To generate the disturbance, we use the spatial transformation and texture disturbance operations. Specifically, the spatial transformation contains rotation ($-60^{\circ},+60^{\circ}$), scaling ($0.5,1$), mirror flip and their combinations. The texture disturbance includes occlusion, blurring and noises. The occlusion disturbance is achieved by using two types of occlusions. The first one is occlusion by black, i.e., the occluded area is covered by black, and the second one is occlusion by part of the original face image. The blurring disturbance means downsampling high-resolution images ($256\times 256$) into low-resolution ($128\times 128$, $64\times 64$, $32\times 32$, $16\times 16$ ) ones with bicubic degradation. We use the Stacked Hourglass Network [38] as our backbone to construct the proposed Self-Calibrated Pose Attention Network, and the spatial size of the output heatmap and field is $128\times 128$. $\lambda_{1}$, $\lambda_{2}$, $\gamma$ and $\eta$ are set to 1, 16, 40 and 4, respectively. $11\times 11$ and $7\times 7$ field regions are used in the training and testing phases, respectively. The training of SCPAN takes 200000 iterations and the staircase function is used to set the learning rate. The initial learning rate is $2.5\times{10^{{\rm{-4}}}}$ and then it is divided by 5, 2 and 2 at iteration 10000, 40000 and 100000, respectively. The SCPAN is trained with Pytorch on 8 Nvidia Tesla V100 GPUs.

Experiment Settings. To evaluate the effectiveness of each module proposed in this paper, we firstly use the Stacked Hourglass Network (SHN) [38] as the baseline, and then we separately construct SHN+BALI and SHN+SCPA by combining SHN with the proposed Boundary-Aware Landmark Intensity (BALI) fields and Self-Calibrated Pose Attention (SCPA) model, respectively. Moreover, we combine SHN with both BALI fields and SCPA model to construct SCPAN (i.e., SHN+BALI+SCPA). Then, as SCPAN can be further boosted by utilizing unlabeled face images (in this paper, we use the CelebA [53] dataset) and the boosted SCPAN is denoted by semi-SCPAN. For CelebA, we use 169854 images for training. To be specific, we firstly use OpenFace [54] to detect 68 landmarks and then obtain the corresponding input images with cutting and scaling operations according to the detected landmarks. For other state-of-the-art methods [55, 30, 56, 35, 57, 12, 58], we either use the original codes released by the authors or restore the experiment, and the results both have achieved the expected effects in the corresponding papers. The detailed comparison is shown below.

For benchmark datasets such as 300W, Menpo 2D, COFW, AFLW and WFLW, faces in 300W common subset, 300W full set, Menpo 2D and AFLW-frontal dataset are closer to neutral faces and have smaller variations on the head pose, facial expression and occlusion. Hence, we evaluate the effectiveness of the proposed SCPAN method under normal circumstances with these four subsets. Table I includes the $\rm NME_{io}$ and $\rm NME_{ip}$ comparisons of state-of-the-art face alignment methods on 300W common subset and 300W full set. Moreover, Table II shows the $\rm NME_{box}$ and $\rm AUC^{7}_{box}$ comparisons on 300W testing set and Menpo 2D (for cross dataset evaluation). As shown in Tables I and II, the proposed SCPAN outperforms state-of-the-art methods [30, 35, 57, 50, 59, 12] on 300W and Menpo 2D datasets. At the same time, the SCPAN also achieves the best score on AFLW-frontal dataset (as shown in Table III). These results indicate that the SCPAN can improve the detection accuracy under normal circumstances, mainly because 1) the proposed BALI fields can introduce both boundary and field constraints to enhance facial shape constraints and achieve high-precision landmark detection. 2) the SCPA model can boost supervision of the network and selectively emphasize important features and suppress less useful ones to learn more representative and discriminative features by introducing the self-calibrated mechanism and the pose attention mask. 3) by integrating the SCPA model and BALI fields into a Self-Calibrated Pose Attention Network (SCPAN), more accurate and effective landmark heatmaps and BALI fields can be generated for achieving robust and precise facial landmark detection.

Variations in occlusion and illumination are classic problems in face alignment task. The state-of-the-art face alignment methods still suffer from heavy occlusions and complicated illuminations. In this paper, we use COFW dataset, 300W challenging subset and WFLW dataset to evaluate the robustness of the proposed SCPAN against occlusion.

For 300W challenging subset, SCPAN can achieve 4.43% $\rm NME_{io}$ as shown in Table I, which outperforms state-of-the-art face alignment methods [30, 35, 57, 50, 59, 12]. This indicates the method can effectively enhance the alignment robustness for faces with heavy occlusions.

For COFW dataset (cross dataset evaluation), the failure rate ($\rm FR_{ip}^{10}$) is defined by the percentage of test images with more than 10% detection error which is normalized by interpupil distance. As illustrated in Table IV, the SCPAN can boost the $\rm NME_{ip}$ to 4.93% and the failure rate to 1.78%, which outperforms the state-of-the-art methods [60, 55, 61, 14, 59, 29, 13]. Moreover, we also use the model trained on 300W training set to further test COFW testing set and the corresponding experimental results (i.e., $\rm NME_{box}$ and $\rm AUC^{7}_{box}$) are shown in Table II. These all suggest that the proposed BALI fields and SCPA model play an important role in boosting the ability to address the occlusion problems.

Since WFLW dataset is composed of the Illumination subset, the Make-Up Subset and the Occlusion Subset, which contain complicated occlusions that can be used to evaluate the robustness of the SCPAN against occlusion. From the experimental results shown in Table V, we conclude that SCPAN is more robust to faces with complicated occlusions.

Hence, from the experimental results illustrated in Tables I, II, IV and V, we can conclude that 1) by fusing boundary heatmaps and landmark intensity fields, the proposed BALI fields is able to better model the facial geometric information and context information for enhancing facial shape constraints. 2) the SPCA model can learn more representative and discriminative features by introducing the self-calibrated mechanism and pose attention mask when faces are corrupted by complicated occlusions. 3) the texture disturbance operation can effectively expand the original dataset and make full use of the original label information, which helps achieve more robust landmark detection for faces with heavy occlusions.

Face with large poses is another great challenge for facial landmark detection. We conduct experiments on AFLW-full, 300W challenging subset and WFLW dataset to further evaluate the performance of the SCPAN for faces with large poses. Fig. 5 shows the landmark detection result comparison with the state-of-the-art methods and the ground-truths (i.e., GT) on 300W challenging subset. Tables I, III and V show the corresponding experimental results. To be specific, for AFLW-full dataset, the proposed SCPAN can achieve 1.31% $\rm NME_{diag}$, 2.05% $\rm NME_{box}$ and 69.8% $\rm AUC_{box}^{7}$, which exceeds state-of-the-art methods [62, 30, 14, 50, 12, 13]. For WFLW dataset, the $\rm NME_{io}$ on Testset, Pose Subset and Expression Subset all beat the other state-of-the-art methods [63, 14, 13, 50, 59, 12]. From the above experimental results, we can conclude that our method is more robust to faces with large poses, mainly because 1) the spatial transformation disturbance operations can effectively enrich the original dataset, i.e., produce many face images with large poses, thus the robustness of our method against large poses has been improved. 2) the SCPAN can better model the facial shape constraints by integrating the BALI fields and SCPA model, which improves the accuracy of landmark detection for faces with large poses.

We evaluate our proposed SCPAN on 300VW dataset with state-of-the-art facial landmark detection methods [64, 65, 66, 67, 7]. From the experimental results as shown in Table VI, we can find that (1) on both Scenario1 and Scenario2, our SCPAN can achieve state-of-the-art accuracy. (2) on Scenario3, our SCPAN beats the best score. These indicate that our proposed SCPAN is more robust to faces under variations on illumination conditions, occlusions, make-ups, expressions and head poses. Moreover, we believe that our SCPAN structure can be further boosted by using a temporal modeling technique [67, 7].

Current facial landmark detection methods still suffer insufficient labeled training samples. With a well-designed SCPA model, the proposed SCPAN is able to use unlabeled face images to boost the detection accuracy. To evaluate this, we use both labeled and unlabeled datasets to train SCPAN, i.e., 300W, AFLW, COFW and WFLW are separately mixed with CelebA dataset to train SCPAN (denoted as semi-SCPAN), the corresponding experimental results are shown in Table I, III, IV and V, respectively. From this we can see that semi-SCPAN outperforms the state-of-the-art methods [63, 30, 14, 13, 50, 59, 12] and other semi-supervised methods [33, 34, 35, 7]. This indicates that by introducing unlabeled face datasets, more prior knowledge can be learned to model facial shape constraints by semi-SCPAN for boosting the detection accuracy.

Heatmap generation. Almost heatmap regression-based facial landmark detection methods use heatmaps generated by a Gaussian distribution to regress and predict landmark coordinates. The closer to the landmark, the greater the belief value is. Moreover, the gradient of belief value is also changed. Therefore, by using the Gaussian distribution, neural networks can quickly and directionally reach the landmark. We also explore other two non-Gaussian distributions, i.e., Generalized Error Distribution (GED) [68, 69] and Student-t Distribution (StD) [70] to generate landmark heatmaps and boundary heatmaps. Note that, when we set $df=+\infty$ or $d=0$, the curves of GED and StD become the Standard Normal Distribution. $df$ means the degree of freedom of StD, $d$ denotes the shape parameter of GED. As shown in Table VII, we can find that using non-Gaussian distributions to generate heatmaps can achieve comparable or even better results than using a Gaussian distribution.

Sensitivity analysis of parameters. By combing $\mathbb{L}_{org}$, $\mathbb{L}_{scl}$ and $\mathbb{L}_{coor}$, we can obtain the overall loss. The losses corresponding to landmark heatmap and boundary heatmap should be given the same weight. However, the sizes of heatmaps in $t=T$ and $t=1\cdots T-1$ are $128\times 128$ and $64\times 64$, respectively. Therefore, $\lambda_{1}$ and $\eta$ are set to 1 and 4, respectively. Besides, the loss of field is calculated with a smaller area, $\lambda_{2}$ should be given a larger weight (i.e., 16). When calculating $\mathbb{L}_{coor}$, we first normalize the landmark coordinates to $[0,1]$, and then $\gamma$ is set to 40. We also conduct the corresponding experiments by using different $\lambda_{1}$, $\lambda_{2}$, $\gamma$ and $\eta$ values on 300W challenging subsets. From the experimental results in Tabel VIII, we can find that $\lambda_{1}=1$, $\lambda_{2}=16$, $\gamma=40$ and $\eta=4$ are good choices to balance these four parts.

Time and memory analysis. The proposed SCPAN contains three SCPA models and an Hourglass Network Unit (HNU). Compared to the HNU, our SCPA model increase both the parameter and computational costs. For the baseline i.e., four stacked SHN, its model size and parameters are 16.05MB and 184MB, respectively. While Our SCPAN takes 16.99MB parameters and its model size is 195MB. Besides, the reference speed of our SCPAN can achieve 40FPS in a single Telsa V100 GPU while the SHN achieves 100FPS. If we use multiple GPUs, the reference will speed up. Among all, compared to SHN, our SCPAN introduces some additional parameters which are insignificant compared to 16GB memory on a tesla v100, and the impact of the consumed computation costs will be decreased with the rapid development of the hardware. To reduce the computational costs or employ lightweight backbones, we also conduct the experiment by reducing the number of SCPA models, i.e., one SCPA model and one HNU are used to construct SCPAN. Its parameters and model size will be reduced to 8.73MB and 100MB, respectively. But the $\rm{NME_{io}}$ on 300W challenging dataset will only rise from 4.43% to 4.61%, which still achieves state-of-the-art accuracy.

Visualization of the proposed BALI fields. Since the proposed SCPAN can produce more effective landmark heatmaps and BALI fields and help achieve more accurate facial landmark detection, we visualize the produced landmark heatmaps and BALI fields in Fig. 6. As shown there, the first three rows visualize the landmark heatmaps, boundary heatmaps, BALI fields, the x-offset of the BALI fields and the y-offset of the BALI fields for faces under large poses, and the last three rows show those for faces with complicated occlusions. From Fig. 6, we can see that the SCPAN is able to generate effective and accurate landmark heatmaps and BALI fields for faces in challenging scenarios. Therefore, SCPAN achieves state-of-the-art performance on challenging benchmark datasets.

The proposed Self-Calibrated Pose Attention Network (SCPAN) contains two pivotal components, namely, the BALI fields and the SCPA model. Moreover, SCPAN (denoted as semi-SCPAN) can be further boosted by using unlabeled images. Therefore, the ablation studies are conducted as follows:

(1) The proposed BALI fields can introduce both boundary and field constraints to the predicted landmarks. To evaluate this, we conduct the experiment by combining SHN [38] with the proposed BALI fields (denoted as SHN+BALI) on challenging benchmark datasets including 300W, AFLW, COFW and WFLW. From the experimental results as shown in Tables I, III, IV and V, we can see that SHN+BALI outperforms SHN, which indicates that the proposed BALI fields can better model the facial shape constraints and help detect more accurate landmarks.

(2) The proposed SCPA model is able to learn more representative and discriminative features for producing more effective landmark heatmaps, hence, we conduct the experiment by combining SHN and SCPA model (denoted as SHN+SCPA). The experimental results in Tables I, III, IV and V shows that SHN+SCPA outperforms SHN, which demonstrates that the proposed SCPA model is able to generate more effective landmark heatmaps and help detect more accurate landmarks by introducing a self-calibrated mechanism and a pose attention mask.

(3) The proposed SCPAN is constructed by combining the SHN [38] with the proposed BALI fields and SCPA model. From the experimental results in Tables I, III, IV and V, we can see that SCPAN (SHN+BALI+SCPA) surpasses the SHN+BALI and the SHN+SCPA, respectively. We can conclude that by integrating the BALI fields and SCPA model into the proposed SCPAN, more effective landmark heatmaps and BALI fields can be produced, which can futher enhance the detection robustness against faces with large poses and heavy occlusions.

(4) Finally, SCPAN can be boosted with unlabeled face images, therefore, we conduct the following experiments (denoted as semi-SCPAN) by mixing CelebA dataset with 300W, AFLW, COFW and WFLW datasets, respectively. As shown in Tables I, III, IV and V, semi-SCPAN beats SCPAN on 300W, AFLW, COFW and WFLW datasets, which demonstrates that SCPAN can use unlabeled face images to improve its detection accuracy.

From the experimental results listed in Tables I-VI and the figures presented in previous subsections, we have the following observations and corresponding analyses.

(1) SCPAN, MHHN [13], AWing [59] and LUVLi [12] are all heatmap regression-based facial landmark detection methods, while the SCPAN outperforms MHHN, Awing and LUVLi as shown in Tables I-VI, which indicates that 1) the proposed BALI fields can effectively enhance the facial shape constraints, 2) the proposed SCPA model is able to learn more representative and discriminative features for producing more effective landmark heatmaps, 3) by integrating the proposed BALI fields and SCPA model into a novel SCPAN framework, the detection accuracy can be further improved.

(2) Facial shape constraints are very important for landmark detection task for faces with large poses and heavy occlusions. SCPAN, MHHN [13], ODN [29] and OpenPose [50] all aim to construct accurate facial shape constraints to improve their detection accuracy. However, from the experimental results in Tables I-VI, we can see that SCPAN outperforms the other methods for faces with large poses and complicated occlusions, which verifies that boundary and field constraints introduced by SCPAN can effectively and precisely model the spatial relationships among landmarks.

(3) Compared to other semi-supervised facial landmark detection methods including Honari et al. [33], SBR [34], TS [35] and STKI [7], semi-SCPAN outperforms them, which indicates that more effective facial prior knowledge can be learned by semi-SCPAN for achieving more robust and precise landmark detection.

Occlusion problems. Our proposed SCPAN can outperform the state-of-the-art methods for faces in normal circumstances and large poses. However, for heavily occluded faces (e.g., hair, cup and microphone as shown in Fig. 6), the field information becomes less accurate as the area around the landmarks may contain a lot of noise. But this situation can be alleviated by using the boundary heatmaps or larger field region.

The inference speed. The baseline SHN can achieve 100FPS in a single Telsa V100 GPU, while our proposed method needs to use both heatmap and field information to predict landmarks. Moreover, the size of the field region also affects the reference speed. When we use the information of the 7x7 field region, our reference speed will drop to 40 FPS. If we use multiple GPUs, the reference will speed up.