ASMNet: a Lightweight Deep Neural Network for Face Alignment and Pose Estimation

Active Shape Model is a statistical model of shape objects. Each shape is represented as $n$ points, $S_{set}~{}=~{}\{(S_{x}^{1},S_{y}^{1}),...,(S_{x}^{n},S_{y}^{n})\}$ that are aligned into a common coordinate system. To simplify the problem and learn shape components, Principal Component Analysis (PCA) is applied to the covariance matrix calculated from a set of $K$ training shape samples. Once the model is built, an approximation of any training sample ($S$) is calculated using Eq. 1:

$$S\approx\overline{S}+Pb$$

(1)

where $\overline{S}$ is the sample mean, $P=(p_{1},p_{2},...,p_{t})$ contains $t$ eigenvectors of the covariance matrix and $b$ is a $t$ dimensional vector given by Eq. 2:

$$b=P^{\intercal}(S-\overline{S})$$

(2)

Consequently, a set of parameters of a deformable model is defined by vector $b$, so that by varying the elements of the vector, the shape of the model is changed. Consider that the statistical variance (i.e., eigenvalue) of the $i^{th}$ parameter of $b$ is $\lambda_{i}$. To make sure the generated image after applying ASM is relatively similar to the ground truth, the parameter $b_{i}$ of vector $b$ is usually limited to $\pm 3\sqrt{\lambda_{i}}$ [7]. This constraint ensures that the generated shape is similar to those in the original training set. Hence, we create a new shape $S_{New}$ after applying this constraint, according to Eq. 3:

$$S_{New}=\overline{S}+P\tilde{b}$$

(3)

where $\tilde{b}$ is the constrained $b$. We also define $\mathcal{ASM}$ operator according to Eq. 4:

$$\mathcal{ASM}:(P_{x}^{i},P_{y}^{i})\mapsto(A_{x}^{i},A_{y}^{i})$$

(4)

$\mathcal{ASM}$ transforms each input point $(P_{x}^{i},P_{y}^{i})$ to a new point $(A_{x}^{i},A_{y}^{i})$ using Eqs. 1, 2, and 3.

In this paper, we propose a deep convolutional neural network architecture that utilizes ASM in the training loss function. Our proposed network (ASMNet) is significantly smaller than its predecessor, MobileNetV2 [33], while its performance is comparable with MobileNetV2 [33] in localizing landmark points and much better in pose estimation.

MobileNet and its variants [33] have received great attention as one of the most well-known deep neural networks for operation on embedded and mobile devices. Especially, MobileNetV2 [33] is shown to cope well with complex tasks such as image classification, object detection, and semantic segmentation. We have designed a network that is about two times smaller than MobileNetV2 [33], both in terms of the number of parameters, and FLOPs. In designing ASMNet, we only use the first 15 blocks of MobileNetV2 [33] while the main architecture which has 16 blocks. Nevertheless, creating a shallow network would eventually lower the final accuracy of the system. To avoid this problem we purposefully add a few new layers. Fig. 2 shows the architecture of ASMNet.

According to [52] the features in a Convolutional Neural Network (CNN) are distributed hierarchically. In other words, lower layers have features such as edges, and corners which are more suitable for tasks like landmark localization and pose estimation, and deeper layers contain more abstract features that are more suitable for tasks like image classification and image detection. Training a network for correlated tasks simultaneously builds a synergy that can improve the performance of each task [6, 54].

Motivated by the approach in [28], we designed a multi-task CNN to detect facial landmarks as well as estimating the pose of the faces (pitch, roll, and yaw) simultaneously. In order to use features from different layers, we have created shortcuts from block-1-batch-normalization, block-3-batch-normalization, block-6-batch-normalization, block-10-batch-normalization, and finally block-13-batch-normalization. We connect each of these shortcuts to the output of block 15 of MobileNetV2 [33], block-15-add, using a global average pooling layer. Finally, we concatenate all the global average pooling layers. Such architecture enables us to use features that are available in different layers of the network while keeping the number of the FLOPs small. In other words, since the original MobileNetV2 [33] is designed for image classification task - where the more abstract features are required - it might not be suitable for face alignment task - which needs both abstract features that are available in the deeper layers as well as features that are available in the lower layers such as edges and corners -.

We Designed ASMNet (see Fig. 2), by fusing the features that are available if different layers of the model. Furthermore, by concatenating the features that are collected after each global average pooling layer in the back-propagation process, it will be possible for the network to evaluate the effect of each shortcut path.

Moreover, we add another correlated task to the network. As Fig. 2 shows, the proposed network predicts 2 different outputs: the facial landmark points (the main output of the network), as well as the face pose. While the correlation and the synergy between these two tasks can result in more accurate results, we also wanted our lightweight ASMNet to be able to predict face pose as well so that it might be used in more applications.

In the following, we describe the loss functions for two different tasks. These tasks are responsible for facial landmark points detection, and pose estimation.

Facial landmark points detection task: The common loss function for facial landmark points detection is Mean Square Error (MSE). We propose a new loss function that including MSE, as the main loss as well an the assistant loss which utilizes ASM to improve the accuracy of the network called ASM-LOSS.

The proposed ASM-LOSS guides the network to first learn the smoothed distribution of the facial landmark points. In other words, during the training process, the loss function compares the predicted facial landmark points with their corresponding ground truth as well as the smoothed version the ground truth which is generated using ASM. Given this, in the early stage of training, we set a bigger weight to the ASM-LOSS in comparison to the main loss – which is MSE –, since the variation of the smoothed facial landmark points are much lower than the original landmark points, and as a rule of thumb, easier to be learned by a CNN. Then, by gradually decrease the weight of the ASM-LOSS, we lead the network to focus more on the original landmark points. In practice, we figured out that this method, which is also can be taken to account as transfer learning, works out well and results in more accurate models.

We also discover that although face pose estimation has a heavy reliance on face alignment, it can achieve good accuracy with the assistant of smoothed facial landmark points as well. In other words, if the performance of facial landmark point detection task is acceptable, which means network can predict facial landmark such that the whole shape of the face is correct, the pose estimation can achieve a good accuracy. Accordingly, using smoothed landmark points and training network using ASM-LOSS will results in a more accuracy in pose estimation task.

Consider that for each image in the training set, there exists n landmark points in a set called G such that $(G_{x}^{i},G_{y}^{i})$ is the coordinates for the $i^{th}$ landmark point. Similarly, a predicted set P contains n points such that $(P_{x}^{i},P_{y}^{i})$ is the predicted coordinates for the $i^{th}$ landmark point.

$$\begin{split}G_{set}=\{(G_{x}^{1},G_{y}^{1}),...,(G_{x}^{n},G_{y}^{n})\}\\ P_{set}=\{(P_{x}^{1},P_{y}^{1}),...,(P_{x}^{n},P_{y}^{n})\}\end{split}$$

(5)

We apply PCA on the training set and calculate eigenvectors and eigenvalues. Then, we calculate set A, which contains n points and each point is the transformation of the corresponding point in G, by applying the $\mathcal{ASM}$ operator according to Eq. 4:

$$\begin{split}A_{set}=\{(A_{x}^{1},A_{y}^{1}),...,(A_{x}^{n},A_{y}^{n})\}\\ \mathcal{ASM}:(G_{x}^{i},G_{y}^{i})\mapsto(A_{x}^{i},A_{y}^{i})\end{split}$$

(6)

We define the main facial landmark point loss, Eq. 7, as the Mean Square Error between the ground truth (G) and the predicted landmark points (P).

$$\mathcal{L}_{mse}=\frac{1}{N}\frac{1}{n}\sum_{j=1}^{N}\sum_{i=1}^{n}\lVert G^{i}_{j}-P^{i}_{j}\rVert_{2}$$

(7)

where $N$ is the total number of images in the training set and $G^{i}_{j}=(G_{x}^{i},G_{y}^{i})$ shows the $i^{th}$ landmark of the $j^{th}$ sample in the training set. We calculate ASM-LOSS as the error between ASM points ($A_{set}$), and predicted landmark points ($P_{set}$) using Eq. 8:

$$\mathcal{L}_{asm}=\frac{1}{N}\frac{1}{n}\sum_{j=1}^{N}\sum_{i=1}^{n}\lVert A^{i}_{j}-P^{i}_{j}\rVert_{2}$$

(8)

Finally, we calculate the total loss for the facial landmark task with according to Eq. 9:

$$\mathcal{L}_{facial}=\mathcal{L}_{mse}+\alpha~{}\times~{}\mathcal{L}_{asm}$$

(9)

The accuracy of PCA have a heavy reliance on the ASM points ($A_{set}$), which means that the more accurate the PCA, the less the discrepancy between the ground truth (G) and the ASM points ($A_{set}$). To be more detailed, by reducing the accuracy of PCA, the generated ASM points ($A_{set}$), will be more similar to the average point set, which is the average of all the ground truth face objects in the training sets. Consequently, predicting points in $A_{set}$ is easier than the points in the $G_{set}$ since the variation of latter is lower than the variation of the former. We use this feature to design our loss function such that we first guide the network towards learning the distribution of the smoothed landmark points – which is easier to be learned – and gradually harden the problem by decreasing the weight of ASM-LOSS.

We define $\alpha$ as ASM-LOSS weight using Eq. 10:

$$\alpha=\left\{\begin{matrix}2&&i<\frac{l}{3}\\ 1&&\frac{l}{3}<i<\frac{2l}{3}\\ 0.5&&i>\frac{2l}{3}\par\end{matrix}\right.$$

(10)

where i is the epoch number and l is the total number of training epochs. As shown in Eqs. 9, at the beginning of the training, the value of $\alpha$ is higher, which means we put more emphasize on ASM-LOSS. Hence, the network focuses more on predicting a simpler task and converges faster. Then after one-third of total epochs, we reduce $\alpha$ to 1, and put equal emphasis on the main MSE loss ASM-LOSS. Finally, after two-third of total epochs, by reducing $\alpha$ to 0.5, we direct the network toward predicting the main ground truths, while considering the smoothed points generated using ASM as an assistant. We also show experimentally in Sec. 4, that such technique leads to more accurate results, specifically when it comes to a lightweight network like ASMNet.

Pose estimation task: We use mean square error to calculate the loss for the head pose estimation task. Eq. 11 defines the loss function $\mathcal{L}_{pose}$, where yaw($y^{p}$), pitch($p^{p}$), and roll($r^{p}$) are the predicted poses and $y^{t}$, $p^{t}$, and $r^{t}$ are the corresponding ground truths.

$$\mathcal{L}_{pose}=\frac{1}{N}\sum_{j=1}^{N}\frac{(y^{p}_{j}-y^{t}_{j})^{2}+(p^{p}_{j}-p^{t}_{j})^{2}+(r^{p}_{j}-r^{t}_{j})^{2}}{3}$$

(11)

Finally, we calculate the total loss as the total weighted loss of the 2 individual losses using Eq. 12:

$$\mathcal{L}=\sum_{i=1}^{2}\lambda_{task_{i}}\mathcal{L}_{task_{i}}$$

(12)

such that $task_{i}$ is the $i^{th}$ element of the task set T = { $\mathcal{L}_{facial}$, $\mathcal{L}_{pose}$ } and the value of $\lambda_{task_{i}}$ corresponds to the importance of the $i^{th}$ task. Since we define facial landmark points detection task to be more important than pose estimation, we choose $\lambda_{task}=\{1,0.5\}$. Fig. 1 illustrates the process of calculating the total loss value.

300W.We followed the protocol described in [29] to train our networks on the 300W [31] dataset. We use 3,148 faces consisting of 2,000 images from the training subset of HELEN [23] dataset, 811 images from the training subset of LFPW [4] dataset, and 337 images from the full set of AFW [60] dataset with a 68-point annotation. For testing, 300W [31] has 3 subsets: Common subset with 554 images, Challenging subset with 135 images, and Full subset, including both Common and Challenging subsets, with 689 images. More specifically, the Challenging subset is the IBUG [31] dataset while the Common subset is a combination of the HELEN test subset (330 images) and LFPW test subset (224 images).

WFLW. WFLW [44], containing 7500 images for training and 2500 images for testing, is another widely used dataset, recently has been proposed based on WIDER FACE [51]. Each image in this dataset contains 98 manual annotated landmarks. In order to be able to evaluate the models under different circumstances, WFLW [44] provides 6 different subsets including 314 expression images, 326 large pose images, 206 make-up images, 736 occlusion images, 698 illumination images, and 773 blur images.

We use the method and algorithm in [30] to calculate the yaw, roll, and, pitch for each image in the dataset since to the best of our knowledge, no dataset provides the annotation for facial landmark points and face pose jointly.

For the training set in each dataset, we crop all the images and extract the face region. Then the face images are scaled to $224\times 224$ pixels. We augment the images (in terms of contrast, brightness, and color) to add robustness of data variation to the network. We use Adam optimizer for training the networks with learning rate $10^{-2}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $decay=10^{-5}$. Then we train networks for about 150 epochs with a batch size of 50. We implemented our codes using the TensorFlow library and run them on a NVidia 1080Ti GPU.

We follow the previous works and employ normalized mean error (NME) to measure the accuracy of our model. We define the normalising factor, followed by MDM [38] and [31] as “inter-ocular” distance (the distance between the outer-eye-corners). Furthermore, we calculate failure rate (FR), defined as the proportion of failed detected faces, for a maximum error of 0.1. Cumulative Errors Distribution (CED) curve as well as the area-under-the-curve (AUC) [49] is also reported. Besides, we use mean absolute error (MAE) for evaluating the pose estimation task.

We conducted four different experiments to evaluate the effectiveness of the proposed ASM assisted loss function. These experiments are designed to assess the performance of MobileNetV2 [33] and ASMNet, with and without the proposed ASM assisted loss function.

Table 1 shows the results of the experiments on Full subsets of 300W [31] (full), and WFLW [44], as well as the number of network parameters(#Params) and the sum of the FLOPs. For simplicity, we name our model as ”mnv2” (MobileNetV2 [33] trained using standard MSE loss function), ”mnv2_r” (MobileNetV2 [33] trained using our ASM assisted loss function), ”ASMNet_nr” (ASMNet trained using standard MSE loss function), and ”ASMNet” ( ASMNet trained using our ASM assisted loss function).

Table 1 shows that the proposed ASM assisted loss function has a lower NME in both cases. Furthermore, while our proposed network architecture is about two times smaller than MobileNetV2 [33], its performance is comparable with it after applying our proposed ASM assisted loss function. It means that without sacrificing accuracy, we have created a network that is smaller and faster in comparison to MobileNetV2 [33]. Such characteristics make the ASMNet suitable for running on mobile and embedded devices.

Evaluation on 300W. The 300W [31] dataset is a very challenging benchmark in facial landmark detection task. Table 2 shows a comparison between ASMNet and the state-of-the-art methods. Although the performance of ASMNet does not outperform the state-of-the-art methods, comparing the number of the parameters, and FLOPs of the models (see Table 6), the accuracy of our proposed model is comparable and accurate in the context of small networks such as MobileNetV2 [33]. Furthermore, As the table 2 shows, the performance of the ASMNet with ASM assisted loss function on 300W [31] is better than the performance of ASMNet without the assisted loss. Fig. 3 shows the some example of facial landmark detection using ASMNet on the Challenging subset of 300W [31] dataset. As we can see, ASMNet performs well, even in challenging face images.

Evaluation on WFLW. Table 3 shows the performance of the state-of-the-art method and our proposed method over WFLW [44] and its 6 subsets. The performance of ASMNet is comparable to the performance of MobileNetV2 [33]. In other words, using the proposed ASM assisted loss function improves the model accuracy. Fig. 4 shows the some example of facial landmark detection using ASMNet on WFLW [44] dataset. While ASMNet can be taken as a very lightweight model, its performance is acceptable under different circumstances such as occlusion, extreme pose, expression, illumination, blur, and make-up.

Pose Evaluation. Neither 300W [31] nor WFLW [44] dataset do not provide the head pose information. Accordingly, we followed the method used by [48] and used another application to synthesizes the pose information. Although [48] used [3] for synthesizing the pose information, we used HopeNet [30] which is a state-of-the-art pose estimation method. Using HopeNet we acquired the yaw, pitch, and roll values of the 300W [31], and WFLW [44] images and used them as the ground truths for our network. Table 4 shows the mean absolute error (MAE) between HopeNet [30] results and our ASMNet.

In addition, we compare the performance of our proposed method with [48] as well as [50] in Table 5 using Full subset of 300W [31] dataset. As the results show, the performance of our lightweight ASMNet is comparable to HopeNet [30], which is a state-of-the-art method and outperforms the other methods as well. Besides, the performance of ASMNet is better than MobileNetV2 [33], even when it utilizes the ASM-LOSS function. Since in pose estimation task aligning the whole shape of the face is more crucial than aligning each landmark point, using ASM-LOSS function will lead to better performance.

Moreover, ASMNet is designed to use features generated in different layers of the neural network which enables it to outperforms MobileNetV2 [33] in pose estimation task. Fig. 5 shows the output of the pose estimation task.

Ablation Study. In Table 7 we study the ASM assisted loss by calculating the difference between normalized mean errors with and without ASM assisted loss both on ASMNet and MobileNetV2 [33]. As shown, using ASMNet utilized with ASM-LOSS function results in 0.96%, and 1.19% reduction in NME on 300W [31], and WFLW [44] respectively. Theses numbers are 0.11%, and 0.16% for MobileNetV2 [33]. According to the Table 7, and Fig. 6, using the ASM assisted loss function resulted in more accuracy improvement for ASMNet compared to MobileNetV2 [33]. Hence, it can be concluded that the ASM-LOSS function is capable of helping the lightweight CNN much more. In other words, when a lightweight network does not perform accurately enough, using the proposed ASM-LOSS function will play a vital role in improving the performance.

Model Size and Computational Cost Analysis. We calculate the number of network parameters as well as FLOPs to evaluate the model size and computational complexity. We calculate the FLOPs over the resolution of $224\times 224$. As Table 6, although ASMNet is the smallest, its performance is comparable with MobileNetV2 [33], one of the best in compact-class models. Furthermore, since the idea behind ASMNet is to put a trade-off between accuracy and model performance, as we can see in Table 6, adding ASM assisted loss to a lightweight model such as ASMNet, and MobileNetV2 [33], results in the accuracy improvement.