Fast and Accurate: Structure Coherence Component for Face Alignment

Since facial landmarks detection requires extreme precise localization, preserving the spatial information are crucial for an accurate model. Heatmap based methods usually uses several hourglass structures [30] to preserve the spatial information. However, such encoder-decoder architecture is extremely heavy and slows down the inference speed. We propose an efficient attention guided multi-scale features module to improve the localization capability. Fig. 3 illustrates the architecture of this module.

Multi-scale Features: The feature maps from shallower layers encode low-level information and spatial details, while deep layers encode high semantic information [4, 27, 26]. We introduce two bottom-up branches to propagate the spatial details from shallow layers to the deepest layer. Specifically, consider a convolutional backbone composed of $L$ convolutional blocks. We denote $\bm{F}_{l}$ as the last feature maps of the $l$-th block. We exploit the spatial information from the feature maps $\bm{F}_{L-1}$ and $\bm{F}_{L-2}$ to augment the localization precision of the features $\bm{F}_{L}$. Each branch is composed of a $3\times 3$ Conv-BN-ReLU, an attention mechanism to filter out noisy information and a down-sampling operation. These feature maps with spatial details are then concatenated with $\bm{F}_{L}$ to form more expressive feature maps.

Semantic-guided Attention: Although the feature maps from shallow layers have rich spatial information, they also contain noisy information which are not informative from the perspective of semantic meaning. We propose semantic-guided attention module to filter out such information. Unlike existing self-attention which uses self-features to compute an attention map, we exploit the high-semantics features maps $\bm{F}_{L}$ to guide the feature maps $\bm{F}_{L-1}$ to suppress noisy information while keeping spatial details. We first upsample the feature maps $\bm{F}_{L}$, concatenate it with $\bm{F}_{L-1}$ and reduce the channel dimension into the channel of $\bm{F}_{L-1}$ via an $1\times 1$ convolution, obtaining $\tilde{\bm{F}}_{L-1}$. As $\tilde{\bm{F}}_{L-1}$ merges the information from $\bm{F}_{L}$ and $\bm{F}_{L-1}$, it contains both semantic information and spatial details. We then use the attention module described in [42] to generate spatial attention $\bm{A}^{s}$ and channel-wise attention $\bm{A}^{c}$ from $\tilde{\bm{F}}_{L}$ as:

where $\mathbf{W}_{0}\in\mathbb{R}^{C/2\times C}$ and $\mathbf{W}_{1}\in\mathbb{R}^{C\times C/2}$, $C$ is the channel number, $\text{conv}_{7\times 7}$ denotes an $7\times 7$ convolution operation, $\sigma$, $\rho$ and $[\cdot;\cdot]$ denote sigmoid function, ReLU activation and concatenation operation respectively, $\tilde{\bm{F}}^{c}_{\text{avg}}$/$\tilde{\bm{F}}^{s}_{\text{avg}}$ and $\tilde{\bm{F}}^{c}_{\text{max}}$/$\tilde{\bm{F}}^{s}_{\text{max}}$ denote spatially/channel-wise average-pooled features and max-pooled features, respectively. The notation $L$ is omitted for more clarity. The attentive features are then obtained via element-wise multiplication and residual addition. Similarly, we compute the attention for feature maps $\tilde{\bm{F}}_{L-2}$ with messages from features $\tilde{\bm{F}}_{L-1}$ and $\tilde{\bm{F}}_{L}$, and obtain the attentive features. Finally, we concatenate these features to form the attention guided multi-scale features $\bm{F}_{A}$. Note that designing the attention module is not our main focus, we adopt the commonly used attention module [42] in our semantic-guided process.

As the relative spatial relationship of facial landmarks is stable, it is desirable to capture and exploit such important cues. We statically calculate the correlation between face landmarks from data analysis and leverage graph relation networks to effectively explore these relation information.

Map-to-Node Module: In order to make our network end-to-end trainable, we design the map-to-node module to seamlessly map convolutional feature maps to graph node representations. The input convolutional feature maps $\bm{F}_{A}\in\mathbb{R}^{C\times H\times W}$ (where $C$, $H$ and $W$ represents number of channels, height and width) are first transformed to the hidden feature maps by non-linear function $\bm{Z}=\phi(\bm{F}_{A})\in\mathbb{R}^{Nn\times H\times W}$, where $n\in\mathbb{Z}^{+}$ is the expansion coefficient and $N$ is the number of landmarks. In this paper, we consider two convolution-BN-ReLU blocks with $n=4$ as the non-linear function $\phi(\cdot)$. $\bm{Z}$ is then reshaped to $\bm{Z}_{\mathrm{node}}\in\mathbb{R}^{N\times nHW}$ to represent the node feature.

Graph Convolution: Unlike standard convolutions that operate on local Euclidean structures, e.g., a image grid, the goal of GCN is to learn a function $f(\cdot,\cdot)$ on a graph $\mathcal{G}$ , which takes node feature $\bm{H}^{l}\in\mathbb{R}^{N\times d_{l}}$ and the corresponding adjacency matrix $\bm{A}\in\mathbb{R}^{N\times N}$ as input, and outputs the node features as $\bm{H}^{l+1}\in\mathbb{R}^{N\times d_{l+1}}$. Here $N$, $l$, $d_{l}$ and $d_{l+1}$ denote the number of nodes, layer index, the dimension of input node features and the dimension of output node feature respectively. Every GCN layer can be written as a non-linear function by,

With the specific graph convolutional operators employed by [21], the layer can be represented as,

where $\bm{W}^{l}\in\mathbb{R}^{d_{l}\times d_{l+1}}$ is a transformation matrix to be learned, $\widetilde{\bm{A}}=\bm{A}+\bm{I}$, $\widetilde{\bm{D}}$ is the degree matrix of $\widetilde{\bm{A}}$, $\widetilde{\bm{D}}^{-\frac{1}{2}}\widetilde{\bm{A}}\widetilde{\bm{D}}^{-\frac{1}{2}}$ is the symmetric normalized version of $\widetilde{\bm{A}}$ and $\psi(\cdot)$ denotes BN-ReLU operation.

Neighborhoods Construction: Graph relation network propagates information between nodes based on the adjacency matrix which is crucial to be correctly constructed. In our problem, due to the lack of pre-defined adjacency matrix for facial landmarks, we build it through a data-driven way, i.e., treating each landmark as a node and mining the correlation between landmarks within the dataset. Specifically, we assemble the landmark coordinates of the dataset into a rank-three data tensor $\bm{T}\in\mathbb{R}^{M\times N\times 2}$ where $M$ is the number of images, and the last dimension represents the $(x,y)$ coordinates. We then slice the tensor $\bm{T}$ along the last dimension to generate $\bm{T}_{x}$ and $\bm{T}_{y}$. Based on $\bm{T}_{x}\in\mathbb{R}^{M\times N}$ and $\bm{T}_{y}\in\mathbb{R}^{M\times N}$, we calculate Pearson’s correlation coefficient in $x$ and $y$ direction respectively to form correlation matrices $\bm{C}_{x}\in\mathbb{R}^{N\times N}$ and $\bm{C}_{y}\in\mathbb{R}^{N\times N}$. Then, the correlation between nodes is defined as:

where $\text{abs}(\cdot)$ returns element-wise absolute value of matrix. Considering the computation cost and noisy edges, we only retain the top $k+1$ largest value of each row of $\bm{C}$ to form a sparse adjacency matrix $\bm{M}$. In other words, most $k$ relevant landmarks are picked as the neighborhood of each landmark. The binary adjacency matrix with self-loops can be written as:

Dynamic Adjacency Matrix Weighting: The static adjacency matrix $\bm{M}$ is constructed based on the geometric structure of facial landmarks, while learning relationship among landmarks for each face aims to take the facial appearance factors like occlusion and head pose into consideration. Given the binary matrix $\bm{M}$ which determines the node neighborhoods, we seek to adaptively adjust its weights.

Formally, given the features $\bm{Z}$ extracted from the map-to-node modules, we use the global average pooling layer followed by two fully connected layers to map $\bm{Z}$ to vector $\bm{a}$ whose size is equal to the non zeros in $\bm{M}$. Finally, we replace the non zeros value in $\bm{M}$ with $\bm{a}$ to form the dynamic adjacency matrix $\bm{A}$. Following the strategy in [54], we adopt a row-wise softmax operation $\sigma_{\text{row}}$ to replace the symmetric normalization in Eq. 3. Softmax operation makes the weights of each node like probabilities over its neighboring nodes, which stabilizes the training process:

We use the binary matrix $\bm{M}$ to hold the neighborhoods and only learn their weights because the facial shape pattern are stable, fix the sparse connection will greatly reduce the training parameters which makes the learning process easier.

Graph Relation Network: Inspired by the success of ResNet[20], we adopt the graph residual block architecture. Each block consists of two graph convolutional layers and can be formulated based on Eq. (2) as

The overall graph relation network architecture is shown in Fig. 2. The input feature $\bm{Z}_{node}$ is first fed to graph convolution, followed by several graph residual blocks. The last graph convolution (without batch normalization and ReLU) block maps the hidden node features to landmark coordinates $\bm{O}\in\mathbb{R}^{N\times 2}$.

Comparison with FC-based regression methods. The fully connected layer and our graph convolutional layers embed the feature of landmarks in two different ways. As shown in Fig.1(a) The CNN backbone and the hidden fully connected layer map the input facial image to the hidden vector, which embed the feature of landmarks globally. Thus, the errors of some parts of the prediction effects the other parts, as they share the same hidden feature. As shown in Fig. 1(b), for the FC-based method, the errors of occluded part interfere the prediction of other visible parts. Meanwhile, our SCC embeds the node feature for each landmark, and propagates node feature according to their relationship. If some parts of predictions fail because of the occlusion, large pose or other hard condition, the node feature of other parts degrade gracefully because of the sparse connection among the node features and the dynamic adjustment of the relationship. As shown in Fig. 1(d), the SCC-based method are more robust to hard cases. Besides, fully connected layers are prone to overfit because of the large number of trainable parameters, while the graph convolution layer requires fewer trainable parameters.

Wing loss[14] has constant gradient when error is large, and large gradient for small or medium range errors, which is defined as:

where $x$ is error and $C$ is $\omega-\omega\ln(1+\omega/\epsilon)$ to smoothly link two piece-wise functions. According to our experiment, the performance of Wing loss is not consistently better than L1 loss, especially when we train the neural networks on difficult dataset with heavy occlusion and blur, such as WFLW. As mentioned in [29], this may be caused by inconsistent annotations due to various reasons, e.g., unclear or inaccurate definition of some landmarks, poor quality of some facial images. Imposing a large gradient magnitude around very small error to force the model exactly fit the ground truth landmarks makes the training process unstable. To alleviate this problem, we present Soft Wing loss to more focus on the errors of medium range:

which is linear for small values, and take the curve of $\ln(\cdot)$ for medium and large values. Similar to Wing loss, we use the non-negative $\omega_{1}$ to switch between linear and non-linear part, and $\epsilon$ to limit the curvature of the non-linear part. $B$ is set to $\omega_{1}-\omega_{2}\ln(1+\omega_{1}/\epsilon)$ to make function continuous at $\omega_{1}$. The visualization of L1, Wing and our Soft Wing loss is shown in Fig.4. Note that we discard the linear part of Wing loss, since our proposed loss can adaptively adjust the magnitude of gradient between medium ($\omega_{1}<|x|<\omega_{2}$) and large errors ($|x|>\omega_{2}$). The magnitude of gradient of the non-linear part is $\frac{\omega_{2}}{|x|+\epsilon}\approx\frac{\omega_{2}}{|x|}$ ($\epsilon$ is commonly set to small value). Our proposed loss is insensitive to outliers where the gradient varies between $[\frac{\omega_{2}}{C},1]$ ($C$ is the image size). Note that $\omega_{2}$ should not set to small value because it will cause gradient vanishing problem.

Datasets: We conduct evaluation on three widely-adopted challenging datasets: WFLW [43], COFW [3] and 300W [33]. WFLW is among the most challenging face alignment benchmark which includes various hard cases such as heavy occlusion, blur and large pose. COFW is collected to present faces with large variations in shape and occlusions in real-world conditions. Various types of occlusions are introduced and result in a $23\%$ occlusion on facial parts on average. We also use the re-annotated test set [18] with 68 landmarks annotation for cross-dataset validation. 300W contains face images with moderate variations in pose, expression and illumination.

Evaluation Metric: We evaluate the proposed method with normalized mean error and failure rate. we use the inter-ocular distance as the normalization factor. Following the protocol in [43], the failure rate for a maximum error of 0.1 is reported. Area under curve (AUC) is also calculated based on the cumulative error distribution for WFLW dataset.

All training images are center-cropped and resized to $256\times 256$. Data augmentation is performed with random rotation ($\pm 40^{\circ}$), translation ($\pm 30\ px$), flipping ($50\ \%$), rescaling ($\pm 10\ \%$) and occlusion ($20\ \%$ of image size). To mitigate the issue of pose variations, we adopt the Pose-based Data Balancing (PDB)[14] strategy with 9 bins. We use ResNet34[20] as our backbone. During the training, we employ vanilla SGD for optimization with a batch size of $64$ for $500$ epochs. We set the weight decay and the momentum to $0.0005$ and $0.9$ respectively. The initial learning rate is $0.01$ which is dropped by 5 every $100$ epochs. The parameters of the Soft Wing loss are set to $\omega_{1}=2$, $\omega_{2}=20$ and $\epsilon=0.5$. The $k$ is set to 3 for adjacency matrix. We use 4 graph residual blocks with hidden feature dimension 128. Our models are trained from scratch using Pytorch.

WFLW: We evaluate our approach on the WFLW dataset and compare with state-of-the-art methods in terms of mean error, failure rate and AUC. To better understand the effectiveness of the proposed method, we analyse the performance on six subsets with specific issue, e.g., large pose, occlusion and exaggerated expression [43]. The overall results are tabulated in Table 1. The proposed method achieves $4.40\%$ NME, $2.88\%$ failure rate and $0.5666$ AUC, which outperforms most state-of-the-art approaches. Our method fails on only $2.88\%$ of all images, which demonstrates the robustness of our model. Qualitative results are depicted in Fig. 5, where our model successively localizes landmarks in hard cases.

COFW: As shown in in Table 2, our method achieves state-of-the-art performance with $3.63\%$ mean error and $0\%$ failure rate. To further verify the generalization capability of our method, we conduct a cross-dataset evaluation using COFW-68 dataset annotated with 68 landmarks [18]. Our method outperforms the existing best approaches by a large margin, with $4.18\%$ mean error and $0\%$ failure rate. Since the COFW dataset is mainly composed of occluded faces, this impressive performance indicates the robustness of our graph relation framework to handle heavy occlusions.

300W: We compare our approach against the existing best performing methods on the 300W dataset. The results are reported in Table 4.3. Our method outperforms most existing approaches. Note that our method achieves the best results on the challenging subset, which highlights the robustness of the proposed approach in hard cases.

Efficiency Comparison: Since facial landmark detection is widely deployed for many real-time applications, the model size, FLOPS and processing speed are key criteria. We evaluate the runtime of our model on a 1080Ti GPU and compare with existing methods in Table 4.3. Our model only takes $23$ ms and $5.165$ FLOPS to process a $256\times 256$ input image and consists of $\sim 24$M parameters. Overall, our model is faster and smaller than most competitors.

Our framework is composed of several pivotal modules such as graph relation network, attention guided multi-scale features, and soft-wing loss. Based on the baseline Resnet34 with $L=5$ layer stages, we examine the contributions of each proposed module on the WFLW dataset and report the overall results in Table 4.4.

Baseline Model: We first utilize the FC layers to directly regress the facial landmarks. This model is our baseline which achieves a NME of $5.95\%$.

Graph Relation Network: The graph relation network is a key part of our Structure Coherency Component. We obtain a $1.31\%$ improvement by replacing the FC layers with our graph relation network, resulting in a NME of $4.64\%$.

Top-k value for adjacency matrix: We report the result with different values of $k$ from $k=1$ to $k=97$ in Fig. 7. When $k=3$, our model achieves the best performance on WFLW dataset. Note that, the performance degrades if the adjacency matrix is too sparse or too dense. When $k$ is too small, each graph node can not get sufficient information from its correlated neighborhoods, meanwhile, when $k$ is too large, the adjacency matrix becomes dense which leads to oversmoothing of the node features.

Dynamic adjacency matrix weighting: We replace the dynamic adjacency matrix with the binary adjacency matrix $\bm{M}$ and we observe a $0.07\%$ degradation.

Attention guided multi-scale feature: The attention guided multi-scale features fusion plays a key role in improving the representation capability of features. By endowing the spatial details to high-semantics features, the model performs a NME of $4.53\%$, which corresponds to a $0.11\%$ improvement.

Semantic-guided attention: We examine the importance of incorporating additional semantic information from deeper layers to guide generating attention maps. To this end, we degenerate our semantic-guided attention structure into a general self-attention mechanism. As shown in Table 4.4, we observe a drop of performance, resulting in a NME of $4.61\%$. This experiment proves that the semantic information from high-level features is crucial to guide generating high quality attention. The quantitative performances are supported by the qualitative results illustrated in Fig. 6. The semantic guidance permits to make the feature maps focus on all visible key facial parts. Since self-attention only explore self-information, it only highlights the high-activated part in feature maps.

Features combination: To improve the localization capability, we propagate the spatial information from the shallower layers. We study which combination of feature layers is the optimal one. As tabulated in Table 4.4, the performance increases with the the additional spatial information propagation and the combination of features $\{\bm{F}_{3},\bm{F}_{4},\bm{F}_{5}\}$ yields best results. Since layer 2 is quite shallow, $\bm{F}_{2}$ consists few useful information and limits the performance due to the noise.

Soft Wing Loss: The Soft Wing loss improves the results of the graph relation network by $0.12\%$. We compare the performance of L1, Wing and our Soft Wing loss based on our baseline model, the results are shown in Table 8. Our Soft Wing loss consistently outperforms Wing loss and L1 loss. As discussed in Section 3.3, the performance of Wing loss degrades when $\epsilon$ decreases, while our loss benefits from imposing larger gradients on medium range errors. The performance of Wing loss is even worse than L1 loss when $\epsilon$ is very small.