Global-to-local Expression-aware Embeddings for Facial Action Unit Detection

Due to the high labor cost of AU annotations, the scale and subject variations of AU detection datasets are usually limited. As a result, previous AU detection methods resort to various kinds of auxiliary information to improve the generalization performance. Facial landmarks are the most widely used pre-trained features for AU detection. JPML [30] employs facial landmark features to crop local image patches for different AUs. EAC-Net [25] constructs local regions of interest and spatial attention maps from the facial landmarks. LP-Net [12] trains an individual-specific shape regularization network from the detected facial landmarks. JÂA-Net [26] jointly performs AU detection and facial landmark detection from the data annotated with both labels.

Other types of auxiliary information have also been explored. Zhao et al. [31] pre-train a weakly supervised embedding from a large number of web images. Cui et al.[16] summarize the prior probabilities of AU occurrences as generic knowledge. Emotions and AUs are jointly optimized under the prior probabilities. Recently, SEV-Net [28] leverages textual descriptions of AU occurrences by employing a pre-trained word embedding to obtain auxiliary textual features. To enhance the generalization of our model, our method introduces a pre-trained expression embedding as the auxiliary information.

Due to the local definitions of AUs, it is essential to extract local AU features. As such, some researchers proposed to obtain local information through patch learning. For instance, Zhong et al. [19] and Liu et al. [20] preprocess an input image into uniform patches before encoding to analyze facial expressions. Taking the head pose into consideration, Onal et al.[21, 32] crop AU-specific local facial patches containing information for specific AU recognition after registering the 3D head pose to reduce changes from head movements. Besides, it is a common practice to use attention mechanisms to highlight the features at the facial AU-based positions. Facial landmarks with sparse facial geometric features show an advantage in operating as a supervised attention prior. EAC-Net [25] creates fixed attention maps related to the correlations between AUs and landmarks. JÂA-Net [26] jointly performs AU detection and facial landmark detection, and the predicted landmarks are used to compute the attention map for each AU. Jacob et al. [27] propose a multi-task method that combines the tasks of AU detection and landmark-based attention map prediction. ARL et al. [24] proposed a channel-wise and spatial attention learning for each AU. And, a pixel-level relation is learned by CRF to refine the spatial attention. Except for the facial landmarks, SEV-Net [28] utilizes the textual descriptions of local details to generate a regional attention map. In this way, it highlights the local parts of global features. However, it requires extra annotations for descriptions. Our work proposes a pixel-wise self-attention map that is data-driven without supervision. Our experiments prove that our attention map is superior to the attention maps produced by prior knowledge.

Expressions are essential auxiliary information for AU detection. Some prior works leverage amounts of accessible expression data to enhance AU detection[11][7][31][16]. Recently, Chang et al.[33] utilize a large amount of unlabeled images to train a representation encoder to extract local representations and project them to a low-dimensional latent space, then improve network performance through contrastive learning. Many methods map face images into a low-dimensional manifold for subject-independent expression representations. Early works [15][34][35] train the embeddings for discrete emotion classification tasks but neglect the facial expression variations within each class. The 3D Morphable Model (3DMM) [36][37] has been proposed to fit identities and expression parameters from a single face image. Expressions are represented as the coefficients of predefined blendshapes in 3DMM. The estimated expression coefficients are then used for talking head synthesis [38][39], expression transfer [40][41], and face manipulation [42]. However, the estimated expression coefficients have weaknesses in representing fine-grained expressions. To solve the problem, Vemulapalli and Agarwala [14] proposed a compact embedding for complicated and subtle facial expressions, where facial expression similarity is defined through triplet annotations. Zhang et al. [43] proposed a Deviation Learning Network (DLN) to remove the identity information from continuous expression embeddings, and thus achieve more compact and smooth representations.

The task of AU detection is to predict the AU occurrence probabilities $[o^{1},o^{2},...,o^{N}]$ given an input face image with a resolution of $256\times 256$. As described in section 1, expressions and AUs are two means of description from global and local levels respectively. They are highly related and thus can be utilized to promote each other. Moreover, facial expression data are easier to obtain and annotate, while AUs are more subtle and difficult to annotate. Intuitively, facial AU information can be extracted from facial expression representations when expression representation perception is adequately fine-grained. Considering the above factors, we propose a global expression encoder(GEE) and pre-train it on a large-scale facial expression dataset, aiming to obtain a powerful and robust expression representation. Furthermore, for the purpose of capturing AU local features, we propose a local AU features module (LAM) constructed with two extractors, namely, the AU mask extractor ($E_{m}$) and the AU feature map extractor ($E_{a}$). The framework is illustrated in Fig 2. In the following, we discuss each main module in detail.

To enhance the generalization of our model, we introduce a compact and continuous expression embedding [14][43] as the prior auxiliary information. Specifically, GEE was pre-trained on the expression similarity task (See Fig. 3) using the FEC dataset [14]. The FEC dataset contains numerous facial image triplets along with the annotations indicating which one has the most different expression in each triplet. We pre-trained $GEE$ using the triplet loss to constrain that more similar expressions have a smaller embedding distance. In each triplet, we call the image with the most different expression as the Negative ($N$) and the other two images the Anchor ($A$) and Positive ($P$), respectively. For a triplet (A, P, N), our $GEE$ outputs three expression embeddings $E_{A}$, $E_{P}$ and $E_{N}$. The pre-trained training process utilizes the triplet loss as follows:

Due to the large number of triplets, the pre-trained feature encoder can produce a compact and identity-invariant expression representation, which is beneficial for the downstream expression-related tasks like AU detection.

We choose the structure of InceptionResnetV1 [44] as our backbone of $GEE$, replacing the last classifier layer with a linear layer followed by normalization. After the dimensionality reduction, it outputs a 16-dimensional vector as the expression embedding. We first pre-train $GEE$ on the expression similarity task and then choose the shallow layers (up to the layer with feature maps in a resolution of $16\times 16$) to extract the global expression representation $h$ from the input face image $im$, which is illustrated as:

Then, our whole framework including $GEE$ is fine-tuned for AU detection.

Our intuition is twofold. First, $GEE$ initialized from expression embedding learning could instruct the AU detection framework to capture more effective expression features. Second, it obtains a good initialization at the beginning of training, which helps to relieve the problem of over-fitting caused by the limited data and identities of AU datasets.

The expression features extracted by the pre-trained $GEE$ describe facial movements globally. However, this global representation may not be suitable directly for the subsequent AU detection task, since AUs are defined locally and focus on subtle local motions. Therefore, it is more appropriate to extract local fine-grained features for each AU individually. For this purpose, we design the $LAM$ with two parallel extractors (i.e., a AU mask extractor($E_{m}$) and an AU feature map extractor($E_{a}$)) to transform the global expression features into local AU-related representations.

Specifically, we first extract from the global expression features to produce AU features $f^{i}$ $(i=1,2,...,N+1)$ with our designed AU feature map extractor $E_{a}$. Each AU feature is related to a specific AU. However, we found that the AU features generated by the $E_{a}$ may contain information from unrelated AUs, so we introduce an AU mask extractor($E_{m}$) to find the region of interest for each AU and also mask out irrelevant content.

AU Feature Map Extractor. The AU feature map extractor $E_{a}$ aims at generating AU-related features. Given the global expression feature $h$, our AU feature map extractor ($E_{a}$) performs as follows:

where $f^{k}\in\mathbb{R}^{3\times H\times W}$ is the $k$-th AU feature map that is related to the $k$-th AU, and $N$ is the total number of all AUs. $f^{N+1}$ represents the content that is not related to any AUs (e.g., background).

Considering that there are many AUs on the face, if we design a extracting branch for each AU individually, there will be a lot of parameters. Therefore, we design the AU feature map extractor to be a shared backbone with multiple prediction heads. The shared backbone is formed by stacking multiple residual blocks [45] and upsampling layers. The upsampling design can enlarge some local facial details that benefit the subsequent AU detection and reconstruction tasks. Each of the prediction heads consists of a convolutional layer and an activation layer, and it is responsible for predicting the facial content related to a specific AU.

AU Mask Extractor. The AU mask extractor $E_{m}$ is responsible for generating AU masks maps from the global expression features $h$. Each mask represents the region of interest of each AU. Given the global features $h$, the mask maps are predicted by:

where $m^{k}\in\mathbb{R}^{1\times H\times W}$ is the $k$-th facial mask for the $k$-th AU, and $m^{N+1}$ is for the background.

To ensure that the values of the predicted facial masks lie in a fixed range of 0 to 1 (0 denotes “no attention” and 1 denotes “the strongest attention”), an activation function is added to the last layer of $E_{m}$. Intuitively, Sigmoid and Softmax are both suitable for this purpose. However, Softmax tend to get sparse attention maps and exclude regions related to the target AU. This may result in the downstream recognition task being unable to learn sufficient information. Hence, we choose Sigmoid to ativate the outputs of $E_{m}$

AU Masked Features. Once obtaining the AU specific feature maps and masks, the final AU masked features are generated by multiplying them:

where $c^{i}$($i\in(1,2,...,N)$) is the computed masked AU feature for the $i$-th AU that is then used for the subsequent AU detection task. While $c^{N+1}$ is the masked background content which is used for the subsequent face image reconstruction task.

To ensure these AU feature maps contain all facial features of the input image, we introduce a reconstruction task. Specifically, we design an image reconstruction head ($D_{r}$), which is fed with all masked AU features, and we expect the reconstruction result to be the same as the input facial image.

where $y_{rec}$ is the reconstruction result.

In this way, the $LAM$ not only realizes the generation of AU local/masked features but also retains the complete information of the input face image. Finally, the generated AU masked features are sent to an AU Classifier (described below) for AU recognition.

Since the outputs of the $LAM$ are a series of RGB feature maps with a high resolution ($256\times 256$), and the number of the feature maps equals the number of AUs in the dataset, the following AU classifier module, denoted as $g^{i}(\cdot)$, is designed to be a multi-layer CNN architecture, which contains 6 convolution blocks and a global average pooling layer. Each of the first five convolution layers is followed by a batch-norm and an ELU activation function, and their output channels are set to 16, 32, 64, 128, and 256, respectively. To make sure the sub-module has a wide receptive field, the five CNNs’ kernel size, stride, and padding are set to 7, 2, and 3, respectively. The global average pooling layer is set after the fifth convolution block, aiming to aggregate the feature maps into AU Embeddings. In this way, more information closely related to the specific AU will be extracted from the input AU masked feature map. After that, the AU embeddings are fed into the last convolution layers, where all of the output channel, kernel size, stride, and padding are set to 1. This procedure can be formulated as the following function:

Here $y^{i}$ indicates the output of the last convolution layer of the $i$-th AU. To obtain the final prediction of the AU’s probability, we use a Sigmoid activation on the $y^{i}$ and obtain the predicted probability $o^{i}$ as follows:

As illustrated in Figure 2, our proposed framework refers to two steps. The pretraining procedure is expression similarity learning, while the AU detection part contains two tasks, namely, AU classification and facial image reconstruction. The pretraining loss ${L}_{tri}$ is a triplet loss, referring to Equation (1), and the AU detection training loss $\mathcal{L}_{total}$ contains two parts as follows:

where $\mathcal{L}_{au}$ is the AU classification loss and $\mathcal{L}_{rec}$ is the facial image reconstruction loss, $\lambda$ is a hyperparameter for adjusting the loss ratio. Our main goal is to recognize the AU, the image reconstruction is merely an auxiliary task, so the $\lambda$ is set to a quite low vale of 0.001 in our experiments.

AU detection can be conceptually regarded as a multi-label binary classification problem, so the AU detection loss is defined as:

where $N$ is the number of AU classes, $p_{i}$ is the ground truth, and $\hat{p}_{i}$ is the predicted probability of the $i$-th AU. As for the image reconstruction loss $\mathcal{L}_{rec}$, we use a pixel-level MSE loss between the input image and the reconstructed one as follows:

where $n$ is the pixels number of input image $im$ and equals to $h_{im}\times w_{im}$. Besides, $y^{i}_{rec}$ and $\hat{y}^{i}_{truth}$ are the ground-truth and reconstructed value of the $i$-th pixel of the input image, respectively.

As mentioned above, our evaluation was performed on three widely-used AU detection datasets: BP4D [8], DISFA [9], and BP4D+ [10]. All the above datasets were annotated with AU labels by certified experts for each frame.

BP4D consists of 328 video clips from 41 participants(23 females and 18 males). Each video frame was annotated with the occurrence or absence of 12 AUs(1, 2, 4, 6, 7, 10, 12, 14, 15, 17, 23, and 24. Their definition can be seen in Table I). In total, about 140,000 frames were annotated frame-by-frame by a group of FACS experts. Following the same subject division [25, 26], we performed three-fold, subject-exclusive, cross-validations for all experiments.

DISFA contains 27 video clips from 27 participants(12 females and 15 males). Each video records the facial activity of a participant when watching a 4-minutes video clip. Each clip consists of 4,845 frames evenly. Altogether, around 130,000 frames were annotated with intensities from 0 to 5. Following previous works [27][25][26][28], we regard intensity labels $\{0,1\}$ as absence and others as presence. The evaluation was performed on 8 AUs(1, 2, 4, 6, 9, 12, 25, and 26. Their definition can be seen in Table I). We also performed the same three-fold, subject-exclusive, cross validations on 8 AUs as in the previous works [25][26].

BP4D+ contains 1400 video clips from 140 participants(82 females and 58 males). Compared to BP4D, it has larger identity variations and video numbers. About 198,000 frames were annotated with the same 12 AUs in BP4D. We performed three-fold, subject-exclusive, cross-validations to compare our method with the state-of-the-art. Besides, to evaluate the generalization performance, our method was trained on BP4D and then evaluated on BP4D+, following the previous works [24][26].

To sum up, the order from largest to smallest of the three datasets is BP4D+, BP4D and DISFA. BP4D and BP4D+ were recorded in similar shooting environments and lighting conditions. Both of them share the same 12 AU classes. In comparison, DISFA is the smallest dataset containing only 8 AU classes and was captured in quite different lighting conditions.

Face images were first aligned and cropped according to landmark detection results. Before feeding to our networks, the input images were resized to $256\times 256$. The resolution of feature maps was reduced to $16\times 16$ after passing through the $GEE$.

To enhance the feature extraction capability of low-level convolution layers, the $GEE$ was initialized with the corresponding network parameters pretrained on the dataset of ImageNet or FEC, and the rest of our networks were set to Kaiming initialization [46]. Except for the AU Mask Extractor($E_{m}$), the whole model parameters were updated by employing an Adam optimizer with hyper-parameters $\beta=0.9$ and weight decay $10^{-6}$.

The networks were trained for 20 epochs with an initial learning rate of $10^{-5}$ on each dataset fold. The mini-batch size was set to 10. We used a warm-up stage with $10^{3}$ steps and then decayed the learning rate exponentially with an exponent $-0.5$. The training and inference pipelines were implemented in Pytorch on 4 GeForce GTX 2080ti GPU.

We compared our method with many state-of-the-art AU detection methods, including LSVM [47], JPML [30], DRML [48], APL [49], EAC-Net [25], DSIN [50], ARL [24], SRERL [51], AU-GCN[23],AU-RCNN[29], ML-GCN [52], MS-CAM [53], LP-Net [12], FACS3D-Net [54], UGN-B [55], JÂA-Net [26], MHSA-FFN [27], SEV-Net [28], HMP-PS [56], D-PAttNe$t^{tt}$[21], CISNet[13] and CaFNet[57].

Specifically, while EAC-Net, ARL, JÂA-Net, and SEV-Net were evaluated on all three datasets, LSVM, JPML, DRML, APL, DSIN, SRERL, AU-GCN, AU-RCNN, LP-Net, UGN-B, MHSA-FFN, HMP-PS, CISNet and CaFNet were evaluated on both BP4D and DISFA. Particularly, D-PAttNe$t^{tt}$ was only evaluated on BP4D. FACS3D-Net, ML-GCN and MS-CAM were evaluated merely on BP4D+. Since BP4D+ is released later, it is not used by all the methods.

Following the experimental settings of most previous works, we also employed the F1 score [58] on the frame level as the evaluation metric. F1-score is a harmonic mean of precision(P) and recall(R), which is defined as $F1=2PR/(P+R)$. Since F1-score takes both false positives and false negatives into account, it reflects a more rational measurement of the model’s performance than accuracy, especially on datasets with uneven class distributions like BP4D, DISFA and BP4D+.

Evaluation on BP4D. The results on BP4D are shown in Table II. The average F1 score of our method is 66.3%, which significantly outperforms all the other comparison methods, improving about 1.4% over the second highest one, CaFNet. In terms of the performances on single AUs, our method achieves the highest F1 scores among all the methods on 5 AUs(i.e., AU1, AU4, AU12, AU15, and AU23) and second highest on 1 AU(AU7) of the 12 evaluated AUs. Besides, our method shows over 3.5% superiority on AU15 and AU23 over the second highest ones.

In terms of AU feature learning, our method outperforms JMPL, DRML, and D-PAttNe$t^{tt}$ about 20.4%, 18% and 1.6%, demonstrating that extracting local features by attention is more effective and our method is superior in AU feature extraction than patch learning.

Compared with EAC-Net, JAA-Net and MHSA-FFN, which utilize attention mechanisms with the supervisions of pre-defined AU centers, our method achieves improvements of 10.4%, 5.2% and 3.9% over them, respectively. This validates the superiority of our non-predetermined AU attention learning approach.

Evaluation on DISFA. Table III shows the evaluation results on DISFA. As shown in this table, our method achieves the average F1 score of 66.6%, outperforming all the compared state-of-the-art methods.

Compared to the most recent works SEV-Net, HMP-PS, MHSA-FFN, CISNet and CaFNet, our method achieves an improvement of 7.8%, 5.6%, 5.1%, 1.9% and 0.8% over them, respectively. Unlike EAC-Net and JÂA-Net which are supervised by both AU and landmark labels, our method only utilizes AU labels, but still obtains significant improvements of 18.1% and 3.1% over EAC-Net and JÂA-Net, respectively, demonstrating the effectiveness of our method in AU feature and attention learning. Note that our method does not explicitly model AU relationships, but significantly exceeds all relation-based methods including JMPL, DSIN, ARL, AU-GCN and AU-RCNN.

Evaluation on BP4D+. Table IV shows the evaluation results of the three-fold cross-validation on BP4D+. In particular, the results of SEV-Net, ML-DCN and MS-CAM are reported in[28] while the results of EAC-Net are reported in [26]. Our method obtains the highest or the second highest results on 11 of the 12 evaluated AUs. It achieves an average F1 score of 65.7% and outperforms all the state-of-the-art methods, exceeding the second highest method(SEV-Net) over 4.2%.

To further evaluate the generalization performance of our method on large-scale test identities, we trained our method on BP4D and evaluated it on the full BP4D+ of 140 subjects. The results are shown in Table V. We use the reported results in the work [26] for comparison. Again, our method outperforms all the compared methods under the large-scale cross-dataset evaluation. It indicates that our method can extract identity-independent information and generalize well to new identities.

In this section, we conduct ablation studies to evaluate the effectiveness of each key sub-module in our framework, including: 1) the prior knowledge of expression similarity learning, and 2) the local AU features module. In particular, all the ablation experiments are conducted on the BP4D dataset due to space limitations. The experimental results are reported in Table VI.

Prior Knowledge. Our framework proposes a vision feature encoder $GEE$ that is pretrained on the FEC dataset via expression similarity learning, which refers to our prior knowledge. To test the efficiency of prior knowledge, we design ablation experiments with different approaches to initialize $GEE$ in the training of GTLE-Net, including:

• a)

  w/o pretrain: $GEE$ is randomly initialized;

• b)

  w.GEE Emotion: $GEE$ is initialized via a pretrained classifier of seven discrete facial expression emotion (e.g. neutral, happy, sad, surprise, fear, disgust, angry) categories trained on the AffectNet dataset;

• c)

  w. GEE ImageNet: $GEE$ is initialized via a pretrained image object classifier built on imageNet [59];

• d)

  GTLE-Net(Full): our proposed method, $GEE$ is initialized via the facial expression similarity learning;

• e)

  w. GEE fixed: $GEE$ is parameterized by the facial expression similarity approach but not updated in the training of GTLE-Net.

Firstly, in comparison with w/o pretrain, w.GEE Emotion, w. GEE ImageNet and w. GEE fixed achieve improvements of 9.8%(from 50.5% to 60.3%), 11.9%(from 50.5% to 62.4%) and 15.8%(from 50.5% to 66.3%), respectively. This indicates that the prior knowledge is necessary and the pretraining progress can bring significant improvement for AU detection.

Secondly, compared with the w. GEE Emotion and w. GEE ImageNet, GTLE-Net obtains the best performance(66.3%), exceeding the other two pretraining methods about 6% and 3.9%. This proves the effectiveness and superiority of the prior knowledge captured by our facial expression similarity learning.

Lastly, compared with GTLE-Net, w. GEE fixed obtains a lower average F1 of 63.5%. It indicates that finetuning the global expression feature encoder is beneficial for extracting features which are more adaptive to the following LAM module. Thus finetuning operation is essential and efficient to improve the performance of our framework.

Facial Components. To validate the effectiveness of the representations extracted by our local AU features module (LAM), we conduct three ablation experiments. Concretely, based on our full framework of GTLE-Net, we remove or replace some specific sub-modules and retrain the whole network, including:

• f)

  w/o $E_{m}$&$E_{a}$: removing both the $E_{m}$ and the $E_{a}$;

• g)

  w/o $E_{m}$: only removing $E_{m}$;

• h)

  w/o $E_{m}$ & w. CNN: removing the $E_{m}$ and replacing the $E_{a}$ with a CNN block in the meanwhile. Particularly, the CNN block has similar number of parameters(around 16.6M) with our feature encoder($GEE$).

Note that the inclusion of $E_{a}$ alone(w/o $E_{m}$) brings an improvement of 0.5%(from 63.3% to 63.8%) in comparision to w/o $E_{m}$&$E_{a}$. When the $E_{a}$ is replaced by the CNN module(w/o $E_{m}$ & w. CNN ), the performance declines by 0.1%(from 63.8% to 63.7%). Moreover, the F1-Score improves by 2.5% when we further add the $E_{m}$ module(GTLE-Net). These study results validate the effectiveness of our AU mask and local feature learning.

We also visualize the feature and mask results of our framework to further validate its effectiveness.

Mask Attention Learning. Due to its anatomical definition, each AU involves a group of muscle motions and is related to a specific region on the face. For instance, AU1 is inner brow raising and AU9 is nose wrinkle. Theoretically our framework should learn a regional mask activation map for each AU, corresponding to its muscle actions.

Figure 5 visualizes the attention mask maps of several examples from DISFA, BP4D and BP4D+, learned by our network. As shown in this figure, we can see that our method successfully captures a discriminative attention mask for these AUs. Taking AU4 (brow low) and AU17 (chin raise) as examples, the learned spatial masks are around the brows and chin, respectively, which demonstrates that our proposed AU mask extractor is effective for AU-related local regional attention learning.

From Figure 4, we can see that each AU’s mask is closely related to its muscle region, even for different expressions such as happy, sad and fear. We also notice that several different AUs may cover the same or similar area. This is attributed to that different AUs may share the same or similar muscle distributions. For instance, all of AU1, AU2 and AU4 are involved with the forehead, while all of AU14, AU15, AU17, AU23 and AU24 are closely related to the mouth area. Besides, considering the existence of potential mutual relations among different AUs, the masks learned merely supervised by AU labels may include other facial regions of correlated AUs. For instance, the masks of AU1, AU2, and AU4 not only cover the regions of the brows but also contain the mouth area.

Mask Features Analysis. To validate the importance of mask learning, we visualize the AU embeddings of the AU classifiers in the GTLE-Net and w/o $E_{m}$, respectively. We sample one thousand images and obtain the t-SNE visualization of their AU embeddings.

As illustrated in Figure 6, each color refers to a class of AU presentations. Obviously, the presentations learned with mask(GTLE-Net) are more compact and have better distribution than that generated by model without mask(w/o $E_{m}$). Moreover, all the AU embeddings learned with mask(GTLE-Net) tend to gather into two clusters(occurred and absent samples). By contrast, the presentations generated without the mask extractor(w/o $E_{m}$) display like a long strip and are not divided into two clusters for the cases of AU2, AU4, AU15, AU17 and AU23. As a result, these AUs cannot be well distinguished. This validates that the proposed LAM of our framework is effective to capture discriminative local features that are beneficial to AU recognition.