Exploring Adversarial Learning for Deep Semi-Supervised Facial Action Unit Recognition

Due to the rapid development of deep learning in recent years, several works have utilized deep AU analysis.

Some works adopt deep networks such as convolutional neural networks (CNNs) to learn spatial representations. They then construct an end-to-end network, treating AU recognition as a multi-output binary classification problem. For example, Ghosh et al. (Ghosh et al., 2015) and Gudi et al. (Gudi et al., 2015) propose a multiple output CNN that learns feature representation among AUs from facial images. Khorrami et al. (Khorrami et al., 2015) experimentally validate the hypothesis that CNNs trained for expression category recognition may be beneficial for AU analysis, since learned facial representations are highly related to AUs. Li et al. (Li et al., 2018) introduce enhancing and cropping layers to a pre-trained CNN model. The enhancing layers are designed attention maps based on facial landmark features, and the cropping layers are used to crop facial regions around the detected landmarks and learn deeper features for each facial region. Han et al. (Han et al., 2018) propose an advanced optimized filter size CNN (OFS-CNN) for AU recognition, which is capable of estimating optimal kernel size for varying image resolutions, and outperforms traditional CNNs. Shao et al. (Shao et al., 2018) propose a joint AU detection and face alignment framework called JAA-Net. Adaptive attention learning module is proposed to localize ROIs of AUs adaptively for better local feature extraction. Shao et al. (Shao et al., 2019) also propose an attention and pixel-level relation learning framework for AU recognition. Sankaran et al. (Sankaran et al., 2020) propose a novel feature fusion approach to combine different kinds of representations. Niu et al. (Niu et al., 2019) utilize local information and the correlations of individual local facial regions to improve AU recognition. A person-specific shape regularization method is also involved to reduce the influence of person-specific shape information. Those works totally ignore AU dependencies while training AU classifiers.

Other works try to incorporate AU dependencies and discriminative facial regions into deep networks. For example, Zhao et al. (Zhao et al., 2016) propose Deep Region and Multi-Label Learning (DRML) to address region learning and multi-label learning simultaneously. They adopt multi-label cross-entropy loss to model AU dependencies. As with Zhao et al. (Zhao et al., 2016), multi-label cross-entropy loss is adopted to model AU dependencies. Multi-label cross-entropy loss is the sum of the binary cross entropy of each label. Therefore, it does not capture label dependencies effectively.

In addition to using CNN to capture spatial representations, several works integrate long short-term memory neural networks (LSTM) to jointly capture spatial and temporal representations for AU recognition. For example, Jaiswal and Valstar (Jaiswal and Valstar, 2016) directly combine CNNs with bidirectional long short-term memory neural networks (CNN-BiLSTM) by using the spatial representations learned from the CNN as the input of BiLSTM. Their work does not consider AU dependencies. Li et al. (Li et al., 2017) propose a deep learning framework for AU recognition that integrates region of interest (ROI) adaptation, multi-label learning, and optimal LSTM-based temporal fusing. Their multi-label learning method combines the outputs of the individual ROI cropping nets, leveraging the inter-relationships of various AUs from facial representations without considering AU dependencies among target labels. Chu et al. (Chu et al., 2017) propose a hybrid network architecture to jointly capture spatial representations, temporal dependencies, and AU dependencies through CNN, LSTM, and multi-label cross-entropy loss respectively. As mentioned above, multi-label cross-entropy loss summarizes cross entropy of each label and does not effectively exploit label dependencies.

In summary, current deep AU recognition requires fully AU-labeled facial images. Although some methods employ multi-label cross-entropy loss to handle recognition of multiple AUs, they cannot model AU dependencies effectively since the summarization of cross entropy from each label does not represent label dependencies.

Unlike the more recent interest in deep AU recognition techniques, AU recognition using shallow models has been studied for years. These shallow models explore AU dependencies through both generative and discriminative approaches.

For generative approaches, both directed graphic models (such as dynamic Bayesian networks (DBNs) (Tong et al., 2007) and latent regression Bayesian networks (LRBNs) (Hao et al., 2018)) and undirected graphic models (such as restricted Boltzmann machine (RBM) (Wang et al., 2013)) are utilized to learn AU dependencies from ground truth AU labels. Corneanu et al. (Corneanu et al., 2018) combine CNN and recurrent graphical model and propose Deep Structure Inference Network (DSIN) to deal with patch and multi-label learning for AU recognition. Through parameter and structure learning, graphic models can successfully capture AU distributions using their joint probabilities. Li et al. (Li et al., 2019) propose an AU semantic correlation embedded representation learning framework (SRERL). AU knowledge-graph is utilized to guide the enhancement of facial region representation. However, these joint probabilities have certain analytical forms and assume the inherent AU distributions follow these forms. Such an assumption may be false, since we do not have complete knowledge about AU distributions caused by muscle interactions. Furthermore, approximation algorithms are often used for the learning and inference of graphic models. This further impedes the ability of graphic models to faithfully represent AU distributions.

Discriminative approaches use additional constraints of the loss function to represent AU relations. For example, Zhu et al. (Zhu et al., 2014) and Zhang et al. (Zhang and Mahoor, 2014) use the constraints of multi-task learning to explore AU co-occurrences among certain AU groups. Zhao et al. (Zhao et al., 2015) investigate the constraints of group sparsity as well as local positive and negative AU relations to select a sparse subset of facial patches and learn multiple AU classifiers simultaneously. Eleftheriadis et al. (Eleftheriadis et al., 2015) propose constraints to encode local and global co-occurrence dependencies among AU labels to project image features onto a shared manifold.

Each of the proposed constraints can model certain AU dependencies, but none represent the hundreds of AU relations embedded in AU distributions. Furthermore, all of these works require fully AU-labeled facial images.

Most current work on AU recognition formulates it as a supervised learning problem and requires fully AU-labeled facial images. Researchers have only recently begun to address AU recognition when data is partially labeled. Some of these works require expression annotations, leveraging the dependencies between AU and expressions to complement missing AU annotations. For example, Wang et al. (Wang et al., 2017) propose a Bayesian network (BN) to capture the dependencies among AUs and expressions. For missing AU labels, the structure and parameters of the BN are learned through structured expected maximization (EM). Ruiz et al. (Ruiz et al., 2015) learn the AU classifier with a massive data set of expression-labeled facial images. They generate pseudo AU labels by exploiting the prior knowledge of the relations between expressions and AUs. Wang et al. (Wang et al., 2018) and Peng et al. (Peng and Wang, 2018) leverage an RBM and adversarial learning, respectively, to model AU relations in the task of AU recognition assisted by facial expression. Zhang et al. (Zhang et al., 2018) propose a knowledge-driven method to jointly learn multiple AU classifiers from images without AU annotations by leveraging prior probabilities of AUs and expressions. These works require expression labels to complement the missing AU labels.

Other methods do not require expression annotations. For example, Wu et al. (Wu et al., 2015) and Li et al. (Li et al., 2016) leverage the consistency between the predicted labels, the ground truth labels, and the local smoothness among the label assignments to handle the missing labels for multi-label learning with missing labels (MLML). However, the smoothness assumption is likely to be incorrect because adjacent samples in the feature space may belong to the same subject rather than the same expression. Song et al. (Song et al., 2015) tackle missing labels by marginalizing over the latent values during the inference procedure for their proposed Bayesian Group-Sparse Compressed Sensing (BGCS) method. Their work handles missing labels by leveraging the characteristics of generative models, but may not outperform discriminative models in classification tasks.

Although the above works consider AU recognition under incomplete AU annotations, they all use hand-crafted features. As such, none can fully explore the advantages of deep learning when studying many facial images.

Wu et al. (Wu et al., 2017) recently proposed a deep semi-supervised AU recognition method (DAU-R) from partially AU-labeled data and AU distributions captured by RBM. Specifically, a deep neural network is learned for feature extraction. Then, an RBM is learned to capture the inherent AU distribution using the available ground truth labels. Finally, the AU classifier is trained by maximizing the log likelihood of the AU classifiers with regard to the learned label distribution while minimizing the error between predictions and ground truth labels from partially labeled data. Their approach makes full use of both facial images with available ground truth AU labels and the massive quantity of facial images lacking annotations. However, as mentioned before, RBM is a graphic model with certain forms of joint probability. Since we do not have complete knowledge about the inherent AU distributions, the assumed analytical forms of probability may be invalid.

Therefore, in this paper we introduce an adversarial learning mechanism to learn the AU distribution directly from available ground truth labels without any assumptions of distribution form. Specifically, we introduce a discriminator $\mathcal{D}$ to distinguish the ground truth AU labels from the AU labels predicted by classifier $\mathcal{R}$. A deep network is used to learn facial representations and classifiers from both AU-labeled and unlabeled facial images by minimizing the recognition errors for the AU-labeled data and maximizing the probability of $\mathcal{D}$ making mistakes. Through an adversarial mechanism, the learned AU classifiers can minimize the predicted errors on labeled images and output AUs from any images following inherent AU distributions.

Our contribution can be summarized as follows. (1) We are among the first to introduce adversarial learning for AU distributions, and to leverage such distributions to construct AU classifiers from partially labeled data. (2) We conduct AU recognition experiments with completely labeled data and partially labeled data. Experimental results demonstrate the potential of adversarial learning in capturing AU distributions, and the superiority of the proposed method over state-of-the-art methods.

The goal of this paper is to learn AU classifiers from a large quantity of facial images with limited AU annotations. Since certain AU distributions are controlled by anatomic mechanisms, they must be generic to all facial images regardless of annotation status. We leverage these inherent AU distributions to construct AU classifiers from partially labeled facial images. In addition to minimizing the loss between the predicted AU labels and the ground truth labels for the labeled facial images, we use adversarial learning to enforce statistical similarity between the AU distribution inherent in ground truth AU labels and the distribution of the predicted AU labels from both labeled and unlabeled facial images.

Let $T=T_{1}\cup T_{2}$ denote the training set, where $T_{1}=\{\mathbf{x}^{t}_{n},\mathbf{y}^{t}_{n}\}_{n=1}^{N}$ denotes the subset of $d$ dimensional training instances $\mathbf{x}^{t}_{n}\in\mathbb{R}^{d}$ with AU annotations and the corresponding ground truth label $\mathbf{y}^{t}_{n}\in\{1,0\}^{l}$, where $N$ is the number of instances and $l$ is the number of AU labels. $\mathbf{X}^{t}=\{\mathbf{x}^{t}_{n}\}_{n=1}^{N}$ and $\mathbf{Y}^{t}=\{\mathbf{y}^{t}_{n}\}_{n=1}^{N}$ store the labeled facial images and corresponding AU labels respectively. $T_{2}=\{\mathbf{x^{\prime}}_{m}\}_{m=1}^{M}$ denotes the subset of training instances without AU annotations, where $M$ is the number of instances. $\mathbf{X}=\mathbf{X}^{t}\cup T_{2}$ stores all facial images. Given the training set with partial AU annotations, our goal is to learn an AU classifier $\mathcal{R}\colon\mathbb{R}^{d}\to\{1,0\}^{l}$ by optimizing the following formula:

where $\mathcal{L}s$ is the loss between the predicted label and the ground truth label. $\mathcal{R}$ is the AU classifier and $\Theta$ are parameters of $\mathcal{R}$. $\mathbf{y}$ and $\mathbf{y^{\prime}}$ are predicted label corresponding to the labeled sample $\mathbf{x}^{t}$ and the unlabeled sample $\mathbf{x^{\prime}}$ respectively. $P_{\mathbf{y},\mathbf{y^{\prime}}}$ is the distribution of the predicted AU labels, and $P_{\mathbf{y}^{t}}$ is the distribution of the ground truth AU labels. $\mathbf{d}$ represents the distance between two distributions. $\alpha$ is the trade-off rate between the two terms in Equation 1.

In practice, it is difficult to model the distribution of the predicted AU labels ($P_{\mathbf{y},\mathbf{y^{\prime}}}$), and errors may occur during the modeling procedure. To alleviate these problems, we close the two distributions through an adversarial learning mechanism.

We propose a novel AU recognition method leveraging adversarial learning mechanisms, inspired by Goodfellow’s generative adversarial network (Goodfellow et al., 2014). In addition to the AU classifier $\mathcal{R}$, we introduce an AU discriminator $\mathcal{D}$ to leverage the distribution constraint from limited ground truth labels. Figure 2 displays the framework of the proposed method.

A training batch consisting of labeled facial images $\mathbf{x}^{t}$ and unlabeled facial images $\mathbf{x^{\prime}}$ sampled from training set $T$ is inputted to AU classifier $\mathcal{R}$. The training batch is used to obtain the predicted AU labels $\mathbf{y}$ and $\mathbf{y^{\prime}}$, which correspond to $\mathbf{x}^{t}$ and $\mathbf{x^{\prime}}$. The predicted AU labels from $\mathcal{R}$ are regarded as “fake”, and the ground truth $\mathbf{y}^{t}$ labels corresponding to $\mathbf{x}^{t}$ are regarded as “real”. Both “real” ground truth AU labels and “fake” predicted AU labels are inputted to AU discriminator $\mathcal{D}$. $\mathcal{D}$ tries to distinguish the “real” AU labels from the predicted AU labels, while $\mathcal{R}$ tries to fool $\mathcal{D}$ into making mistakes. By leveraging the competition between $\mathcal{R}$ and $\mathcal{D}$, the distribution of the predicted AU labels nears the distribution of the ground truth AU labels until convergence. Ground truth AU labels could provide supervisory information for learning the AU classifier, minimizing the error between the predicted AU labels and the ground truth AU labels for AU-labeled samples. Therefore, we get the following objective:

Observing the above equation, when $\alpha=0$, the AU classifier $\mathcal{R}$ is learned without the constraints of the label distribution.

The proposed method can be trained in a fully supervised manner if all facial images in $\mathbf{X}$ are labeled with action units. Otherwise, the proposed method is learned in a semi-supervised manner.

Similar to (Goodfellow et al., 2014), we do not optimize Equation 2 directly, but use an iterative and alternative learning strategy as Equations 3 and 4.

$\mathcal{L}_{\mathcal{D}}$ is for AU discriminator $\mathcal{D}$ and $\mathcal{L}_{\mathcal{R}}$ is for AU classifier $\mathcal{R}$. In practice, it is better for $\mathcal{R}$ to minimize $-\log\mathcal{D}(\mathcal{R}(\mathbf{x}))$ instead of minimizing $\log(1-\mathcal{D}(\mathcal{R}(\mathbf{x})))$, to avoid gradient vanishing of the AU classifier (Goodfellow, 2016). The detailed training procedure is described in Algorithm 1.

In our work on AU recognition, we use cross-entropy loss, so $\mathcal{L}s$ can be written as Equation 5:

We use a three-layer feedforward neural network for the structure of the AU discriminator. For the AU classifier, we enable end-to-end learning directly from the input facial image via a deep CNN. Specifically, ResNet-50 (He et al., 2016) is used as the backbone network and one fully connected layer is built upon the 2048D feature vectors.

Any gradient-based method could be used to update the parameters and optimize Equations 3 and 4. The Adam (Kingma and Ba, 2014) optimization algorithm is used in our work.

To validate the proposed adversarial AU recognition method, we conducted experiments on two benchmark databases: the BP4D database (Zhang et al., 2013) and the Denver Intensity of Spontaneous Facial Action (DISFA) database (Mavadati et al., 2013).

The BP4D database provides both 2D and 3D spontaneous facial videos of eight facial expressions recorded from 41 subjects. Among them, 328 two-dimensional videos in which each frame is a 2D image are coded with 12 AUs: 1, 2, 4, 6, 7, 10, 12, 14, 15, 17, 23, and 24. In total, there are around 140,000 valid image samples. Like most related works, we use all available AUs and all valid samples.

The DISFA database contains spontaneous facial videos from 27 subjects watching YouTube videos. Each image frame is coded with 12 AUs: 1, 2, 4, 5, 6, 9, 12, 15, 17, 20, 25, 26. The annotations of AUs are represented as intensities ranging from zero to five. The number of valid image samples is around 130,000. Like most related works, we treated each AU with an intensity equal or greater than 2 as active, and considered 8 AUs (i.e., 1, 2, 4, 6, 9, 12, 25, and 26). For both databases, the number of occurrences per AU are shown in Figure 3.

Facial alignment was taken into account to extract the face region for each image. Specifically, the face region is aligned based on three fiducial points: the centers of each eye and the mouth. After cropping and warping the face region, all images in both databases were normalized to $224\times 224\times 3$ pixels.

We conduct semi-supervised AU recognition experiments on images with incomplete AU annotations and fully supervised AU recognition experiments on completely AU-annotated images on both databases. For semi-supervised scenarios, we randomly miss AU labels according to certain proportions: 10%, 20%, 30%, 40%, and 50%. On the BP4D database, we adopt three-fold subject-independent cross validation, a commonly used experimental strategy for AU recognition. For a fair comparison to (Wu et al., 2017) and (Han et al., 2018), we apply their AU selection and data split strategy on the BP4D database. On the DISFA database, similar to most related works, we adopt 3-fold subject-independent cross validation. Because of the different experimental conditions with (Wu et al., 2017) and (Han et al., 2018), we don’t conduct 9-Folds and 10-Folds experiments on the DISFA database. We adopt F1 score, AUC and accuracy to evaluate the performance of the proposed method. Moreover, the best values of alpha in the above-mentioned experiments are the same of the optimal values in Section 4.3.2.

To demonstrate the effectiveness of the adversarial mechanism in the semi-supervised scenario, we compare our method to the method without label constraint (“O-wlc” for short). For this method, we set the tradeoff rate $\alpha=0$ in Equation 2 to remove the AU label distribution constraint. For the semi-supervised scenario, we also compare the proposed method to DAU-R (Wu et al., 2017) and BGCS (Song et al., 2015). We re-conduct the semi-supervised experiments of BGCS using their provided codes, since Song et al. (Song et al., 2015) didn’t conduct semi-supervised experiments on the BP4D or DISFA databases. Totally, for experiments on the DISFA database, we compare our method to BGCS and O-wlc. DAU-R is not involved due to the difference of experimental conditions. And O-wlc, DAU-R and BGCS are employed to compare with our method on the BP4D database. In addition, we don’t compare our results with MLML(Wu et al., 2015) on both databases because of the memory problems. We do not compare our method to the methods found in (Ruiz et al., 2015; Wang et al., 2017; Wang et al., 2018; Peng and Wang, 2018) and (Zhang et al., 2018) either, since these works require expression annotations.

For the fully supervised scenario, we compare the proposed method to state-of-the-art works, including ARL (Shao et al., 2019), LP-Net (Niu et al., 2019), SRERL (Li et al., 2019), U-Net (Sankaran et al., 2020), DSIN (Corneanu et al., 2018), JAA-Net (Shao et al., 2018), EAC-Net (Li et al., 2018), OFS-CNN (Han et al., 2018), DAU-R (Wu et al., 2017), CPM (Zeng et al., 2015), DRML (Zhao et al., 2016), JPML(Zhao et al., 2015), and APL(Zhong et al., 2014). The results of CPM are taken from (Chu et al., 2017), since Zeng et al. (Zeng et al., 2015) did not provide the results on three-fold protocol. The experiments conducted in Hao et al.’s work (Hao et al., 2018) are based on the apex facial images with expression labels, and (Li et al., 2017) and (Chu et al., 2017) use temporal models, so these methods are ignored in the comparison. In (Li et al., 2017), the best results from non-temporal model (ROI) are used for comparison.