Self-Supervised Regional and Temporal Auxiliary Tasks for Facial Action Unit Recognition

Many efforts have been dedicated to AU recognition for the past decades. For the purpose of performance improvement, discriminative regional feature learning, AU relationship embedding and temporal feature learning are fully explored in supervised methods.

In order to capture regional features for different facial structures, Zhao et al. (Zhao et al., 2016) proposed a region layer which divides the feature maps into identical regions, then independent convolutions were applied to each region. Niu et al. (Niu et al., 2019b) proposed the LP-Net to model the person-specific shape information and explore the relationship among local regions simultaneously. Corneanu et al. (Corneanu et al., 2018) and Zhang et al. (Zhang et al., 2019) cropped several relatively large regions from the original facial image and employed individual CNNs on them. Li et al. (Li et al., 2017) defined AU centers based on the rough position relationship between AUs and landmarks, and proposed a region of interest network to crop regions around these AU centers. This landmark based region localization method was followed by (Shao et al., 2018; Li et al., 2019b).

The relationship between AUs has been exploited to boost the recognition performance. Corneanu et al. (Corneanu et al., 2018) proposed to capture AU correlations by deep structured inference network which was applied on the AU predictions in a recurrent manner. Li et al. (Li et al., 2019b) proposed a semantic relationship embedded representation learning (SRERL) framework to embed the relation knowledge in regional features with gated graph neural network. Graph convolutional network (GCN) (Kipf and Welling, 2017) was also utilized to encode the prior AU relationship information into feature representation (Niu et al., 2019a; Liu et al., 2020; Song et al., 2021; Yan et al., 2021). The graph propagation based methods often utilize a fixed AU relation graph to embed the relationship which is not suitable for some facial samples. In our framework, a transformer is leveraged due to the flexibility and the data driven manner and a self-supervised task is proposed to enhance the AU relation learning.

From the temporal aspect, an action unit essentially describes the motion of facial muscles. Thus it is helpful to leverage the temporal features of facial textures. To capture such dynamic change information, Chu et al. (Chu et al., 2016) and Jaiswal et al. (Jaiswal and Valstar, 2016) combined CNN and LSTM to model the AU sequence spatially and temporally. Similarly, Li et al. (Li et al., 2017) employed LSTM to fuse the temporal features of AU RoIs and improved the model performance greatly compared to the model without using temporal information. However, these methods are performed on the frame sequence and cannot be adapted to the image based models. Meanwhile, the complex network structures for temporal feature learning make the calculation inefficient during inference process. In our framework, a self-supervised auxiliary task is proposed to enhance the temporal representation capacity during training with a single static image, thus it does not increase the computational cost for AU recognition.

Some efforts were also made to integrate auxiliary tasks with AU recognition. Most of the approaches leveraged the recognition of facial expression or other related facial attributes which still required for corresponding manual annotations (Yang et al., 2016; Chang et al., 2017; Wang et al., 2017). Shao et al. (Shao et al., 2018) integrated the facial alignment task with AU recognition to learn better local features, while in our method, the two auxiliary tasks are designed to model AU characteristics from three crucial components via the unlabeled data in a self-supervised manner, which is more effective and does not need additional annotations.

SSL methods first deduce the ground truth labels or other supervisory signals from the designed pretext task itself. Then the ground truth is leveraged to supervise the training of the feature learning model. Through the designed SSL task, generic features are learned for the downstream task. For AU recognition, current works are mainly motivated by the temporal movement of facial muscles. Wiles et al. (Koepke et al., 2018) proposed FAb-Net to generate from the source frame to target frame by low-dimensional attribute embedding. Li et al. (Li et al., 2019a) also utilized the transformation between the source and target frames and furthermore proposed the twin-cycle autoencoder to obtain dedicated AU features without head motions. Inspired by the intrinsic temporal consistency in videos, Lu et al. (Lu et al., 2020) used a triplet-based ranking approach to rank the frames. These methods mainly focus on discriminative global feature learning and ignore some unique properties of AUs. Regional feature leaning and AU relation embedding are not taken into consideration. Meanwhile, these SSL tasks are performed on video samples and cannot be adopted to tremendous unlabeled facial images.

The backbone network is illustrated in the middle part of Figure 2. ResNet-18 (He et al., 2016) is chosen as the foundation of the backbone due to the effectiveness and efficiency in feature learning. As each AU is defined on the corresponding local facial region, individual RoI feature learning module which consists of two convolutional layers for each AU is utilized to deal with different facial textures and structures. We follow the AU RoI definition proposed in (Li et al., 2017). In order to get relatively high spatial resolution feature maps while keeping sufficient semantic information before cropping AU RoIs, the high-level feature maps are gradually upsampled and added with the low-level feature maps. Then features of each AU RoI are cropped from the fused global feature maps. Due to the symmetry of human face, each AU corresponds to two RoI features. A lightweight transformer with one encoder and one decoder is employed to model the AU relationship adaptively and encode the relation to RoI features, as shown in Figure 3. We calculate the average of the two enhanced RoI features derived from the transformer for each AU and regard it as the AU RoI representation. Finally, based on local RoI features and overall global features, AU predictions made by separate fully connected layers are fused together by maximizing.

The transformer was originally designed for nature language processing (NLP) tasks based on the attention mechanism (Vaswani et al., 2017). Compared to recurrent neutral networks, it is more parallelizable and easier to train. A typical transformer is composed of multiple encoders and decoders. For a transformer encoder, it takes the feature sequence as input and output the attention features adaptively. Feature positions in the input sequence are represented by the positional encoding. As there is no ordinal relation between AUs in the sequence, positional encoding is not added in the encoder.

As a key module in both encoders and decoders, the self-attention layer aims to explore relationship between each input feature and all others in the sequence via the self-attention mechanism. For a transformer encoder, the related AU RoI features can be attended to the corresponding one adaptively and the attention feature can be formulated as follows,

where $Q$, $K$ and $V$ are linear transformations of the input feature sequence, which is formed as a matrix $X$. In our model, feature maps obtained from RoI feature learning of each AU are vectorized first by average pooling, and then joined together to form the input feature matrix $X$. Assume there are $N$ AUs and the feature dimension of each RoI is 128, then $X\in\mathbb{R}^{128\times N}$. After linear transformation, the feature dimension is denoted as $d$. Other modules in the encoder such as add & norm and feed forward network are the same with the standard structure.

As shown in Figure 3, the transformer decoder transforms the feature embeddings with multi-head self-attention and encoder-decoder attention. It is notable that as the decoder is permutation invariant, in order to capture the aggregated regional features for each AU correctly, $N$ positional encodings, named AU indicators, are added to the input of the decoder. Thus each output feature of the decoder can be matched with the exact corresponding AU. Similar to word embedding in NLP, AU indicators are formulated as a $d\times N$ matrix where each $d$-dimensional vector indicates one specific AU. After training the learned AU indicators should contain AU relation knowledge. We will visualize the encoded relationship in the next section.

The RoI features output from the transformer, which contain enough discriminative and representative information, are then utilized to obtain regional AU predictions and to generate the missing patches for RoI inpainting. Compared to conventional graph propagation methods, where the AU relation graph is often fixed based on prior knowledge or the statics of AU labels in database, transformer is more flexible and adaptive for each sample. The learning procedure of AU relation and feature embedding is totally data driven and does not require to build the AU relation graph manually.

As AU relationship learned from the limited annotated data may not be accurate and complete, we expect to mine reasonable and comprehensive AU relation knowledge from a large amount of unlabeled data. To this end, we propose a task of RoI inpainting (RoII), which aims for recovering the cropped regions of a random AU in the original facial image. Instead of utilizing global features to generate the cropped patches as conventional image inpainting does (Pathak et al., 2016), we leverage RoI features of the intact regions and the learned adaptive relationship to represent the cropped parts for recovery to enhance the AU relationship modeling. It is notable that some AUs share the same RoIs, such as AU12 (lip corner puller) and AU15 (lip corner depressor). As they can hardly be activated together, there is no positive correlation between them. When these shared RoIs are cropped, the RoII task can still be completed by the related RoI features.

In addition to the enhanced discriminative regional features, it is valuable to exploit the motion information of facial muscles for AU recognition. Previous SSL approaches utilized complex networks to model the transformation from the source frame to the target one to obtain the AU related feature representation. Here we propose a self-supervised task of optical flow estimation based on the static facial image to model the temporal change of facial muscles elegantly and effectively. As optical flow exactly describes the pixel movements of facial muscles between two frames, it can be served as the supervisory signals of the dynamic change of facial textures to guide the network to exploit motion information of corresponding local muscles.

For a video sequence, we first extract the TV-L1 optical flow (Zach et al., 2007) between two facial frames. As the frame difference between a short time step can hardly be captured while the frame after a long step is not effective for AU recognition in the current frame, the step length is set to 3 frames empirically. Meanwhile, to alleviate the effect of head motion as much as possible, we align the face in the latter frame with the transformation matrix obtained in the previous frame. The extracted optical flow is prepared before training.

As shown in the red dashed box in Figure 2, global feature maps output from ResNet-18 are leveraged to estimate the optical flow. In this way, the backbone network is forced to learn motion related features from the static image. The network structure for optical flow estimation is composed of 2 layers of transposed convolution which aims to gradually model the small pixel-wise displacement. Thus the objective function can be formulated as follows,

where $f_{g}$ and $f_{p}$ are the TV-L1 and predicted optical flow respectively. Similar to the reconstruction loss in the previous section, $l_{1}$-norm is used to measure the difference between the predicted and the TV-L1 optical flow. Another implicit benefit of OFE is that the temporal changes often occur on local facial regions, such as eyes and lip corners, so that the backbone network can also learn to focus on these important regions under this supervision.

In traditional SSL methods, the representation capacity of the trained model via the pretext task is evaluated by training a linear classifier with the labeled data. However, the model performances of previous SSL based AU recognition methods are not satisfactory (Li et al., 2019a; Lu et al., 2020). On the other hand, training the self-supervised task together with the downstream task end-to-end is more efficient and can obtain better results (Zhai et al., 2019). Motivated by this, we incorporate the two auxiliary tasks with the supervised AU recognition task, as illustrated in Figure 2, and form the end-to-end trainable RTATL framework. During training, the auxiliary tasks are also performed on the labeled data. The overall loss function is formulated as:

where $\mathcal{L}_{Sup}$ is the binary cross entropy loss function which applied to the labeled data and $\lambda_{f}$ is the hyper parameter of loss weight for OFE. It is notable that for the image samples with cropped RoIs, we calculate $\mathcal{L}_{Sup}$ only on AUs in the intact regions. During inference phase, the intact facial image is input to the model. Only the backbone network is utilized and the modules for two auxiliary tasks are removed.

We evaluate our methods on two popular benchmark databases, i.e., BP4D (Zhang et al., 2013) and DISFA (Mavadati et al., 2013). BP4D consists of facial expression videos collected from 41 participants, including totally 146,847 frames with valid AU labels. DISFA is composed of 27 videos. There are totally 130,815 frames and each frame is labeled with AU intensity from 0 to 5. For DISFA database, frames with intensity greater than 1 are treated as positive and the rest are negative. For all experiments, data from these two databases are treated as labeled data and we follow the protocol of subject independent 3-fold cross validation. For unlabeled data, we randomly sample 100,000 images from Emotionet (Fabian Benitez-Quiroz et al., 2016) which is a facial expression database containing around one million facial images from the Internet.

For each frame, we first perform face detection and alignment based on similarity transformation and obtain a $200\times 200$ RGB face image. Ordinary data augmentations such as randomly cropping and horizontally flipping are conducted to enhance the diversity. All images are resized to $192\times 192$ as the input of the network. During testing phase, only center cropping is employed.

The transformer in our model is composed of one encoder and one decoder. For all multi-head self-attention layers, the number of heads is set to 8 empirically. $d$ is set to be 128. Thus, the dimension of each AU indicator is also 128.

For RoII, we follow (Li et al., 2017) to locate the AU centers and crop a region of $48\times 48$ around each center point. Facial landmarks are detected with Dlib (King, 2009). $D$ and $C$ are composed of 5 layers of convolution while $G$ consists of 5 layers of transposed convolution. $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{f}$ are empirically set as 0.1, 0.1 and 0.2, respectively. The OFE is addressed with 2 layers of transposed convolution. ResNet-18 in the backbone takes pre-trained ImageNet model weight as initialization and the rest parameters are initialized randomly. All models are trained via Adam (Kingma and Ba, 2014) with the learning rate 0.0003. All experiments are implemented with PyTorch (Paszke et al., 2019) and conducted on one Nvidia Tesla V100.

In the training phase, there are totally 54.62 million parameters and the computational cost is 10.75 GFLOPs. However, during inference, there are only 19.12 million parameters in the network and the computational cost is reduced to 5.57 GFLOPs since the modules working for auxiliary tasks are only effective during training, which validates the computation efficiency in model deploying.

Instead of GCN, we propose to utilize a transformer to embed AU relationship adaptively. To demonstrate the superiority of the transformer, we first conduct experiments based on the backbone network with GCN and with a transformer. Here only labeled BP4D samples are used in a fully supervised manner. As shown in Table 1, compared to GCN, the transformer improves the average F1-score by 1.4%, which proves its advantage of adaptive AU relation learning. The backbone model with a transformer is regarded as the baseline to compare with the subsequent SSL auxiliary tasks.

Then we conduct experiments to compare the proposed self-supervised auxiliary tasks with several other successful SSL tasks in computer vision, including rotation classification (Gidaris et al., 2018), exemplar (Dosovitskiy et al., 2014) and jigsaw puzzles (Noroozi and Favaro, 2016). Under the same experimental setting, these ones are also served as the auxiliary tasks and jointly trained with AU recognition. Appropriate branches are added to the backbone to solve the corresponding tasks. All experiments are conducted with the same backbone except for jigsaw puzzle as it requires to divide the input image into 9 patches which is not suitable for RoI feature learning. For this task, only ResNet-18 is employed. The performances are presented in Table 1. Since the jigsaw task divides the input image into patches which neglects the spatial relations of the joint parts, the model performance is significantly worse than others. After incorporating other SSL tasks, AU recognition results are improved to different extents. The rotation and exemplar improve the baseline by 0.4% and 0.7% due to better global feature representation.

The proposed OFE task obtains an improvement of 1.3% in average compared to the baseline and the performance on AU24 is dramatically boosted, which demonstrate the effectiveness of the motion features for certain AUs. The result is also better than examplar as the motion information provides more valuable cues for AU recognition. By enhancing the AU relation learning via RoII, the model achieves the best result among all single SSL tasks, which is 64.1% in average and improves the baseline by 1.6%. The performance validates the necessity of considering the unique properties of AUs when designing self-supervised tasks. Moreover, when we integrate the two tasks together and form the RTATL framework, the average F1-score is further improved by 0.5% compared to RoII alone, as OFE and RoII aim to obtain discriminative AU features from complementary perspectives.

Furthermore, to evaluate the model performance when only limited labeled data is accessible, we conduct experiments with a gradually increasing number of labeled training samples. Following the subject independent 3-fold cross validation protocol, instead of using all 27 subjects for training in each fold, we randomly select 3, 9, 15 and 21 subjects and discard the others in the original training set. The unlabeled and validation data are the same with previous experiments. Under the same setting, the performance comparison of three methods are shown in Figure 4. Except for the experiment with only 3 labeled subjects, RTATL outperforms the rotation and exemplar tasks consistently. With around half the original labeled training samples, the RTATL achieves comparable result with the backbone network trained with all training samples, which demonstrates the data efficiency of our method.

We compare RTATL with previous state-of-the-art methods on BP4D and DISFA, including the supervised methods such as JAA (Shao et al., 2018), SRERL (Li et al., 2019b), LP-Net (Niu et al., 2019b), UGN (Song et al., 2021) and the self-supervised methods such as TCAE (Li et al., 2019a) and TC-Net (Lu et al., 2020).

Experimental results on BP4D are presented in Table 2. Obviously there is a performance gap between previous SSL and our method. It is because the two-stage training framework does not take advantage of the labeled AU data to train the backbone network. Meanwhile, previous SSL methods mainly focus on global feature learning while our SSL tasks are more AU specific, which incorporates regional and temporal feature learning, and AU relation encoding. Compared to the fully supervised models, RTATL fully explores the unlabeled data to capture better feature representation via the auxiliary SSL tasks, resulting in an improved performance. In comparison with other auxiliary task, such as face alignment in JAA, ours does not require additional annotations but gets a better result. It is notable that compared to the semi-supervised method MLCR (Niu et al., 2019a) which also leverages the unlabeled data, RTATL still outperforms it by 4.8% and sets a new state-of-the-art performance.

Table 3 presents the experimental results on DISFA. Compared to the previous state-of-the-art supervised method UGN, RTATL achieves an improvement of 3.1% and obtains the overall performance of 63.1%. In comparison with BP4D, DISFA contains less training samples with more severe data imbalance problem. Most of the previous methods suffer from significant performance degradation. However, in our method, by incorporating RoII and OFE in a unified framework, both local and global features in the backbone are enhanced. Therefore, RTATL maintains competitive performances on majority of the AUs consistently and achieves the new state-of-the-art performance on DISFA.

The estimation of optical flow for 6 subjects in BP4D is illustrated in Figure 5. We use grayscale to visualize the optical flow in horizontal and vertical directions respectively, where $I_{x}$ is shown in left and $I_{y}$ is in right. TV-L1 optical flow extracted between two frames is treated as ground truth for the task of single image based OFE. As can be seen, optical flow describes the motion information of important facial regions where an AU is activated. The predictions are shown in the third row. Most of the predictions are similar to the ground truth which indicates the learned temporal features encoded in the global features are effective. Thus, through the OFE task, the backbone network can extract temporal features from single image to further enhance the discrimination of global features for AU recognition.

The visualization for RoI inpainting is shown in Figure 6. In the first row, symmetrical RoIs of a random AU are removed and replaced with white color. The original images are shown below. The third row presents the results of RoI inpainting. From the aspect of facial semantic, most of the regions around eyes and lips are recovered correctly which verifies the effectiveness of RoI features embedded with adaptive AU relation via the transformer. By the auxiliary task of RoI inpainting, the representation and discrimination of regional features are enhanced. However, as the dimension of regional feature used for recovery is only 128, and the network module for patch generation utilized is not sophisticated, some generated regions may seem poor in appearance and suffer from misalignment or mode collapse, e.g., some left and right eyes are not distinguishable. Nevertheless, the proposed RoII task aims to better explore the AU relation adaptively and embed it to the regional feature representation rather than pursue great facial appearance details in the color space. In other words, the muscle movements of the cropped region derived from other intact regions via AU relation embedding matters more in this task. From this perspective, the RoII works well to meet our demand.

As elaborated in Section 3.2, AU indicators in the transformer decoder should contain AU relation knowledge implicitly after training. In order to visualize the learned AU relation, we calculate the cosine similarity between each AU indicator and show the similarity matrix with heatmap in Figure 7. The relation of two AUs is closer if the blue color of the corresponding region is darker. Most of the relationships presented in the Figure 7 are consistent with the facial action coding expertise (Peng and Wang, 2018), e.g., AU10 (upper lip raiser) and AU17 (chin raiser) are related in BP4D, and AU6 (cheek raiser and lid compressor), AU12 and AU26 (jaw drop) are closely related in DISFA. Moreover, the similarity matrix of BP4D also reveals some unique relations in the database, e.g., there exists close relationship between AU6, AU7 (lid tightener), AU10 and AU12. It is because AU6 and AU7 are very similar, and the annotators of BP4D tend to annotate AU10 as positive when the subject is laughing which is reasonable. Therefore, the transformer in our model successfully captures the AU relation adaptively in a data-driven manner.