Learning Contrastive Feature Representations for Facial Action Unit Detection

One common technique to address the issue of false data is label smoothing, where a small amount of uncertainty is incorporated into the training process by replacing one-hot labels with smoothed probability distributions. However, it is challenging to directly apply label smoothing to contrastive learning as the computation of contrastive loss does not involve explicit labels. To overcome this challenge, we turn to the memory effect of neural networks. Previous research has shown that neural networks tend to first fit clean samples and then memorize the noise in incorrectly labeled samples [32]. This memory effect is independent of the choice of neural network architecture. The positive samples that are easier fit are those that are closer to the anchor point and have higher similarity. Inspired by this observation, for a given sample $x$ and M positive samples ${x}_{i=1}^{M}$ with noisy labels belonging to the same class, we calculate the similarity scores between the representation of sample $x$ and the representations of all M positive samples. We then select the positive feature representation with the highest similarity to sample $x$:

By selecting the positive samples with the highest similarity, we prioritize clean and reliable positive samples while minimizing the influence of noisy positive samples during the training process.

In addition to label noisy learning, we incorporate self-supervised learning techniques into the training process. By designing these auxiliary tasks carefully, we can encourage the model to learn more robust and invariant representations, which can help mitigate the effects of label noise and improve generalization performance. Specifically, our second method to obtain positive feature involves performing semantically invariant image augmentation on the anchor point $x$:

where $\mathcal{T}(\cdot)$ is a stochastic data augmentation module that applies random transformations to a given data example $x$, generating two correlated views referred to as $x_{s1}^{+}$ and $x_{s2}^{+}$, which we consider as a positive pair. Follow SimCLR [10], we employ a sequential application of three simple augmentations: random cropping and flipping followed by resizing back to original size, random color distortions, and random Gaussian blur.

Our third method to obtain positive examples involving a mixture of positive feature representations, which can be seen as a mixup of positive samples to obtain the most representative positive instances:

The primary distinction from data augmentation approaches exemplified by SimCLR lies in the nature of positive examples. In SimCLR, positive examples are augmented versions of the anchor point, implying that two views of the same sample are defined as positives, with each sample representing a unique class, leading to instance-level contrast. Under such circumstances, positive examples do not fully encapsulate the general class information. In contrast, the proposed method defines positive examples as the mean of the same-class samples that are identical to the anchor point, essentially representing the most representative features fo positive samples, which can be interpreted as class prototypes. Compared to instance-level contrast, the feature representations of class prototypes offer more robust information for comparison, thereby better capturing class information. The positive feature representation $f(x_{m}^{+})$ obtained from Eq. (10) represents the class centroid of M positive examples $\{x\}_{i=1}^{M}$. In other words, it synthesizes a virtual representative positive sample.

The aforementioned three methods of obtaining positive examples combine the use of label noisy learning for supervised signals and data augmentation for self-supervised signals. These three types of positive examples are randomly mixed with probabilities $p_{1}$, $p_{2}$, and $p_{3}$ respectively, where $p_{1}+p_{2}+p_{3}=1$. Our approach aims to improve the quality of supervised signals utilized in the training process. By integrating these techniques, we effectively mitigate the impact of noisy and false labels, increase the resilience of models to errors and variations in labeled datasets, and ultimately enhance the generalization ability of models to unseen data.

By addressing the above three challenges, our approach improves the encoding capability of the shared encode, ultimately leading to improved generalization performance in AU detection. The pseudocode for the algorithm is provided in Algorithm. 1.

In our experiments, we extensively utilize three well-established datasets in constrained scenarios for AU detection, namely BP4D [34], DISFA [35], GFT [36], and one well-established dataset in unconstrained scenarios, namely Aff-Wild2 [37]. Our primary focus is on assessing frame-level AU detection, as opposed to datasets that solely provide video-level annotations.

BP4D [34]: This dataset comprises 41 participants, including 23 females and 18 males. Each individual is featured in 8 sessions, recorded using both 2D and 3D videos. The dataset encompasses approximately 140,000 frames, each annotated with AU labels indicating occurrence or non-occurrence. Additionally, each frame is meticulously annotated with 68 facial landmarks. For BP4D dataset, we employ a three-fold cross-validation methodology for rigorous evaluation.

DISFA [35]: This dataset is composed of 27 videos, recorded from 15 men and 12 women. Each video comprises 4,845 frames, resulting in a total of 130,788 images. Frames in this dataset are annotated with 68 facial landmarks and AU intensities ranging from 0 to 5. Frames with intensities equal to or greater than 2 are considered positive, while others are classified as negative. We also employ the same three-fold cross-validation methodology as BP4D dataset for rigorous evaluation of DISFA dataset.

GFT [36]: This dataset comprises 96 participants, organized into 32 groups of three individuals, engaged in unscripted conversations. Each participant is recorded via video, with the majority of frames capturing moderate out-of-plane poses. Annotations for each frame include 10 AUs, alongside 68 facial landmarks. In accordance with the standard training and testing partitions, our training set consists of 78 participants, amounting to approximately 108,000 frames, while the testing set includes 18 participants, totaling around 24,600 frames.

Aff-Wild2 [37]: This dataset is an extensive in-the-wild facial expression dataset, with videos sourced from YouTube, encompassing a broad range of ages, ethnicities, professions, emotions, poses, lighting conditions, and occlusions. This dataset comprises 305 videos, totaling approximately 1,390,000 frames for the training set, and 105 videos, with around 440,000 frames for the validation set. Each frame is annotated with 12 AUs and 68 facial landmark annotations. Following the methodology of Zhang et al. [38], we train our AUNCE on the training set and evaluate it on the validation set.

The occurrence rates of AUs in the training sets of these datasets are presented in Table 1. It is evident from the table that the class imbalance issue of each type of AU is prevalent across all datasets, especially on the DISFA, GFT and Aff-Wild2 datasets.

Initially, the 68 facial landmarks undergo a conversion into 49 facial internal landmarks, excluding those associated with the facial contour. Subsequently, the images undergo augmentation through similarity transformation, encompassing in-plane rotation, uniform scaling, and translation, utilizing the 49 facial landmarks. The resulting images are standardized to a resolution of 256$\times$256$\times$3 pixels and then randomly cropped into 224$\times$224$\times$3 pixels, with the addition of a random horizontal flip.

For the training process, the encoder is conducted on one NVIDIA GeForce RTX 3090 GPU with 24GB for one day and implemented by PyTorch using an AdamW optimizer with $\beta_{1}$= 0.9, $\beta_{2}$= 0.999 and weight decay of $10^{-6}$. We set a learning rate of $10^{-5}$ and 60 epochs. Due to our discriminative contrastive learning framework that sample pairs are selected in a same batch, so the batch size has a strong influence on the experimental results. In terms of the hyperparameters in AUNCE, $n_{au}$ is set as 12 for BP4D and 8 for DISFA respectively, and the temperature coefficient $\tau$ is 0.5. The analysis for $\beta$ and $p_{1}$, $p_{2}$, $p_{3}$ will introduce in Section. 4.4.2.

In assessing the acquired representations, we adhere to the established linear evaluation protocol, as commonly employed in previous studies [12, 10]. This involves training a linear classifier atop the fixed base network, and the highest test accuracy across our models serves as a surrogate measure for the quality of the learned representations.

Aligned with current state-of-the-art methodologies, we utilized two prevalent frame-based metrics for AU detection: F1-score($\%$) and Accuracy($\%$). F1-score represents the harmonic mean of precision and recall, widely employed in the context of AU detection. For each approach, we calculate average metrics across all AUs.

Since we use AU labels to assist the process of selecting positive sample pairs, we introduce five groups of supervised AU detection frameworks to compare with our method. The quantitative results of state-of-the-art methods from the past seven years for BP4D dataset are shown in Table 2. Compared to the latest state-of-the-art results, our method AUNCE demonstrates significantly superior performance.

Methods in the first group are based on feature extraction of regional area around AUs, including EAC-Net [4], ROI [3] and ARL [5]. Our approach achieves a notable improvement, with an increase in F1-score ranging from 5.0% to 10.2% over these methods. This advancement is primarily due to early feature extraction methods neglecting critical correlation information among AUs and failing to specifically address the class imbalance issue for each AU. Furthermore, the precision of the encoders utilized by these methods is generally lower.

Methods in the second group are based on a multi-task learning strategy, including J$\hat{\rm A}$A-Net [7], GeoConv [40], MAL [39] and GLEE-Net [8]. Our method surpasses MAL by 3.9%, primarily due to the ability of AUNCE to help the encoder address the class imbalance issue among different AUs, and the facial expression recognition component in MAL introduces some negative transfer effects on AU detection. The 3.7% improvement over J$\hat{\rm A}$A-Net can be attributed to the influence of the facial landmark detection task within the multi-task learning framework on AU detection, which is also affected by the weighting of different losses. Additionally, the modest 2.5% and 0.4% improvement over GeoConv and GLEE-Net is due to the relatively limited impact of integrating 3D information on enhancing the performance of 2D AU detection.

Methods in the third group mainly utilize the semantic prior knowledge, including MMA-Net [41] and SEV-Net [42]. Our approach demonstrates an average improvement of 2.7% and 2.2%, respectively. This phenomenon can be attributed to the inherent limitations of these methods in effectively capturing and representing the distinct information encoded within AUs using simple semantic descriptions.

Methods in the fourth group focus on capturing correlational information among AUs, including MCM [43], AAR [45], ISTR [46], HMP-PS [44] and MEGraph [33]. Our approach exhibits a performance advantage of approximately 0.6%-4.7% in F1-score. This underscores the efficacy of our supervised discriminative contrastive learning paradigm in effectively capturing differential information between AUs from each sample pair instead of pixel-level information of the entire face associated with AU. It is noteworthy that the encoder used in our method is only the same as the network MEGraph used for the first stage of training, with reduced parameters by nearly 7M. This highlights learning only the difference information among features is more efficient than learning the whole pixel-level information.

Methods in the last group are based on contrastive learning, including SIMCLR [10], MoCo [17], CLP [13], BTFM [47], SupHCL [28], KSRL [24] and CLEF [48]. When compared to these methods, our approach demonstrates a performance advantage of approximately 0.2% to 16.3% in F1-score. This improvement is attributed to the efficacy of our proposed negative sample re-weighting strategy and positive sample sampling strategy, specifically tailored for the AU detection task using static images. In addition, compared to contrastive learning methods based on weighted cross-entropy loss, our approach significantly reduces the extraction of pixel-level redundant information.

Comparable outcomes are evident in the DISFA dataset, as outlined in Table 3, where AUNCE attains the highest overall performance in AU detection. Furthermore, a comparison between Table 2 and Table 3 reveals that recent methods such as SEV-Net, MEGraph, and BTFM demonstrate strong performance on the BP4D dataset but achieve only mediocre results on the DISFA dataset. This disparity suggests their limited generalizability. The more pronounced class imbalance issue in DISFA, as shown in Table 1, likely contributes to this performance gap. In contrast, AUNCE consistently delivers robust performance on both BP4D and DISFA datasets and achieves more stable results across various AUs due to our proposed negative sample re-weighting strategy.

Table 4 presents the F1-score results on the GFT dataset, where our AUNCE method demonstrates significantly superior performance compared to previous approaches. Unlike BP4D and DISFA, which primarily consist of near-frontal facial images, GFT includes many images captured under out-of-plane poses. In this challenging scenario, other methods such as EAC-Net, ARL and CLP so on struggle to perform well. In contrast, AUNCE achieves a commendable average F1-score of 63.0%.

We have validated the performance of our AUNCE method in constrained scenarios. To further assess its effectiveness in unconstrained scenarios, we compared it with other approaches on the challenging Aff-Wild2 dataset, as shown in Table 5. Our AUNCE method demonstrates a higher average F1-score than previous methods. Notably, while EAC-Net and PASN leverage external training data, our method achieves superior overall performance using only the Aff-Wild2 dataset. By comparing the results of ARL, J$\hat{\rm A}$A-Net, AAR to our AUNCE method across Table 2 to Table 5, it is evident that the performance margins between our AUNCE method and other methods on GFT and Aff-Wild2 are larger than those on BP4D and DISFA. This indicates that AUNCE is more effective at handling challenging cases in AU detection.

Due to the primary innovation of this paper being the introduction of a novel contrastive loss, AUNCE, which is compatible with various AU detection encoders, we conducted a comparative analysis between the proposed AUNCE loss and the traditional weighted cross-entropy loss (WCE) across multiple AU detection encoders, as detailed in Table 6. The findings reveal that AUNCE demonstrates superior performance in AU detection. Notably, the MEGraph encoder achieved the best results, with the F1-score on the BP4D dataset improving from 63.6% to 66.1%. This underscores the validity of capturing unique features of different samples over focusing solely on pixel-level information of the entire face associated with AU.

Furthermore, as shown in Table 7, when the number of encoder parameters is the same, our proposed AUNCE loss increases the frame rate and reduces the training time of one epoch by nearly half compared to WCE loss. This indicates that our approach of capturing only differential information between AUs from each sample pair has higher efficiency for training encoder than methods that capture the entire facial information.

We further evaluate the effectiveness of weight coeffcient $\omega_{i}$, negative sample re-weighting strategy and positive sample samplying strategy. Quantitative results of our method with the above component combinations are summarized in Table 8. The baseline (BL) is rooted in SIMCLR [10], but positive sample pairs are changed to images with the highest similarity to original images. Other components are the weight coefficient ($w_{i}$) for both WCE and AUNCE, the positive sample sampling strategy (PS), and the negative sample re-weighting strategy (NS).

Weight coefficient $w_{i}$: The weights $w_{i}$ have been pointed out as a fixed value in Section. 3.1, which achieve an increase of 0.9% in F1-score and 0.2% in Accuracy. For an AU, its activation indicator is a binary variable, and the corresponding information content can be measured using entropy: $-\log(p_{i})$. Therefore, AUs with lower activation probabilities carry more information and should be assigned higher weights. This forms the theoretical basis for our weighting strategy.

Positive sample sampling strategy: As outlined in Table 8, the positive sample sampling strategy contributes to a 2.8% increase in F1-score and a 0.5% improvement in Accuracy compared to model B. This validates that selecting different types of positive sample pairs with appropriate probabilities can mitigate issues arising from AU labels tainted by noise and errors. The strategy achieves this by incorporating three types of positive sample pairs: the self-supervised signal pair, the highest similarity pair, and the mixed positive features pair. These pairs provide the models with diverse information from various views, thereby enhancing the generalization ability of the discriminative contrastive learning framework.

Negative sample re-weighting strategy: The negative sample re-weighting strategy to minority AUs results in a 3.1% increase in F1-score and a 0.6% increase in Accuracy compared to model B. This illustrates that the strategy can change the backpropagation gradient of minority and majority class samples, facilitating faster label category gradient updates with a smaller number of samples. Additionally, model E achieves an average increase of 2.3% in F1-score compared to model C, further supporting the efficacy of the re-weighting strategy.

We further show the visualization using T-SNE dimensionality reduction of the learned feature representations optimized by AUNCE and WCE to qualitatively assess the efficacy of our proposed discriminative contrastive learning paradigm, as illustrated in Fig. 4. For each dataset, a random sample of 200 images is utilized for illustration purposes. A comparison between Fig. 4(a), Fig. 4(f) against Fig. 4(g), Fig. 4(l) reveals that the learned representations optimized by AUNCE exhibit greater separability than those optimized by WCE. Additionally, Fig. 4(b)-(f) and Fig. 4(g)-(l) qualitatively showcase the effectiveness of each component in our ablation study.

Based on all ablation studies, it demonstrates that our proposed discriminative contrastive learning paradigm enhances the encoding capability of the shared encoder, improves classification accuracy by addressing the class imbalance issue of each AU type, and bolsters model robustness against AU labels affected by noise and errors. This, in turn, leads to enhanced generalization performance in AU detection. The introduction of each component proves beneficial for performance improvement, with NS emerging as the most effective component. Moreover, the results on each AU are also shown in Table 8. It can be seen that the results on basically every AU have improved or approached after we add $w_{i}$, PS and NS.

In our paper, $p_{1}$ represents the probabilities of images with the highest similarity to original images, $p_{2}$ represents the probabilities of self-supervised images enhanced by simple augmentations, $p_{3}$ represents the probabilities of a mixture of all positive samples, and $p_{4}$ represents the probabilities of images with the lowest similarity to original images. Given that $p_{1}$, $p_{2}$, $p_{3}$, and $p_{4}$ address the problem of AU labels tainted by noise and errors, we systematically vary these probabilities from 0 to 1 to investigate their impact on our framework. As $p_{1}$ and $p_{2}$ are deemed reliable positive samples, we set their values to be the same during the variation.

The experimental results are presented in Table 9. When $p_{1}$=0.15, $p_{2}$=0.15, $p_{3}$=0.7, and $p_{4}$=0, the experimental result is optimal. It is observed that images with the lowest similarity to original images contribute minimally to the training process, as these images essentially represent noise and errors. The conclusion is consistent with [55]. The mixture of all positive samples plays a pivotal role, as it encourages the feature representation of each instance to be closer to the class centroid, facilitating the capture of essential characteristics of the positive class. The indispensability of the remaining two positive samples lies in their ability to mitigate the impact of noisy and false samples, thereby reinforcing the robustness of the training process.

As is well understood, the hyperparameter $\beta$ plays a crucial role in governing the relative importance of minority and majority class samples during the parameter update process. It ensures that the model allocates more attention to minority class samples when necessary and balances the gradient magnitudes accordingly. To explore its effects, we systematically vary the hyperparameter $\beta$ for the minority class from 0.8 to 1.8, with $\beta$ for the majority class fixed at 0.4. Additionally, we vary $\beta$ for the majority class from 0.2 to 1.2, keeping $\beta$ for the minority class fixed at 1.2.

The experimental results are depicted in Fig. 5. The optimal experimental outcome is observed when $\beta$ for the minority class is set to 1.2, and $\beta$ for the majority class is fixed at 0.4. Hence, when $\beta_{\text{minority}}>\beta_{\text{majority}}$, the impact of the class imbalance issue can be effectively mitigated.