Learning Contrastive Feature Representations for Facial Action Unit Detection

One common technique to address the issue of false data is label smoothing, where a small amount of uncertainty is incorporated into the training process by replacing one-hot labels with smoothed probability distributions. However, it is challenging to directly apply label smoothing to contrastive learning as the computation of contrastive loss does not involve explicit labels. To overcome this challenge, we turn to the memory effect of neural networks. Previous research has shown that neural networks tend to first fit clean samples and then memorize the noise in incorrectly labeled samples. This memory effect is independent of the choice of neural network architecture. The positive samples that are easier fit are those that are closer to the anchor point and have higher similarity. Inspired by this observation, for a given sample $x$ and M positive samples ${x}_{i=1}^{M}$ with noisy labels belonging to the same class, we calculate the similarity scores between the representation of sample $x$ and the representations of all M positive samples. We then select the positive feature representation with the highest similarity to sample $x$:

By selecting the positive samples with the highest similarity, we prioritize clean and reliable positive samples while minimizing the influence of noisy positive samples during the training process.

In addition to label noisy learning, we incorporate self-supervised learning techniques into the training process. By designing these auxiliary tasks carefully, we can encourage the model to learn more robust and invariant representations, which can help mitigate the effects of label noise and improve generalization performance. Specifically, our second method to obtain positive feature involves performing semantically invariant image augmentation on the anchor point $x$:

where $\mathcal{T}(\cdot)$ is a stochastic data augmentation module that applies random transformations to a given data example $x$, generating two correlated views referred to as $x_{s1}^{+}$ and $x_{s2}^{+}$, which we consider as a positive pair. Follow SimCLR [13], we employ a sequential application of three simple augmentations: random cropping and flipping followed by resizing back to original size, random color distortions, and random Gaussian blur.

Our third method to obtain positive examples involving a mixture of positive feature representations, which can be seen as a mixup of positive samples to obtain the most representative positive instances:

The primary distinction from data augmentation approaches exemplified by SimCLR lies in the nature of positive examples. In SimCLR, positive examples are augmented versions of the anchor point, implying that two views of the same sample are defined as positives, with each sample representing a unique class, leading to instance-level contrast. Under such circumstances, positive examples do not fully encapsulate the general class information. In contrast, the proposed method defines positive examples as the mean of the same-class samples that are identical to the anchor point, essentially representing the most representative features fo positive samples, which can be interpreted as class prototypes. Compared to instance-level contrast, the feature representations of class prototypes offer more robust information for comparison, thereby better capturing class information. The positive feature representation $f(x_{m}^{+})$ obtained from Eq. (10) represents the class centroid of M positive examples $\{x\}_{i=1}^{M}$. In other words, it synthesizes a virtual representative positive sample.

The aforementioned three methods of obtaining positive examples combine the use of label noisy learning for supervised signals and data augmentation for self-supervised signals. These three types of positive examples are randomly mixed with probabilities $p_{1}$, $p_{2}$, and $p_{3}$ respectively, where $p_{1}+p_{2}+p_{3}=1$. Our approach aims to improve the quality of supervised signals utilized in the training process. By integrating these techniques, we effectively mitigate the impact of noisy and false labels, increase the resilience of models to errors and variations in labeled datasets, and ultimately enhance the generalization ability of models to unseen data.

By addressing the above three challenges, our approach improves the encoding capability of the shared encode, ultimately leading to improved generalization performance in AU detection. The pseudocode for the algorithm is provided in Algorithm. 1.

In our experiments, we extensively utilize four well-established datasets in constrained scenarios for AU detection, namely BP4D [27], DISFA [28], BP4D+ [29], GFT [30], and one well-established dataset in unconstrained scenarios, namely Aff-Wild2 [31]. Our primary focus is on assessing frame-level AU detection, as opposed to datasets that solely provide video-level annotations.

BP4D [27]: This dataset includes 41 participants (23 females, 18 males), each appearing in 8 sessions with 2D and 3D video recordings. It contains approximately 140,000 frames, each labeled for AU occurrence and annotated with 68 facial landmarks. A three-fold cross-validation is used for evaluation.

DISFA [28]: Comprising 27 videos from 15 men and 12 women, this dataset includes 130,788 frames annotated with 68 facial landmarks and AU intensities (0-5). Frames with intensities $\geq$ 2 are positive; others are negative. Three-fold cross-validation is applied for evaluation, similar to BP4D.

BP4D+ [29]: Featuring 82 females and 58 males across 10 sessions, this dataset expands upon BP4D in scale and diversity. Of the 10 sessions, 4 provide 197,875 AU-labeled frames annotated with 49 facial landmarks. Following [6], we train on BP4D and test on BP4D+.

GFT [30]: This dataset records 96 participants in 32 unscripted conversational groups, primarily capturing moderate out-of-plane poses. Annotations include 10 AUs and 68 facial landmarks. Training involves 78 participants (108,000 frames), while testing uses 18 participants (24,600 frames) based on standard splits.

Aff-Wild2 [31]: This in-the-wild dataset includes 305 training videos (1.39 million frames) and 105 validation videos (440,000 frames), sourced from YouTube with diverse demographics and conditions. Each frame is annotated with 12 AUs and 68 facial landmarks. We follow [32] by training on the training set and evaluating on the validation set.

The occurrence rates of AUs in the training sets of these datasets are presented in Table 1. It is evident from the table that the class imbalance issue of each type of AU is prevalent across all datasets, especially on the DISFA, GFT and Aff-Wild2 datasets.

Following the preprocessing method used in MEGraph [26], the 68 facial landmarks undergo a conversion into 49 facial internal landmarks, excluding those associated with the facial contour. Subsequently, the images undergo augmentation through similarity transformation, encompassing in-plane rotation, uniform scaling, and translation, utilizing the 49 facial landmarks. The resulting images are standardized to a resolution of 256$\times$256$\times$3 pixels and then randomly cropped into 224$\times$224$\times$3 pixels, with the addition of a random horizontal flip.

For the pre-training process, the encoder is conducted on one NVIDIA GeForce RTX 3090 GPU with 24GB for one day and implemented by PyTorch using an AdamW optimizer with $\beta_{1}$= 0.9, $\beta_{2}$= 0.999 and weight decay of $10^{-6}$. We set a learning rate of 10^-5 and 60 epochs. In our discriminative contrastive learning framework, where sample pairs are selected within the same batch, the batch size significantly impacts the experimental results. For consistency, we have set the batch size to 32 in our experiments. In terms of the hyperparameters in AUNCE, $n_{au}$ is set as 12 for BP4D, BP4D+, Aff-Wild2, 10 for GFT, and 8 for DISFA respectively, and the temperature coefficient $\tau$ is 0.5. The analysis for $\beta$ and $p_{1}$, $p_{2}$, $p_{3}$ will introduce in Section. 4.4.1.

For the linear evaluation protocol [9, 13], we extract $n_{au}$ 2048-dimensional feature vectors from the model pre-trained using the AUNCE loss. A linear classifier is then trained on these features using AdamW optimizer with the same settings as the pre-training phase, a batch size of 256 on a single GPU, a learning rate of 10^-3, and 100 epochs. The results are obtained through training with the weighted cross-entropy loss (WCE), and the highest test accuracy across our models serves as a surrogate measure for the quality of the learned representations.

Aligned with current state-of-the-art methodologies, we evaluate our AU detection method using three frame-based metrics: F1-macro score, F1-micro score, and Accuracy. The F1-macro score is obtained by calculating the F1 score for each AU class individually and then averaging these values, which is a standard metric employed by many existing methods. Inspired by the work of Hinduja et al. [33], we also incorporate the F1-micro score, which is computed by aggregating the true positives, false positives, and false negatives across all classes before calculating the overall F1-score. This metric is particularly useful for evaluating performance in imbalanced datasets. Additionally, we include Accuracy, which is defined as the ratio of correctly classified frames to the total number of frames, to provide a more comprehensive assessment of model performance. For simplicity, we will refer to the F1-macro score as F1-score in the remainder of the text.

Since we use AU labels to assist the process of selecting positive sample pairs, we introduce five groups of supervised AU detection frameworks to compare with our method. The quantitative results of state-of-the-art methods from the past seven years for BP4D dataset are shown in Table 2. Compared to the latest state-of-the-art results, our method AUNCE demonstrates significantly superior performance.

Methods in the first group are based on feature extraction of regional area around AUs, including EAC-Net [4], ROI [3], ARL [5] and AC²D [34]. Our approach achieves a notable improvement, with an increase in F1-score ranging from 1.5% to 10.2% over these methods. This advancement is primarily due to the feature extraction methods of these frameworks neglecting critical correlation information among AUs and failing to specifically address the class imbalance issue for each AU. Furthermore, the precision of the encoders utilized by these methods is generally lower.

Methods in the second group are based on a multi-task learning strategy, including MAL [35], J$\hat{\rm A}$A-Net [6] and GeoConv [36]. Our method surpasses MAL by 3.9%, primarily due to the ability of AUNCE to help the encoder address the class imbalance issue among different AUs, and the facial expression recognition component in MAL introduces some negative transfer effects on AU detection. The 3.7% improvement over J$\hat{\rm A}$A-Net can be attributed to the influence of the facial landmark detection task within the multi-task learning framework on AU detection, which is also affected by the weighting of different losses. Additionally, the modest 2.5% improvement over GeoConv is due to the relatively limited impact of integrating 3D information on enhancing the performance of 2D AU detection.

Methods in the third group mainly utilize the semantic prior knowledge, including MMA-Net [37] and SEV-Net [38]. Our approach demonstrates an average improvement of 2.7% and 2.2%, respectively. This phenomenon can be attributed to the inherent limitations of these methods in effectively capturing and representing the distinct information encoded within AUs using simple semantic descriptions.

Methods in the fourth group focus on capturing correlational information among AUs, including AAR [32], SACL [39] and MEGraph [26]. Our approach exhibits a performance advantage of approximately 0.5%-2.3% in F1-score. This underscores the efficacy of our supervised discriminative contrastive learning paradigm in effectively capturing differential information between AUs from each sample pair instead of pixel-level information of the entire face associated with AU. It is noteworthy that the encoder used in our method is only the same as the network MEGraph used for the first stage of training, with reduced parameters by nearly 7M. This highlights learning only the difference information among features is more efficient than learning the whole pixel-level information.

Methods in the last group are based on contrastive learning, including SIMCLR [13], MoCo [14], EmoCo [23], CLP [11], KSRL [21] and CLEF [19]. When compared to these methods, our approach demonstrates a performance advantage of approximately 0.2% to 16.3% in F1-score. This improvement is attributed to the efficacy of our proposed negative sample re-weighting strategy and positive sample sampling strategy, specifically tailored for the AU detection task using static images. In addition, compared to contrastive learning methods based on weighted cross-entropy loss, our approach significantly reduces the extraction of pixel-level redundant information.

Comparable outcomes are evident in the DISFA dataset, as outlined in Table 3, where AUNCE attains the highest overall performance in AU detection. Furthermore, a comparison between Table 2 and Table 3 reveals that recent methods such as EmoCo, CLP, and MEGraph demonstrate strong performance on the BP4D dataset but achieve only mediocre results on the DISFA dataset. This disparity suggests their limited generalizability. The more pronounced class imbalance issue in DISFA, as shown in Table 1, likely contributes to this performance gap. In contrast, AUNCE consistently delivers robust performance on both BP4D and DISFA datasets and achieves more stable results across various AUs due to our proposed negative sample re-weighting strategy.

To assess the model’s performance on a larger and more diverse test set, we train AUNCE on the full BP4D and evaluate it on the entire BP4D+ using a cross-dataset testing protocol. The comparative results are presented in Table 4. Notably, AUNCE achieves superior performance in terms of the average F1-score, outperforming previous methods. This indicates that AUNCE maintains robust performance even when the testing data significantly increases in both scale and diversity.

The F1-score results on GFT are summarized in Table 5, where the proposed AUNCE method demonstrates a clear advantage over previous approaches. Unlike BP4D, BP4D+, and DISFA, which predominantly consist of near-frontal facial images, GFT features a substantial number of images captured with out-of-plane poses, posing greater challenges for accurate AU detection. While methods such as ARL, CLP and AC²D encounter significant performance limitations under these conditions, AUNCE achieves an impressive average F1-score of 63.0%, highlighting its robustness in handling complex pose variations.

We have validated the performance of our AUNCE method in constrained scenarios. To further assess its effectiveness in unconstrained scenarios, we compared it with other approaches on the challenging Aff-Wild2 dataset, as shown in Table 6. Our AUNCE method demonstrates a higher average F1-score than previous methods. Notably, while EAC-Net leverage external training data, our method achieves superior overall performance using only the Aff-Wild2 dataset. By comparing the results of ARL, J$\hat{\rm A}$A-Net, AAR and AC²D to our AUNCE method across Table 2 to Table 6, it is evident that the performance margins between our AUNCE method and other methods on GFT and Aff-Wild2 are larger than those on BP4D, BP4D+ and DISFA. This indicates that AUNCE is more effective at handling challenging cases in AU detection.

To evaluate the generalization and cross-dataset reliability of our framework, we compare AUNCE with two baselines: one using the same encoder with weighted cross-entropy loss (WCE) and another, AC²D [34]. Results are presented in Table 7. Since the 10 AUs in the GFT dataset overlap with the 12 AUs in BP4D, we perform cross-dataset evaluations between BP4D and GFT. For BP4D-to-GFT, we use the three BP4D models trained via 3-fold cross-validation and report the average results. For GFT-to-BP4D, the trained GFT model is tested on the three BP4D test sets, and average results are reported.

As shown in Tables 2 and 5, the performance of AUNCE is notably lower in this cross-dataset evaluation. This performance drop can be attributed to the substantial domain gap between BP4D and GFT. Additionally, our findings show that AUNCE demonstrates superior performance compared to the other two approaches that rely on WCE loss in both BP4D → GFT and GFT → BP4D tasks. This result further underscores the robustness and generalization capability of our proposed discriminative contrastive learning framework.

The primary innovation of this paper is the introduction of AUNCE, a novel contrastive loss designed to be compatible with various AU detection encoders. To assess its effectiveness, we compare the proposed AUNCE loss with traditional weighted cross-entropy loss (WCE) and the recently introduced weighted focal loss (WFL) [7] for AU detection during the pretraining phase, across multiple AU detection encoders. The results, presented in Table 8, demonstrate that AUNCE consistently outperforms the other losses in AU detection. Notably, the MEGraph encoder achieves the highest performance, with scores of 66.1% and 66.4% on BP4D and DISFA datasets, respectively, when trained using AUNCE. This highlights the advantage of emphasizing the capture of distinct features from individual samples, rather than solely relying on pixel-level information of the entire face in relation to AU detection.

We compare the computational complexity and efficiency of the three losses. WCE and WFL both have an asymptotic complexity of $O(NC)$, as both of them involve computing probabilities for $C$ classes and summing over all $N$ samples, and WFL introducing negligible overhead due to the power and multiplication operations from the probability margin $\gamma$. In contrast, AUNCE has a lower complexity of $O(N+2)$, as it processes one anchor point, one positive sample, and $N$ negative samples, making it approximately twice as efficient as WCE and WFL when $C=2$. This suggests that AUNCE, by focusing on differential information between AUs, is more efficient for training the encoder compared to methods that capture entire facial information.

To further validate this advantage, we evaluate computational efficiency using two metrics: frame rate and training time per epoch. As shown in Table 9, under identical encoder parameters, AUNCE improves frame rate and reduces epoch training time by nearly half compared to WCE and WFL. These results confirm AUNCE’s superior efficiency in training, delivering comparable or better performance with significantly reduced computational overhead.

We further assess the effectiveness of all proposed components. Quantitative results of ablation study are summarized in Table 10. The baseline (BL) is based on SIMCLR [13], with positive sample pairs replaced by images most similar to the original ones. Other components include the weight coefficient ($w_{i}$) for both WCE and AUNCE, the positive sample sampling strategy (PS), and the negative sample re-weighting strategy (NS).

Weight coefficient $w_{i}$: The weights $w_{i}$ have been pointed out as a fixed value in Section. 3.1, which achieve an increase of 0.9% in F1-macro score, 0.2% in F1-micro score and 0.2% in Accuracy. For an AU, its activation indicator is a binary variable, and the corresponding information content can be measured using entropy: $-\log(p_{i})$. Therefore, AUs with lower activation probabilities carry more information and should be assigned higher weights. This forms the theoretical basis for our weighting strategy.

Positive sample sampling strategy: As shown in Table 10, the positive sample sampling strategy improves the F1-macro score by 2.8%, the F1-micro score by 0.4%, and Accuracy by 0.5% compared to model B. This demonstrates that selecting diverse positive sample pairs with appropriate probabilities helps mitigate issues related to noisy or false AU labels. The strategy utilizes three types of positive sample pairs: self-supervised signal pairs, highest similarity pairs, and mixed feature pairs. By incorporating these varied pairs, the model benefits from more informative inputs, enhancing its generalization ability and discriminative power. These improvements enable more effective AU detection, even in the presence of label noise and class imbalance, reinforcing the model’s robustness across various AU detection scenarios.

Negative sample re-weighting strategy: The negative sample re-weighting strategy, which prioritizes minority AUs during training, improves the F1-macro score by 3.1%, the F1-micro score by 0.5%, and accuracy by 0.6% compared to model B, demonstrating its ability to adjust backpropagation gradients for minority and majority classes, enabling faster updates with fewer samples. Additionally, model E achieves a 2.3% average increase in F1-macro score compared to model C, further validating the strategy’s effectiveness. As shown in Fig. 4, model E (top row) focuses more on facial regions relevant to rare AUs, enhancing the detection of these challenging cases and reinforcing its utility in addressing class imbalance.

To qualitatively assess the efficacy of our discriminative contrastive learning paradigm, we visualize the learned feature representations optimized by AUNCE and WCE using T-SNE dimensionality reduction, as shown in Fig.5. A random sample of 200 images from each dataset is used for illustration. Comparing Fig.5(a), Fig.5(f) with Fig.5(g), Fig.5(l) demonstrates that features learned with AUNCE show greater separability than those optimized by WCE. Additionally, Fig.5(b)-(f) and Fig. 5(g)-(l) highlight the contribution of each component in our ablation study.

The ablation studies confirm that the proposed discriminative contrastive learning paradigm enhances the encoder’s feature representation capability, addresses class imbalance for each AU type, and improves robustness to noisy and false labels, ultimately boosting generalization performance in AU detection. Each component contributes to performance gains, with NS proving most effective. Furthermore, as shown in Table 10, the addition of $w_{i}$, PS, and NS consistently improves or maintains performance across nearly all AUs.

In our paper, $p_{1}$ represents the probabilities of images with the highest similarity to original images, $p_{2}$ represents the probabilities of self-supervised images enhanced by simple augmentations, $p_{3}$ represents the probabilities of a mixture of all positive samples, and $p_{4}$ represents the probabilities of images with the lowest similarity to original images. Given that $p_{1}$, $p_{2}$, $p_{3}$, and $p_{4}$ address the problem of AU labels tainted by noise and errors, we systematically vary these probabilities from 0 to 1 to investigate their impact on our framework. As $p_{1}$ and $p_{2}$ are deemed reliable positive samples, we set their values to be the same during the variation.

The experimental results are presented in Table 11. When $p_{1}$=0.15, $p_{2}$=0.15, $p_{3}$=0.7, and $p_{4}$=0, the experimental result is optimal. It is observed that images with the lowest similarity to original images contribute minimally to the training process, as these images essentially represent noise and errors. The mixture of all positive samples plays a pivotal role, as it encourages the feature representation of each instance to be closer to the class centroid, facilitating the capture of essential characteristics of the positive class. The indispensability of the remaining two positive samples lies in their ability to mitigate the impact of noisy and false samples, thereby reinforcing the robustness of the training process.

However, hyperparameter tuning is a limitation of our paper. Although the proposed method yields promising results, we observed that the performance is highly sensitive to the values of $p_{1}$, $p_{2}$, and $p_{3}$, with their optimal settings varying across different datasets. Moreover, the parameter tuning process can only be guided by experience gained from extensive experimentation, which results in a tuning scheme that lacks flexibility.

As is well understood, the hyperparameter $\beta$ plays a crucial role in governing the relative importance of minority and majority class samples during the parameter update process. It ensures that the model allocates more attention to minority class samples when necessary and balances the gradient magnitudes accordingly. To explore its effects, we systematically vary the hyperparameter $\beta$ for the minority class from 0.8 to 1.8, with $\beta$ for the majority class fixed at 0.4. Additionally, we vary $\beta$ for the majority class from 0.2 to 1.2, keeping $\beta$ for the minority class fixed at 1.2.

The experimental results are depicted in Fig. 6. The optimal experimental outcome is observed when $\beta$ for the minority class is set to 1.2, and $\beta$ for the majority class is fixed at 0.4. Hence, when $\beta_{\text{minority}}>\beta_{\text{majority}}$, the impact of the class imbalance issue can be effectively mitigated.