LRDif: Learning Representations in Diffusion Models for Under-Display Camera Emotion Recognition

Generally, a FER system mainly consists of three stages, namely face detection, feature extraction, and expression recognition. In face detection stage, several face detectors like MTCNN [44] and Dlib [2]) are used to locate faces in complex scenes. The detected faces can be further aligned alternatively. For feature extraction, various methods are designed to capture facial geometry and appearance features caused by facial expressions. Wang et al. [wang2021light] leverages a spatial attention mechanism to focus on emotion-relevant image areas, enhancing accuracy under real-world conditions using a lightweight network embeddable in standard convolutional neural networks. MRAN[chen2023multi] enhances performance under uncontrolled conditions by integrating spatial attention on both global and local facial features, employing transformers to understand relationships within and between these features, and leveraging a sample relation transformer to focus on similarities among training samples, all optimized through a joint strategy. MPACNN [li2021learning] uses multiple paths to extract diverse features, which are then attentively fused for both basic and compound expression recognition. Concurrently, BS loss optimizes these features by maximizing inter-class and minimizing intra-class distances, ensuring their discriminative quality for high-accuracy recognition, with the BS loss functioning as the objective for MPACNN, enabling simultaneous learning of informative and discriminative features. FG-AGR[li2023fg] addresses challenges in uncontrolled environments by using an Adaptive Salient Region Induction (ASRI) to highlight key facial areas, a Local Fine-grained Feature Extraction (LFFE) via Visual Transformers for detailed feature extraction, and an Adaptive Graph Association Reasoning (AGAR) with Graph Convolutional Networks to learn combinations of these fine-grained features effectively. Zhang et al.[zhang2023transformer] propose using Transformer-based Multimodal Emotional Perception (T-MEP) framework to enhance dynamic expression recognition by employing specialized transformer-based encoders for audio, image, and text modalities, alongside a multimodal information fusion module that utilizes both self-attention and cross-attention mechanisms to robustly integrate and augment cross-modal representations for a comprehensive understanding of human emotions.

Rombach et al. [rombach2022high] enhances the efficiency and visual fidelity of diffusion models by training them in the latent space of pre-trained autoencoders, integrating cross-attention layers for versatile conditioning inputs, enabling high-resolution image synthesis with reduced computational demands and preserving detail. DiffusionDet[chen2023diffusiondet] is an innovative object detection framework that treats the detection process as a denoising diffusion from random to precise object boxes, learning to reverse the diffusion from ground truth during training and iteratively refining randomly generated boxes during inference, offering flexibility in the number of boxes and evaluation method. StyleSwin[zhang2022styleswin] is a high-resolution image synthesis framework using a pure transformer-based generative adversarial network, incorporating Swin transformer in a style-based architecture with double attention for larger receptive fields and absolute position knowledge, enhanced by a wavelet discriminator to overcome blocking artifacts and maintain spatial coherence.Yang et al. [yang2023paint] advances exemplar-guided image editing by employing self-supervised training to skillfully disentangle and rearrange elements from a source image and an exemplar, using an information bottleneck and strong augmentations to avoid copy-paste artifacts, and introducing an arbitrary shape mask with classifier-free guidance for enhanced control, all within a single forward pass of a diffusion model. Whang et al. [whang2022deblurring] utilizes conditional diffusion models to train a stochastic sampler that refines outputs from a deterministic predictor, offering a diverse set of perceptually superior reconstructions for a given blurred input, while also achieving efficiency in sampling and competitiveness in standard distortion metrics like PSNR.

Before introducing pretraining DTnetwork, we would like to introduce two networks in the first stage, including a compact fusion prior extraction network (FPEN) and a dynamic transfomer network (DTnetwork). The structure of FPEN is shown in Fig. 2 yellow box, which is mainly consist of linear layers to extract the compact emotion prior representation (EPR). After that, DTnetwork can use the extracted EPR to restore label of UDC images. The structure of the DTnetwork is shown in Fig. 2 red box which consist of dynamic gated feed-forward network DGNet and dynamic multi-head transposed attention DMNet, which can use EPR as dynamic modelation parameters to add label restoration details into feature maps for emotion recognition.

In the pretraining (Fig. 2 (a)), we train $FPEN_{S1}$ and DTnetwork together. We first concatenate ground-truth label and UDC images together obtained by CLIP[radford2021clip] text and image encoders to obtain the input for $FPEN_{S1}$. Then, $FPEN_{S1}$ extract the EPR $Z\in R^{C}$ as:

Then EPR $Z$ is sent into DGNet and DMNet of DTnetwork as dynamic modulation parameters to guide label restoration:

where $\circ$ indicates element-wise multiplication, LN denotes layer normalization, $W^{l}$ represents weights of linear layer, $F$ and $F^{{}^{\prime}}$ are input and output feature maps.

Then, we aggregate global spatial information in DMNet. Specifically, $F^{{}^{\prime}}$ is projected into query $Q$, key $K$ and value $V$ by using convolution layer and depth-wise convolution layer. After that, we reshape the query $Q$ to $H^{"}W^{"}\times C^{"}$, key $K$ to $C^{"}\times H^{"}W^{"}$ and value $V$ to $H^{"}W^{"}\times C^{"}$. After that, we perform dot-product between $Q$ and $K$ to generate a transposed-attention map $A$ with size $C^{"}\times C^{"}$. The overall process of DMNet can be descrive as follows:

where $\alpha$ is a learnable scaling parameter. Next, in DGNet, we aggregate local features. We use $1\times 1$ Conv to aggregate information from different channels and adopt $3\times 3$ depth-wise Conv to aggregate information from spatially neighboring pixels. Besides, we adopt the gating mechanism to enhance information encoding. The overall process of DGNet is defined as:

For DILnetwork, we use window-based cross-attention mechanism to extract facial landmarks features and UDC images features. For UDC image features $X_{udc}\in R^{N\times D}$, we first divide the UDC them into several non-overlapping windows $x_{udc}\in R^{M\times D}$. For the facial landmarks feature $X_{flm}\in R^{C\times H\times W}$, we first down-sample it into the window size $x_{flm}\in R^{c\times h\times w}$, where $c=D$ and $M=h\times w$. So, we can do cross-attention with $N$ heads on the facial landmarks and UDC images.

where $w_{Q}$ $w_{K}$, $w_{V}$, $w_{O}$ are the mapping matrix and $b$ is the relative position bias.

We perform the above cross-attention calculation for all windows. We refer to this cross-attention mechanism as window-based multi-head cross-attention (MHCA).Thus the cross-fusion transformer encoder in LRDif can be expressed as follows:

We need to merge the output features $F$ from DTnetwork and the output features $O$ from DILnetwork to get the fused multi-scale features $x_{1}$, $x_{2}$ and $x_{3}$, where $x_{1}=Concat(F_{1},O_{1})$, $x_{2}=Concat(F_{2},O_{2})$ and $x_{3}=Concat(F_{3},O_{3})$. Then, the fused features $X$ will input into vanilla transformer blocks for processing.

where $MSA$ is multi-head self-attention blocks. $LN$ is layer normalization function.The training loss is defined as follows:

We use cross-entropy loss to train our model, where N is the number of samples. M is the number of classes. $y_{ic}$ indicates whether class c is the correct classification for sample i (1 if true, 0 otherwise). $p_{ic}$ is the predicted probability of sample i belonging to class c.

In the second stage (Fig. 2), we exploit the strong data estimation ability of the DM to estimate EPR. Specifically, we use the pretrained FPEN_S1 to caputure the EPR $Z\in R^{C}$. After that, we apply the diffusion process on Z to sample $Z_{T}\in R^{C}$, which can be described as :

, where T is the total number of iterations, $\bar{\alpha}_{T}=\prod_{i=0}^{T}\alpha_{i}$ and $\alpha_{t}=1-\beta_{t}$. $\beta_{t}$ is the predefined scale factor. $\mathcal{N}(.)$ represents the Gaussian distribution.

In the reverse process of DM, we first use CLIP image encoder $E_{I}$ to encode the input UDC image $x$. After that the encoded features will be sent to FPEN_S2 to abtain a conditional vector $x_{S2}\in R^{C}$ from UDC images.

,where $\text{FPEN}_{S2}$ has the same structure as $\text{FPEN}_{S1}$ except the input dimension of the first Linear layer. Then, we use the denoising network $\epsilon_{\theta}$ to estimate noise in each time step $t$ as $\epsilon_{\theta}(\text{Concat}(Z_{t}^{{}^{\prime}},t,x_{S2}))$. The estimated noise is substituted into Eq. (17) to obtain $Z_{t-1}^{{}^{\prime}}$ to start next iteration:

After T times iterations, we obtained final estimated EPR $Z_{0}^{{}^{\prime}}$. We joint train FPEN_S2, denoising network, and UDCformer using $\mathcal{L}_{total}$:

where $Z_{\text{norm}}(i)$ and $\bar{Z}_{\text{norm}}(i)$ are EPRs extracted by LRDif_S1 and LRDif_S2 and normalized with softmax operation. $\mathcal{L}_{kl}$ is a variant of the Kullback-Leibler divergences, summing over C dimenstions. We add Kullback-Leibler divergence loss $\mathcal{L}_{kl}$ (Eq. 18) and Cross-Entropy loss $\mathcal{L}_{ce}$ (Eq. 14) to form the total loss $\mathcal{L}_{total}$ (Eq. 19). Since the EPR includes the UDC image and related emotion label encoded by CLIP, LRDif_S2 can use few iterations and small model size to obtain quite good estimations.

In the inference stage, we only use the reverse diffuion process, which is described in Algorithm 2. FPEN_S2 extracts a conditional vector $x_{S2}$ from UDC images encoded by CLIP, and we randomly sample a Gaussian noise $Z_{t}^{{}^{\prime}}$. Denoising network utilises the $Z_{t}^{{}^{\prime}}$ and $x_{S2}$ to estimate the EPR $Z^{{}^{\prime}}$ after T iterations. After that, UDCformer exploits the EPR to restore label $y^{{}^{\prime}}$.

RAF-DB [rafdb] is acquired through in the wild, with each image meticulously annotated by 40 independent raters to ensure precision. It is composed of two subsets: a basic subset with single annotations and a compound subset with dual annotations. Our study employs the basic subset. This particular subset includes seven emotional categories: surprise, fear, disgust, happiness, sadness, anger, and neutral. The dataset is partitioned into a training set comprising 12,271 images and a testing set consisting of 3,068 images, both of which exhibit a nearly equivalent distribution of expressions.

FERPlus [ferplus] serves as an extension of the FER2013 [goodfellow2013challenges], enriched with a fresh set of labels created by ten annotators. We use the enhanced dataset includes 28,709 images for training, 7,178 for testing. FERPlus incorporates ‘Contempt’ as an additional emotion category, thereby extending the classification categories to eight distinct emotional classes.

KDEF [kdef] represents a comprehensive dataset encompassing a collection of 4,900 images, meticulously designed to describe human emotional expressions. This dataset includes images of 70 amateur actors, balanced with 35 females and 35 males, each demonstrating seven distinct emotional expressions. A unique aspect of this dataset is the multi-angled approach, where each expression is captured from five different angles, offering a rich, diverse set of visual data. The participants were selected based on specific criteria: they are between 20 and 30 years of age and exhibit a clear, unobstructed facial appearance.

UDC-RAF-DB dataset comprises a training set with 12,271 images and a testing set encompassing 3,068 images, offering a robust platform for developing and testing FER algorithms in UDC environments.

UDC-FERPlus dataset extends UDC realm, providing an expansive collection of 28,709 UDC images for training and 7,178 for testing.

UDC-KDEF dataset includes a total of 4,900 UDC images, offering varied perspectives with images captured from five different angles. The training set comprises 3,920 UDC images, while the testing set includes 980 images, ensuring a wide range of data for effective training and evaluation of FER systems in UDC contexts.

We trained a SOTA MPGNet [zhou2022modular] for modeling the UDC imageing degradation process and a U-Net-like network called DWFormer for UDC image generation. The UDC imaging degradation process contains brightness attenuation, blurring, and noise corruption. We use the pretrained MPGNet to synthesize three benchmark Facial Expression Recognition (FER) datasets: the UDC-RAF-DB dataset, UDC-FERPlus dataset, and UDC-KDEF dataset. All of the experiments are implemented with PyTorch, and the models are trained with a GTX-3090 GPU.

Table V provides a comparative analysis of various facial expression recognition (FER) models, focusing on the Parameters and FLOPs as evaluated on the UDC-RAF-DB and UDC-KDEF dataset. Our proposed model stands out with a substantial 211.5 million parameters and 8.9 billion FLOPs, suggesting a more complex and computationally intensive architecture. Nevertheless, it demonstrates a high level of accuracy with 87.90% on UDC-RAF-DB dataset and a notable 94.07% on UDC-KDEF dataset, indicating that increased computational requirements are justified by enhanced performance outcomes. In contrast, models like ARM[2021arm], RUL[2021rul], and SCN[2020scn], while being more lightweight with around 11.2 million parameters and 1.8 billion FLOPs, exhibit moderate accuracy levels. The POSTER++ [2023posterv2] model, with a middle-ground approach using 58.3 million parameters and 8.5 billion FLOPs, achieves substantial accuracy, particularly on the UDC-KDEF dataset with 91.92%. MANet [2021manet], another relatively large model with 50.5 million parameters and 3.7 billion FLOPs, shows competitive but slightly lower accuracy scores compared to our model. EAC [2022eac], with 23.5 million parameters and 3.9 billion FLOPs, records the lowest accuracy on UDC-KDEF dataset, suggesting a possible inefficiency in its architecture when dealing with the specific challenges of UDC images. The table effectively underscores the trade-offs between model size, computational cost, and performance, showcasing our model’s ability to achieve robust results while maintaining computational efficiency, thereby providing an effective solution for high-accuracy FER in UDC contexts. Visualisation of probability distribution of labels

We use t-SNE to visualise the learned feature distributions of different methods to show effectiveness of LRDif on UDC images. The results are shown in Fig. 5. It is shown that the comparison of different facial expressions encourages intra-class compactness and inter-class seperability of the learned features. Compared with Fig. 5 (a) and Fig. 5 (b), SCN can’t seperated the emotion categories well, especially on UDC images. While LRDIf(Ours) can recognize expressions from the mixed feature and noisy images, and it will be forced to learn the most discriminative feature that can tell an expression image apart from all the other expression. Compared with Fig . 5 (c) and Fig. 5 (d), we can draw the conclusion that LRDif can automatically prevent the model from remembering noisy features and learn useful features from UDC images.

The loss functions for DM.

Impact of the number of iterations.

In this part, we explore how the numbe of iterations in DM affects the performance of LRDif_S2. We set different number of iterations in LRDif_S2 and tune the $\beta_{t}$ ($\alpha_{t}=1-\beta_{t}$) in Eq. 15 to make $Z$ be Gaussian noise $Z_{T}\sim\mathcal{N}(0,1)$ after diffusion process (i.e., $\alpha_{T}\rightarrow 0$). The results are shown in Fig. 7. As iterations increase to 4, the performance of LRDif_S2 will significantly improve. As the number of iteration is larger than 4, LRDif_S2 almost keep stable, which means it reaches the upper bound. Besides, we can see that our LRDif_S2 has more quick convergence speed than traditional DM (requiring more than 50 iterations). That is because we merely perform DM on EPR (a compact one-dimensional vector).