Is It Still Fair? Investigating Gender Fairness in Cross-Corpus Speech Emotion Recognition

Speech emotion recognition (SER) has made significant strides by reshaping the landscape of human-computer interaction and enabling emotion-aware applications [1, 2, 3]. Having more generalized SER models has led to growing interest in cross-corpus SER modeling, which aims at developing models capable of generalizing across diverse corpora and linguistic contexts. This entails many strategies to compensate features, domains, or label mismatches, using techniques such as transfer, semi-supervised, and few-shot learning [4, 5, 6, 7]. While these models have shown remarkable recognition performance on the target corpus but what about the fairness in performance?

In recent years, there has been an increasing emphasis on the trustworthiness of intelligent models, particularly concerning aspects such as privacy, fairness, safety, etc. Specifically, The notion of fairness has gained substantial attention and become a focal point of extensive discussions. Numerous studies within the SER domain have recognized the issue of fairness, highlighting the model exhibits biases with sensitive attributes. To address these fairness concerns, various techniques have been proposed, including the use of adversarial networks, reweighing schemes, and the development of loss functions designed to mitigate biases [8, 9, 10, 11]. Some studies have specifically aimed at neutralizing different attributes, such as gender or age, within SER approaches [12, 10]. Most of these efforts have been focused on single-dataset (source-only) scenarios. However, the unfairness can manifest across multiple levels including sample, data distribution, modality, labeling, and domain-related aspects [9, 10, 12]. The effectiveness of the model's fairness performance when deployed on another corpus has received limited, if any, exploration in the context of cross-domain SER scenarios. Emotions, with their intricate nature and cultural influences, manifest a diverse spectrum of expressions and interpretations across different domains and subjects. Hence, it becomes uncertain whether a SER model excelling in one emotional domain will demonstrate similar performance in another, particularly when accounting for cultural-linguistic variations.

However, a SER model tailored to a specific source corpus may introduce unfairness related to sensitive attributes like gender or age in the target corpus, amplifying some emotions while neglecting others. Cross-corpus SER models, designed for generalization across domains or cultural contexts [13, 14], must also ensure fairness across diverse demographics within these domains. This study emphasizes the dual considerations of performance and fairness in cross-corpus SER models. We investigate the generalizability of fairness in cross-lingual SER models, employing a few state-of-the-art transfer learning and fairness techniques from the literature. Gender is specifically examined as the protected attribute of interest. Our study evaluates whether a cross-lingual SER model, which demonstrates fairness within its source corpus by showing no gender sensitivity, maintains this fairness when applied to the target corpus. This study introduces a novel perspective in cross-corpus SER, underscoring the importance of integrating fairness considerations alongside performance when deploying SER systems across corpora.

In this study, for cross-corpus SER fairness investigations, we utilize two large naturalistic speech corpora, MSP-Podcast [15] and BIIC-Podcast [16]. Our experimental results reveal two insights, first despite exhibiting source-specific fairness with various transfer learning approaches, cross-corpus SER generalization can introduce gender biases for the target corpus which questions the fairness of the SER model. Second, we propose a simple fairness mechanism of combined fairness adaptation that integrates fairness towards the source and the target corpus while modeling a cross-corpus SER. Our proposed cross-corpus setting based Combined Fairness Adaptation (CFA) idea achieves significantly better gender fairness (GF) compared to the considered baselines of this study. The preliminary findings of this study demonstrate the effectiveness and necessity of research in this direction.

Building on insights from Section II, cross-corpus SER models face challenges with GF generalizability in target corpora. To address this, we propose a strategy (Fig. 1) for the GF generalized cross-corpus SER. The architecture has two key blocks: an emotion classification (EC) block and a combined fairness adaptation (CFA) block. For testing our idea, in the EC block, we use a basic SER architecture; processing wav2vec2.0 [20] features with a transformer and four fully-connected layers for binary emotion classification. The CFA block adds an auxiliary adversarial network for gender classification (GC) with a reverse gradient layer which transforms EC features into a gender-neutral state by minimizing GC's ability to predict gender from both corpora. Other experimental settings and evaluation metrics match those in Section II-B.

The training process unfolds in stages. Initially, we train the primary SER model on the source corpus, focusing on capturing gender-neutral emotional features while disregarding gender-specific information. Subsequently, we create mini-batches that combine data from both the MSP-P and BIIC-P for gender-neutral adversarial training. The EC aims to generate gender-neutral embeddings, while the GC aims to make accurate gender predictions. This training process uses a binary cross-entropy loss function for classifying gender, penalizing the classifier for making gender predictions based on shared feature representations. In each training iteration, we update both the primary model's weights and the gender classifier's parameters. The inserted reverse gradient layer in learning modifies the gradients backpropagated from the GC branch to make the shared feature representations more gender-neutral. Equation 1 shows the binary classification loss used for EC and GC tasks.

where $z$ is the intermediate embeddings produced by the primary EC model. $y$ be the true labels for emotions (0 for target, 1 for other) for EC and the gender of speakers (0 for male, 1 for female) for GC task. $D$ represents the EC or GC, $P(y|z)$ be the predicted target attribute probability distribution given $z$, and $\theta_{D}$ be the parameters. To enforce a similarity between source and target genders, we use a contrastive loss that encourages samples with the same gender to be close in the feature space and those with different genders to be distant from each other. The goal is to make the learned representations of gender-related features similar between the source and target domains. The Equation 4 is used to integrate a source and target similarity loss.

where D is the Euclidean distance between the embeddings of samples $x_{1}$ and $x_{2}$. m is the margin and here is fixed to 1. This loss encourages the model to minimize the distance between samples with the same gender ($y=0$) and maximize the distance between samples with different genders ($y=1$). Equation 5 illustrates the total loss for the PA-CFA model, which combines the EC loss and the GC loss.

where ${L}_{emo}$ and ${L}_{GC}$ are the binary emotion classification and gender classification loss. ${L}_{GSim}$ is the loss for the gender similarity. $\alpha$ and $\beta$ are set to 0.5. Here, during training, target gender labels are needed, but not during inference.

To align genders between source and target domains, we apply a contrastive loss to encourage similar gender samples to be closer in feature space and dissimilar ones to be farther apart. Equation 4 illustrates how we integrate this source and target similarity loss.

where D is the Euclidean distance between the embeddings of samples $x_{1}$ and $x_{2}$. m is the margin and here is fixed to 1. This loss encourages the model to minimize the distance between samples with the same gender ($y=0$) and maximize the distance between samples with different genders ($y=1$). Equation 5 illustrates the total loss for the PA-CFA model, which combines the EC loss and the GC loss.

where ${L}_{emo}$ and ${L}_{GC}$ are the binary emotion classification and gender classification loss. ${L}_{GSim}$ is the loss for the gender similarity. $\alpha$ and $\beta$ are set to 0.5. Here, during training, target gender labels are needed, but not during inference.

Table II shows gender-specific performance and fairness metrics for Base-CFA with a basic SER model and PA-CFA, combined with the PA TL model. Table II demonstrates that on cross-evaluation (on BIIC-P test set), our proposed CFA baseline Base-CFA method outperforms PA-ReW in terms of GF generalizability across most emotion categories. For example, for Anger, Base-CFA achieves balanced gender-specific performance (M:69.28%, F:71.56%) compared to PA-ReW (M:52.98%, F:45.37%). In terms of fairness metrics, CFA shows a significant reduction in both $\Delta SP$ and $\Delta EO$ across all emotions compared to the PA-ReW. For instance, the SP value for Anger decreases by 0.107, reflecting improved GF. Additionally, PA-CFA, which combines phonetic-based constraints with CFA, captures language-specific information in cross-lingual tasks, exceeds the performance of PA-ReW. For example, for Anger,$\Delta SP$ drops from 0.363 to 0.260, a consistent trend across both fairness metrics.

We validate our approach by training a model without the gender similarity loss (${L}_{GSim}$), called PA-Adv. Table II shows that PA-CFA still achieves better fairness. For example, in Anger, PA-Adv has a $\Delta SP$ of 0.322, while PA-CFA achieves 0.260, highlighting the effectiveness of ${L}_{GSim}$ in improving fairness by aligning gender features across corpora. For reference, Table III presents the overall UAR performance of the CFA model. From Table III, we can see that PA-CFA shows a slight decrease in performance compared to PA, but it remains competitive over all emotion categories. We believe this minor performance drop is a worthwhile trade-off for improved GF. For reference, the overall UAR represents recall across all classes, irrespective of gender. In Table III, UAR values are shown separately for males and females, so the overall UAR shown in Table III is not an average of gender-specific UARs.

We also analyze embeddings from both the PA-ReW and proposed PA-CFA models to assess gender-related information across the MSP-P and BIIC-P contexts. Figure 3 presents the gender detection (GD) accuracy, with MSP-P on the left and BIIC-P on the right. PA-ReW exhibits varying GD accuracy: lower for MSP-P and higher for BIIC-P, suggesting residual gender information specific to BIIC-P despite lower MSP-P performance. Conversely, PA-CFA shows improved gender neutralization, with similar GD accuracy across both corpora (e.g., for Anger, 36% MSP-P, 35% BIIC-P in Figure 3(b)). In PA-CFA, there are improvements in emotions like Neutral and Anger for MSP-P GD accuracy compared to PA-ReW, reflecting compensation in MSP-P performance due to enhanced gender fairness across MSP-P and BIIC-P corpora.

This research addresses overlooked fairness concerns in cross-corpus SER, particularly focusing on gender neutrality. Our findings reveal that cross-corpus SER models, while fair within their source corpus, introduce biases when generalized across different corpora. We propose an initial approach, combined fairness adaptation (CFA), to enhance gender neutrality across both source and target corpora in emotion transfer tasks. Initial experiments demonstrate the efficacy of our approach in creating gender-fair cross-corpus SER systems. Future research will refine our fairness mechanism through feature-side analysis to pinpoint specific areas where fairness issues arise in cross-corpus SER settings.