Improving fairness in speaker verification via Group-adapted Fusion Network

The base and group-adapted encoders are three separate deep neural networks with the same architecture (ResNet-34 variants [22]). All are trained with metric learning objectives [22]. We first train the base encoder with gender-mixed data to capture generic voice characteristics, and then fine-tune the pre-trained base encoder with gender-specific training data. In the training stage, inputs to each encoder are mini-batches (batch size $N$) of audio features $X_{1},X_{2},...,X_{N}\in R^{T\times F}$ (e.g., log Mel filter banks), where $T$ is the number of audio frames and $F$ is the feature dimension. Outputs from each encoder are length-normalized embeddings $E\in R^{D}$, where $D$ is the embedding dimension. Embeddings are fed into the metric loss function for network training. We use $E_{i}^{B},E_{i}^{M},E_{i}^{F},i=1,...,N$ to denote base embeddings, female-adapted embeddings and male-adapted embeddings produced from corresponding encoders.

We next leverage a score fusion model to aggregate the three embeddings, to predict if an utterance pair $({X}_{1},{X}_{2})$ is from the same speaker. In the score fusion stage, the inputs are utterance-level base embedding pairs $({E}^{B}_{1},{E}^{B}_{2})$, female- and male-adapted embedding pairs $({E}^{F}_{1},{E}^{F}_{2})$ and $({E}^{M}_{1},{E}^{M}_{2})$. First, we compute cosine similarities $S^{B},S^{F},S^{M}$ between pairs for base, female-adapted and male-adapted embeddings, respectively:

$$\begin{split}S^{B}&=\text{CosineSimilarity}({E}^{B}_{1},{E}^{B}_{2}),\\ S^{F}&=\text{CosineSimilarity}({E}^{F}_{1},{E}^{F}_{2}),\\ S^{M}&=\text{CosineSimilarity}({E}^{M}_{1},{E}^{M}_{2})\\ S&=\text{Sigmoid}(f([S^{B},S^{F},S^{M}];W)).\end{split}$$

(1)

We then use a score fusion model $f(\cdot)$ to aggregate the three similarity scores into the fused score for speaker verification. We employ a multilayer perceptron (MLP) $f(\cdot)$ with the three similarity scores $[S^{B},S^{F},S^{M}]$ as inputs and $W$ as the learnable model weights. To train the score fusion model, we construct positive and negative training pairs from the training set for contrastive learning. Positive utterance pairs $\mathcal{P}$ are sampled from the same speaker; negative pairs $\mathcal{N}$ are formed by sampling utterances from different speakers of both same and different gender. We train the fusion model with binary cross-entropy loss

$$\begin{split}L=-\frac{1}{M}\left(\sum_{n\in\mathcal{P}}y_{n}\log S_{n}+\sum_{n\in\mathcal{N}}(1-y_{n})\log(1-S_{n})\right)\end{split}$$

(2)

where $S_{n}$ is the fused score output of the $n$-th utterance pair $({X}_{1},{X}_{2})_{n}$, $M=(|\mathcal{P}|+|\mathcal{N}|)$ and $y_{n}$ is the corresponding label ($y_{n}=1$ indicates the paired utterances are from the same speaker, $y_{n}=0$ otherwise).

During inference, given a pair of utterances for verification, we first extract their base, female- and male-adapted embeddings from the three encoders. The three embeddings are fed into the score fusion model producing score $S$. If $S$ is greater than a predefined threshold, we predict that two utterances are from the same speaker; otherwise, they are deemed from different speakers.

We first investigate the impact of imbalanced training dataset on model fairness in typical deep speaker verification models. We consider two state-of-the-art deep neural networks [22], a quarter-channel ResNet-34 baseline (Q/RN) with 1.4M parameters and a larger half-channel ResNet-34 baseline (H/RN) with 5.6M parameters. The two baselines are trained on the VoxCeleb2-GRC datasets and evaluated on the VoxCeleb1-F dataset.

As showed in Figure 2,²²2All results in tabular form are also available on the website given in Footnote 1. when the total number of speakers in training set are kept the same, the majority group has better group-wise EER than the minority group. This is, increasing dominance of one gender group (e.g., from ratio 4:1 and 9:1) leads to increasing performance gap or model unfairness, indicated by the increasing $DS$ values (e.g., from 1.71 to 3.70). For example, when F:M = 9:1 in the training set, the Q/RN baseline has a female-group EER of 3.52, which is much better than the male-group EER of 7.22. A similar gap between female EER and male EER is observed in the setting of F:M = 1:9. When the training set is balanced, both baselines achieve roughly equal group-wise EER for both genders, indicated by the lowest $DS$ values ($<0.35$). Notably, overall EER increases as the training dataset becomes increasingly imbalanced, even though total training data sizes remain similar.³³3The differences between operating points for group-wise EERs and overall EER are less than 0.09.

Among the two baselines, H/RN baseline achieves better group-wise and overall EERs among all gender ratios and smaller $DS$ than those of Q/RN. We attribute the fairness improvement to the larger learning capacity of the H/RN baseline. However, the H/RN baseline, which is four times of Q/RN baseline in model size, can only provide relatively small improvement on group-wise EERs and overall EER ($\approx$5%).

Now we consider the performance of the proposed group-adapted fusion network (GFN) model. The GFN has 4.3M parameters, which is around 3 times that of Q/RN and smaller than H/RN. Compared to the two baselines (in Figure 2), the GFN achieves better group-wise and overall EERs regardless of gender group imbalance in the training sets. In particular, GFN achieves a female group EER of 3.12 (realizing 11.4% and 6.9% improvement relative to Q/RN and H/RN, resp.), and a male group EER of 5.88 (18.6%/13.7% improvement) in the F:M=9:1 setting. For overall EERs, GFN achieves an EER of 5.84 (13.9%/9.6% improvement) and 5.08 (28.6%/29.0% improvement) in the F:M = 9:1 and 1:9 settings, respectively.

Note that GFN offers larger relative improvements for the minority group than for the majority group, thereby reducing the model unfairness or disparity score $DS$ in the imbalanced cases. Specifically, GFN achieves a $DS$ of 2.76 (25.4%/20.2% improvement relative to Q/RN and H/RN) and 2.23 (24.2%/21.2% relative improvement) in the F:M = 9:1 and 1:9 settings, respectively.

To shed light on the cause of model unfairness and the effects of GFN modeling, we first use t-SNE to visualize utterance embeddings (10000 female and male base embeddings from Q/FN trained on F:M=1:1 dataset) in a 2D space. As shown in Figure 3(a), utterances of the same gender tend to aggregate in separate regions of the embedding space, which is consistent with the perception that same-gender voices sound relatively similar compared to between-gender differences. We hypothesize that in an imbalanced training setting, adapting encoders separately for different genders allows each encoder to improve the representation of different subregions of the embedding space, thus alleviating the bias toward the dominant group found in the single-model framework.

The t-SNE method can visualize the benefit of adapting and fusing group-specific embeddings by the clustering and separation of embeddings from different speakers. Figures 3 (b) and (c) show the low-dimensional utterances embeddings from the baseline Q/RN encoder and from the concatenated three GFN embeddings, respectively. The adapted encoders tend to generate more compact speaker clusters with more separation between speakers, compared to the baseline encoder. The compactness of speaker clusters can also be quantified via silhouette coefficients (SC) [23] used in clustering analysis, implemented by Scikit-learn [24]. SC measures how similar an utterance is to its own speaker cluster (compactness) compared to other speaker clusters (separation); a higher SC is better. We compute the SC from 80 utterance embeddings of 8 randomly-sampled speakers. The mean SC from the Q/RN baseline is 0.64, while the mean SC from the adapted encoder is 0.83, indicating that the adapted encoder extracts better embeddings for speaker differentiation purposes.

Several ablation studies reveal the benefits of different aspects of our model, as summarized in Figure 4. We first examine the impact of using only one adapted encoder without fusing other encoders. Using a single male-adapted encoder (M-FT) typically degrades the overall EER, particularly the female group-wise EER although it might improve the male group-wise EER slightly when the male group is the minority. A similar phenomenon is observed when only using a female-adapted encoder (F-FT). Since male and female voices have distinct characteristics, we hypothesize that fine-tuning an encoder to one gender subset can cause the encoder to forget the learned features of the other gender. Therefore, it is necessary to fuse separately-adapted encoders to achieve better performance. To further verify the efficacy of the score fusion strategy we compare GFN’s 3-layer MLP score fusion model with an equal-weight score (ES) fusion strategy (i.e., 1/3 for each input score). ES fusion achieves better group-wise EER than the baselines for gender-imbalanced settings, but gives overall worse performance than GFN.

Additionally, we consider an alternative embedding adaptation method named “gender batching with weighted loss” (GBWL), which fine-tunes the Q/RN base model by 1) alternating all-female and all-male mini-batches and 2) reciprocal weight for the majority-gender minibatch (i.e., when training on F:M=1:9, we scale the loss from all-male minibatches by 1/9). The GBWL method significantly degrades group-wise and overall EER in the baseline Q/RN (and similarly for H/RN). This shows the benefit and necessity of performing embedding adaptation using separate networks.

We also investigated the gated mixture-of-experts (MOE) [20] strategy for fusing scores, which incorporates both adapted embeddings and similarity scores as inputs. However, gated-MOE achieves worse performance in the current score fusion training setting.