Model-Based Approach for Measuring the Fairness in ASR

Poisson regression serves as an appropriate approach to model rate data [15], where the rate is a count of events (e.g. word errors in our use case) divided by some measure of that unit’s exposure (e.g. number of words in the reference). An offset variable is needed to scale the modeling of the mean parameter in Poisson regression with a log link. Here, the underlying assumption is that the number of word errors occurred in any utterance is proportional to the number of words in the corresponding reference text.

More specifically, to measure the effect of factor $f(\cdot)$ on WER results across $l$ different subgroups, the vanilla Poisson regression model is described as follows:

where $C_{s}$ is the count of word errors (sum of insertion, deletion, and substitution errors), $\lambda_{s}$ is the Poisson (mean) parameter, $N_{s}$ is the number of words in the reference text for the $s$th utterance in the evaluation dataset, and $\mu_{f(s)}$ refers to the factor effect corresponding to the subgroup of $f(s)$. The notation of i.i.d in (2) represents independent and identically distributed, where we will revisit this distribution assumption later in this section. Note that any utterance with empty reference text should be removed from the analysis since it does not provide any insight on fairness measurement.

This model can be fitted using maximum likelihood approach. Standard statistical testing, for example, likelihood ratio test (LRT) [16], can be conducted afterwards to compute the $p$-value of the factor $f(\cdot)$ on measured WER results.

Sometimes, it is possible to analyze rate data using a binomial response model. However, in our application, number of word errors occurred in some utterance could be larger than the total number of words in the reference, which limits the use of binomial regression here. If the rate is relatively small, the Poisson approximation to the binomial is effective.

One of the key features of Poisson distribution is that the variance equals the mean. In certain circumstances, it is found that the empirical variance is greater than the mean, known as overdispersion [17, 18]. A common reason is the omission of relevant explanatory variables, or the present of dependent samples, which we will explore more in the next two subsections.

It is natural and flexible to extend the vanilla Poisson regression model (2) (3) to include additional explanatory or confounding covariates, which can be utilized to capture effects of nuisance variables on WERs among different subgroups:

Here, $x_{s}$ represents the vector of any explanatory variables in the regression model and $\theta$ refers to the coefficient parameter vector that shall be learned. For example, in a racial disparity analysis, we would want to add the gender or age information of speakers to the regression model in order to control any nuisance effects.

In particular, we can include any representative vector [19], for example, sentence embedding, of the true reference text per each utterance as extra explanatory variables, which would help us understand the source of any performance gap between different subgroups of interest. For instance, after controlling the effect of sentence embedding covariates that account for grammatical, lexical, or semantic characteristics, if the factor effect of interest is still statistically significant, we can tell that phonetic, phonological, or prosodic characteristics substantially contribute to the overall disparities among different subgroups of the factor $f(\cdot)$. Thus this can provide insights on whether language model or acoustic model should be responsible for the overall disparities in WERs if at all.

Block-structured evaluation data arises naturally in any real-world speech recognition applications. In particular, utterances from the same speaker could share common correlated features (e.g. accent of speaker), and thus analyses that assume independence of these observations will be inappropriate. The use of random effect [13, 14] is one usual and convenient way to model such structure.

Suppose we want to investigate the effect of race on speech recognition accuracy across a sample of speakers. Typically, we would treat the racial effect as fixed in the regression. On the other hand, it makes most sense to treat the speaker effect as random. It is reasonable to consider these speakers as being randomly selected from a larger collection of speakers whose characteristics we would like to estimate. We are not particularly interested in these specific speakers, but in the whole population. Generally, blocking factors can often be viewed as random effects.

A mixed-effects Poisson regression is a model containing both fixed effect and random effect. Regarding the fairness measurement of speech recognition accuracy among different subgroups of the factor $f(\cdot)$, we describe the model in detail as follows:

where the utterance-level index of subscription notation $ij$ represents the $j$th utterance from the $i$th speaker, $r_{i}$ denotes the speaker-level random effect that is independently sampled from a Gaussian distribution with mean 0 and variance $\sigma^{2}$ which is learnable. Note that any $C_{ij}$ and $C_{ij^{\prime}}$ are no longer independent for $j\neq j^{\prime}$ since they are observed from the same speaker $i$, while any $C_{i\cdot}$ and $C_{i^{\prime}\cdot}$ are still independent for $i\neq i^{\prime}$ since they are observed from different speakers. Also, we use $\mu_{f(i)}$ to denote the fixed effect for the factor $f(\cdot)$ of primary interest, since typically it is at speaker level.

This mixed-effects model can be fitted via maximum likelihood and the expression for its likelihood is an integral over the random effect, which must be approximated, for example, via adaptive Gauss-Hermite quadrature [20]. Again, LRT can be performed to calculate the $p$-value of the factor $f(\cdot)$ on measured WER results. In practice, it would be particularly interesting to extract the conditional modes of the speaker-level random effect for subsequent analysis and assumption verification.

We generate synthetic data to investigate the effect of confounding factor on ASR fairness measurements over case group and control group, defined by some primary factor of interest.

Under the scenario that recognition errors from different utterances are independent from each other, the number of errors on the $s$th utterance is randomly sampled from a Poisson distribution with the mean parameter written as

where $f(s)\in\{{\text{case}},\text{{control}}\}$ indicates which group the utterance comes from, $N_{s}$ denotes the number of words in the reference, $\mu_{f(s)}$ refers to the group effect, $\theta_{s}$ represents the effect of confounding factor, and $p_{f(s)}$ is the mean parameter of a Bernoulli distribution which controls the frequency that the confounding effect is present in the corresponding group.

In our experiment, we set $N_{s}=10$, $\mu_{f(s)}=\log(0.05)$, $\theta_{s}=0.1$ for every $s$, and $p_{f(s)}$ is varied at 50%, 60%, 70%, 90% for the case group, and 50%, 40%, 30%, 10% for the control group, respectively. For each of case or control group, we generate 5,000 utterances independently.

Here, we would like to evaluate the ratio of WERs between the case and control groups, and in particular, conduct statistical testing to determine whether significant difference on WERs exists between the two groups. Based on our setup, the ground truth WER ratio is 1.0, that is, in theory there is no WER difference between the two groups. Notice that the presence of confounding factor could introduce nuisance and mislead the results since it raises up the mean number of errors by $\exp(0.1)-1\approx 11\%$ at utterance level.

In this study, we consider the baseline measurement method as the one that computes the ratio of empirical WER of case group over the one of control group. The bootstrap method [21, 22] is applied to compute the 95% confidence interval (CI) of the ratio. Then if the CI does not cover the point of 1.0, we claim the WER gap between the two groups is statistically significant. Regarding model-based approach, we fit a Poisson regression according to (2) (4) which linearly incorporates the confounding factor, and then compute the 95% CI associated with the group effect ratio.

For each approach, we repeat the simulation for 1,000 times and compare the average of estimated WER ratios as well as the false positive rate, that is, the frequency of times that the statistical significance on WER ratio is falsely claimed by the method. Strictly speaking, a 95% CI means that if we were able to have 100 different datasets from the same distribution of the original data and compute a 95% CI based on each of these datasets, then approximately 95 of these 100 CIs will contain the true value of the statistic of interest [23, 24, 25]. Thus in theory, we expect 5% false positive rate if the method works correctly and generates valid CIs.

The result is shown in Table 1, where we can see that the mean ratio and false positive rate of the baseline method increase dramatically when the confounding rate differs more and more between case and control groups. This is expected since the baseline method does not take into account the information of confounding factor which does harm to the inference. For model-based approach, we observe the mean ratios are around 1.0 and the false positive rates are around 5% for all the setups, which demonstrates that it can successfully address the confounding effect and result in valid estimates of WER ratios and corresponding CIs.

In this experiment, we generate synthetic data to study the impact of speaker effect on ASR fairness measurements of the two groups.

For any of case or control group, assume there are $I$ distinct speakers and each speaker has equal number of utterances. For the $i$th speaker and $j$th utterance from the speaker, the number of errors is sampled from a Poisson distribution with the mean parameter written as

where $f(i)$ indicates which group the speaker is from, $N_{ij}$ denotes the number of words in the reference, $\mu_{f(i)}$ refers to the group effect, and $r_{i}$ represents the speaker effect drawn from a Gaussian distribution with mean 0 and standard deviation $\sigma$.

In our experiment, we set $N_{ij}=10$, $\mu_{f(i)}=\log(0.05)$ for every $i,j$, number of speakers $I$ is varied at 100, 500 and standard deviation $\sigma$ is varied at 0.2, 0.4. For any of case or control group, we generate 5,000 utterances.

Again, we want to evaluate the ratio of WERs and the ground truth shall be 1.0. The baseline method is the same with the one used for confounding factor experiment while for the model-based approach, we fit a mixed-effects Poisson regression according to (5) (6) (7) which treats the speaker identification as a random effect with learnable $\sigma$.

The result is shown in Table 2, where we can see that the mean ratios of both methods are around 1.0. However, for the baseline method, we observe high false positive rates and in particular, the higher the standard deviation $\sigma$ or the smaller the number of speakers, the larger the false positive rate. Instead, the model-based approach always results in approximate 5% false positive rate. This demonstrates that it can successfully deal with speaker effect and is superior than the traditional baseline method.

We consider the following two ASR datasets in the experiments:

• •

  LibriSpeech [26]. A widely used voice dataset which consists of 960 hours transcribed training utterances. The evaluation dataset has the splits of Test-Clean from 40 speakers and Test-Other from 33 speakers.

• •

  Voice Command. This is a de-identified dataset collected using mobile devices through crowd-sourcing from a data supplier for ASR. No personally identifiable information (PII) is contained in this dataset. The participants are instructed to say voice commands on the topics of calling friends, playing music, etc. It consists of 2,440 hours transcribed training utterances. The evaluation set contains around 18K utterances from 95 speakers.

Table 3 shows details of the two evaluation datasets on number of utterances and number of speakers.

The ASR system in this investigation is an RNN-T model with Emformer encoder [27], LSTM predictor, and a joiner, having approximately 80 million parameters in total. For each of LibriSpeech or Voice Command data, the ASR model is trained from scratch using the corresponding training utterances.

For the LibriSpeech data, we study the ASR fairness on gender, that is, we would like to test whether there exists statistical significance on the WER ratio between male speakers and female speakers.

The baseline approach, which is widely used in practice, computes the ratio of empirical WER from male speakers group over the empirical WER from female speakers group. The bootstrap method is applied to compute the 95% CI of the ratio. For model-based approach, we fit a mixed-effects Poisson regression based on (5) (6) (7) with gender as the fixed effect and speaker label as a random effect.

Result is shown in Table 4. We can see that the baseline method leads to statistically significance claims on both Test-Clean and Test-Other sets, and interestingly, their conclusions are actually opposite. Specifically, on Test-Clean split of evaluation dataset, the baseline method shows that male speakers group has significant lower WER compared to female speakers group, while on Test-Other split, male speakers group has significant higher WER compared to the group of female speakers. On the other hand, the model-based approach does not claim any significant results on both splits. This makes sense since numbers of speakers in both splits are quite small, which results in high variance estimation that does not lead to statistically significant results. Thus utterances from more speakers are needed to reduce the standard errors and draw a more sound conclusion.

To further trace the source of WER gap, Table 5 shows the result of mixed-effect Poisson regression with sentence embedding of the true reference text as extra explanatory variables. We use pre-trained fastText word embeddings [28] with 300 dimensions and take their average to obtain the representation at sentence level. From the result, after excluding the effect from grammatical, lexical, or semantic characteristics, the WER gap between the two groups become smaller. Although it is not statistically significant, acoustic characteristics appear to contribute to the WER disparity on Test-Other.

We also investigate ASR fairness on gender for Voice Command dataset. The baseline and model-based methods are the same with the ones applied for LibriSpeech. Result is shown in Table 6. The baseline method does not claim that the WER on male speakers group is statistically significantly higher than the WER of female speakers group, but it is very close. The model-based method clearly does not lead to significant result, due to the relatively small number of speakers in each group.