On the Impact of Voice Anonymization on Speech Diagnostic Applications: a Case Study on COVID-19 Detection

Speech-based diagnostic systems can be categorized into two groups: ones that rely on carefully designed hand-crafted features coupled with conventional machine learning classifiers, and ones that input raw signals directly into a deep learning model for classification. In the latter ‘end-to-end’ scenario, the deep learning model serves as a feature extractor and feature mapping function in one.

When it comes to feature extraction from speech, the openSMILE toolkit [31] is by far the most popular. The largest feature set of openSMILE extracts over 6,000 acoustic features, including mel-frequency cepstral coefficients (MFCC), pitch contours, voicing-related information, as well as several other low-level descriptors (LLDs). This feature set has been used together with conventional classifiers, such as support vector machines (SVM), for the detection of different diseases [32, 33, 34, 35]. More recently, it has been employed as a benchmark feature set for the INTERSPEECH 2021 ComParE COVID-19 Detection Challenge [36]. For in-the-wild speech analysis, on the other hand, the modulation spectral representation (MSR) has shown benefits over openSMILE features for different applications (e.g., [37, 38]), including disease characterization (e.g., [39, 40]) and COVID-19 detection [10].

Existing end-to-end systems, in turn, have relied on variants of the spectrogram representation as input, including the mel-spectrogram or the log-mel-spectrogram, as well as convolutional or recurrent neural network architectures for classification. Han et al., for example, showed that VGGish neural networks outperformed conventional methods in classifying different COVID-19 symptoms [35]. Akman et al. developed a ResNet-like architecture for speech and cough-based COVID-19 detection [41]. The Bi-directional Long-Short-Term-Memory (BiLSTM) neural network was used in the top-performing system competing in the second Diagnosis of COVID-19 using Acoustics (DiCOVA2) Challenge [12]. Compared to conventional systems, end-to-end systems have demonstrated overall higher performance on several datasets without the need for a separate feature extraction step [42, 43, 12]. Nonetheless, recent research has shown that while end-to-end models achieve state-of-the-art accuracy on a particular dataset, those results do not transfer well to other unseen datasets, where accuracy can drop to below chance levels [44]; this was not the case with hand-crafted features and conventional classifiers.

Anonymization techniques comprise two categories: speech transformation and speech conversion. The former refers to modifications directly to the original speech, such as pitch shifting and warping [45, 46], to remove personal identifiable information from the speech signal. The latter, in turn, converts one’s voice to sound like that of another without changes in linguistic content [47]. As voice privacy concerns are on the rise, voice anonymization has gained popularity recently and, in 2020, the Voice Privacy Challenge (VPC) was created [24]. A popular method from the 2020 and 2022 VPCs employs the so-called McAdams coefficients [24, 25], where shifts in the pole positions derived from linear predictive coding (LPC) analysis of speech signals [48] are used to achieve anonymization. Another popular voice transformation method is termed voicemask [49], where certain frequency components are compressed (or stretched) to generate a lower-pitched (or higher-pitched) voice signal. Voice conversion systems, on the other hand, have usually relied on modifications to speaker embeddings, such as the x-vector [50] and the ECAPA-TDNN embeddings [51], which are assumed to only carry nonverbal information that pertains to the speaker identity alone. The modified speaker embeddings are then input with speech content sequence to a speech synthesis module to reconstruct a new speech waveform [26]. Several innovations have been proposed to the speech synthesis module to make the outcome sound more natural and of greater quality and intelligibility [52, 53, 54, 27].

Figure 1 depicts the diagram of an anonymized speech diagnostics (SD) system. Conventionally, the original voice of user X is input to a diagnostic system that will generate a positive or negative output for the tested disease and/or symptom. If an automatic speaker verification (ASV) system was trained with data from user X, the ASV system would be able to detect user X’s voice. In practice, SD systems are complex and models are often stored on the cloud, thus requiring the user’s voice (or features) to be uploaded to the cloud. This transmission of data could result in privacy concerns. To overcome this, voice anonymization can be employed locally and anonymized data (or features) are sent to the cloud. In this case, user X would not be identified by the ASV system and speech-based diagnostics could proceed in a more secure and private manner.

Based on previous experiments on COVID-19 detection (e.g., [10]), the five top-performing diagnostics systems are explored herein:

A voice transformation and two voice conversion methods are explored here to gauge their differences in speech diagnostics performance. More details are provided below.

At the time of writing, most existing COVID-19 sound datasets target cough sound, such as the COUGHVID [68], Tos COVID-19 [69], Virufy [70], and NoCoCoDa [71]. Speech sound, on the other hand, is included in fewer datasets. To maximize the variability of data distribution and avoid biased results from one single dataset, we included three publicly available COVID-19 speech datasets, namely the multilingual 2021 ComParE COVID-19 Speech Sub-challenge (CSS) dataset [11], the second DiCOVA Challenge dataset [12], and the English subset from the Cambridge COVID-19 sound database [55]. These datasets are referred to hereinafter as CSS, DiCOVA2, and Cambridge set, respectively. The demographics of three datasets are summarized in Table I. It should be noted that though the full Cambridge database contains more speech samples, the English subset has been more carefully examined by the data holders to avoid potential confounding factors (e.g., languages, data quality, class balance, etc.) [55], hence is considered more suitable for our analysis.

All three datasets were crowdsourced, volunteers across the globe were encouraged to upload their voice data and metadata via apps. The same speech content was required per dataset. With CSS, participants were asked to utter the sentence “I hope my data can help to manage the virus pandemic” at most three times in their mother tongue, with the majority of samples being uttered in English, Portuguese, Italian, and Spanish. The same speech content was used for the Cambridge set but in English only. With DiCOVA2, participants did number counting from 1 to 10 at a normal pace in English. For all datasets, participants were asked to self-declare whether they were COVID-negative (including healthy or having COVID-like symptoms) or COVID-positive (including symptomatic and asymptomatic cases). It can be noticed from Table I that all three sets contained 10% to 30% asymptomatic COVID-positive cases. Additionally, nearly half of the COVID-negative samples in CSS and Cambridge are symptomatic, which is three times higher than that in DiCOVA2.

The CSS and Cambridge datasets were partitioned into three separate subsets by the challenge organizers, namely training, validation, and test. For comparisons, we employed the same challenge partition in this study. It should be emphasized that in the CSS dataset, several COVID-positive recordings were originally sampled at 8 kHz while the majority of the other files were sampled at 16 kHz. As suggested in [36] and our previous exploration [10], keeping these up-sampled recordings has been shown to lead to overly-optimistic results since classifiers learned to capture the difference in sampling rates instead of the actual pathological pattern. Thus, we removed them from our analysis. The DiCOVA2 dataset, in turn, is comprised of development and evaluation subsets, with the evaluation data being accessible only to challenge participants. Hence, we performed a speaker-independent training-test split (80/20%) using the development subset only and left the evaluation set for testing.

As our final goal is to not only provide accurate diagnostics decisions but also ensure the protection of privacy of speaker identity, the evaluation was divided into three tasks. In the first task, we compared the effectiveness and complexity of different anonymization techniques. In Task-2, we then quantified the impact of anonymization techniques on diagnostics accuracy in different conditions. Finally, we provided explanations for the impact seen in Task-2, and explored solutions for improving the proposed systems.

Since all three datasets are imbalanced, the area under the receiver-operating-characteristic curve (AUC-ROC) was chosen as the primary metric to measure the diagnostics accuracy. We further calculated the 95% confidence intervals (CIs) using 1000$\times$ bootstrap with replacement on the test set. According to [75], CIs can reflect the variability of diagnostics accuracy when the model is applied to a different population.

As mentioned previously, cosine similarity and misclassification rate were used to quantify the effectiveness of the three anonymization methods. The similarity scores are averaged across samples from all three datasets per method, where 0 represents no resemblance and 1 represents a perfect match between two tested speech conditions. While the Equal Error Rate (EER) is commonly used to evaluate anonymization efficacy, ground-truth speaker identifiers are required for each recording in order to verify if samples are from the same or different speakers. However, speaker identifiers were not available for the CSS and DiCOVA2 datasets. Instead, we rely on misclassification rate by employing a pre-trained speaker verification model from [62]. For each recording, the model outputs a binary decision (yes/no) if it believes a pair of anonymized and clean speech signals come from the same speaker. The misclassification rate is then calculated by dividing the number of misclassified pairs over the total number of pairs per method, which reflects the percentage of successfully anonymized recordings. For an ideal anonymization system, the misclassification rate is expected to be 100%, i.e., the model should decide that all anonymized signals do not come from the same speaker as the clean signal counterpart. Lastly, computation time was recorded for each anonymization method, including the loading and exporting of the audio files. In the case of the two GAN methods, the loading time of the model itself was not taken into account in this computation.

The average cosine similarity scores between the speaker embeddings of speech files anonymized by the different methods together with the misclassification rates are shown in Figure 6. As can be seen, near perfect anonymization performance was achieved with both GAN-based methods (misclassification rates), with almost no similarity with either the original speech or the speech anonymized by other methods. On the other hand, nearly half of the McAdams-anonymized samples can be successfully detected, suggesting some speaker-unique information still remained.

The computational complexity of the three anonymization methods is presented in Table III for all three datasets. While the GAN-based methods are shown to provide better anonymization effectiveness, it requires computational times approximately 10-20 times longer than using the McAdams coefficient method. The longest time was seen with Ling-Pros-GAN, since it requires extra time to extract prosody, which involves an online training loop, and to generate and find embeddings in real-time. As model loading time was not taken into account, the computation footprint of the GAN-based methods could be larger in real-world settings. Additionally, the GAN-based methods rely on several pre-trained neural networks with millions of parameters (e.g., 22.3 million for ECAPA-TDNN embedding extractor; 10 million for the generator), which could make it challenging to be deployed on mobile devices.

The within-dataset performance of the five diagnostics systems under different anonymization scenarios is demonstrated in Figure 7. As can be seen from the average AUC-ROC scores per scenario, the highest performance is achieved under scenario A, i.e., when anonymization is not performed. When the test data are anonymized using the McAdams coefficient (scenario B1), the average AUC-ROC score over all systems dropped by 8.9% (CSS), 5.9% (DiCOVA2), and 6.3% (Cambridge) relative to scenario A. A substantial decrease was observed when using both the Ling-GAN and Ling-Pros-GAN anonymizers (scenario B2 and B3), where an average relative drop 22.5% and 18.1% was achieved respectively. Moreover, nearly all systems degraded to chance levels under scenario C where models were trained with data anonymized by one method and tested with data anonymized with another, suggesting that anonymization may drastically remove COVID-19 speech information. Diagnostic performance in the fully-informed scenarios is shown to be close to scenario A. Among the three anonymizers, McAdams anonymization leads to higher diagnostic performance on average in scenario D. Compared to the Ling-GAN, Ling-Pros-GAN shows higher performance on the English datasets (DiCOVA2 and Cambridge) and lower performance on the multilingual one (CSS).

Next, we evaluate the sensitivity of different diagnostics systems to anonymization and explore the relative drop in accuracy from scenario A to scenario B. Table IV reports the average drops seen per dataset. As can be seen, the two GAN-based methods resulted in a substantially higher degradation relative to the McAdams coefficient method, with the Ling-GAN leading to the most severe decrease. This was expected and corroborates Task-1 results, where speaker embeddings of the GAN-anonymized speech showed practically no similarity to the original speech. Meanwhile, since Ling-Pros-GAN leaves the prosody intact and generates more COVID-like embeddings, it is likely to preserve more COVID-19 attributes than the Ling-GAN, thus rendering higher anonymized diagnostic performance. Previous studies have shown that speaker embeddings (e.g., x-vector) also contain other nonverbal information and can be used for speech para-linguistic tasks [76, 77], such as speech emotion recognition [78] and disease detection [79, 80]. While the GAN-based anonymizers substitute the original speaker embedding with a dissimilar speaker embedding, the obtained results suggest that health-related vocal characteristics are likely also discarded, thus resulting in significant drops in diagnostics accuracy.

Lastly, we use scenario A as the baseline and calculate the average drop in accuracy for scenario C, showing the impact that training models completely on anonymized data would have. For both openSMILE and MSR methods, we use the PCA-SVM pipeline to avoid the effects of difference in the number of features. The comparative results are reported in Table V. As can be seen, all three diagnostic systems show degradaded performance, with the logmelspec+BiLSTM system shown to be on average more robust (21.6%) to the semi-informed anonymization scenario. Notwithstanding, it should be highlighted that the logmelspec+BiLSTM system achieved the lowest AUC-ROC in scenario A. Interestingly, with the CSS dataset, the diagnostic system based on a BiLSTM and log-mel spectrogram input resulted in substantially lower degradation percentage compared to the two other systems based on traditional engineered features and classifiers. CSS is a multilingual dataset, thus hand-crafted features (e.g., syllabic rate, speech production features) used in these models may show more sensitivity to language.

Figure 8 shows the cross-dataset performance under the thirteen different testing scenarios. In line with previous studies [44, 81], all five diagnostics systems demonstrated significantly lower performance relative to within-dataset results; the logmelspec+BiLSTM achieved the greatest drop in performance. Interestingly, in a few scenarios anonymization helped systems become more generalizable relative to the unprotected setting (e.g., scenarios B2 and C3 for the CSS-DiCOVA2 cross-database experiment). Figure 9 depicts the average change in accuracy relative to scenario A for all scenarios and diagnostic systems. While on average a 6.6% drop in accuracy was seen across all five systems, an increase of 2% and 5% was achieved with MSR+SVM and logmelspec+BiLSTM systems for scenarios C4 and C2, respectively. It is important to note that both scenarios involved GAN-based anonymized test data, thus had typically the lower cross-dataset results to start off with.

While our study shows that typical anonymization systems lead to degraded diagnostic performance, it is unclear why different systems caused different levels of degradation and why some diagnostic models could still perform decently after anonymization. To answer these questions, we performed a comprehensive evaluation of the impact of different speech aspects on diagnostic performance, including the linguistic content, speaker representation, and prosody. Similar to the experimental setup of Task-1, we now compare the within-dataset performance obtained by three categories of speech features, namely (1) the phoneme-level features, including the number of mispronunciations (as opposed to the speech script), number of pauses, and number of phonemes uttered per second; (2) the speaker representation extracted by concatenating the pre-trained x-vector and ECAPA-TDNN embeddings [51]; and (3) prosodic features, such as the low-level descriptors of the F0 contour.

A Linear Discriminant Analysis (LDA) classifier is applied on top of each of the feature sets for classification. The results achieved by these features are reported in Table VI. Among the three feature sets, speaker embeddings appear to be the most crucial features for all datasets, corroborating with Task-1 results where the GANs suffered the most severe degradation, where the original speaker embeddings were entirely substituted. Such finding also suggests that speaker-unique attributes and health-related information are highly entangled in the speaker embeddings. Considering that existing anonymization systems rely heavily on these off-the-shelf speaker embeddings, it remains challenging to preserve the health information while altering only the speaker identifier.

While a group of studies reported prosody as a key biomarker to characterize speech disorders, such as dysarthria [82, 83, 84], our results show that phoneme-level linguistic features outperform prosodic features for COVID-19 detection. Specifically, we found the number of pauses and number of mispronunciations to be the most important phoneme-level features, with COVID-positive samples demonstrating more mispronunciations and fewer pauses. While the correlation between phoneme-level features and COVID-19 status has not been systematically studied, similar features have been examined for other diseases affecting speech production. For example, [85] shows that individuals with Parkinson’s disease produced fewer pauses at syntactic boundaries; the statistics of pauses have been shown crucial for diagnosing neuromuscular disorders, such as dysarthria [86]. Since GAN-based systems left linguistic content intact during anonymization, these findings help explain why the diagnostic models could perform above chance-level even when only the phoneme sequences were preserved during anonymization.

To better understand the impact of different anonymization methods on speech characteristics, we first visualize the waveform of the speech processed by the three anonymizers (see Fig. 10) for a direct comparison. As can be seen, those processed by the McAdams anonymizer and Ling-Pros-GAN share higher similarities in the waveform envelope shape with the original signal compared to the one generated by Ling-GAN. The difference seen in the plot is in line with the architecture design of different anonymizers. Among the three, Ling-GAN loses prosody and most of the speaker attributes, hence is expected to cause the highest amount of changes in the anonymized speech. The Ling-Pros-GAN and McAdams anonymizer, in turn, leave the speech rhythm untouched (i.e., duration and energy of phonemes), hence leading to higher resemblance in the waveform envelope.

Next, t-SNE plots are used to visualize the distribution of the speech features in two dimensions. Figure 11 shows the clusters of speech anonymized with different methods (computed from the training and validation data) and for the three features modalities explored herein: openSMILE (subplots a), MSR (b), and logmelspec (c). As can be seen, for all three feature sets, the distribution of clean speech (blue) is closer to that of the McAdams anonymized speech (orange) and Ling-Pros-GAN anonymized speech (red), while the Ling-GAN anonymized speech (green) shows the least similarity with the other two, corroborating findings from Tasks 1 and 2.

Moreover, it can be seen from Figure 11a and Figure 11b that the clusters computed from openSMILE and MSR features show little overlap, while clusters of the logmelspec features show great overlap (Figure 11c). Together with Task-2 results, this shift in the feature space is likely the main cause of the higher decrease observed in the openSMILE and MSR systems under different anonymization settings. Meanwhile, since all anonymization methods keep the speech content intact and change only the nonverbal attributes, a greater shift in feature space may indicate a stronger correlation with the para-linguistic aspect and less with the linguistic aspect. This echoes with previous studies which showed that openSMILE and MSR features are preferred over logmelspec features in characterizing emotional and unnatural speech [87, 88].

Lastly, we investigate the impact of using anonymized external data for data augmentation and see its impact on the performance achieved with scenarios B and C. With scenario C, we chose sub-condition C1 and C3. To quantify the relative improvement, we used the within-dataset performance achieved in scenario A as the baseline and calculated the amount of performance increase seen (in percentage). The relative changes observed with the three diagnostics systems are reported in Table VII. Here, we explore augmentation with two different datasets and with four different methods: original, McAdams, Ling-GAN, and Ling-Pros-GAN. As can be seen, when test data are anonymized using the McAdams coefficient (B1), the highest improvement is generally achieved when the diagnostic system is augmented with the original data. In turn, when the test data are anonymized using the GAN-based method (B2), augmenting the set with GAN-anonymized data from another dataset leads to a higher increase. Similar results are shown in scenarios C1 and C2, where clean and GAN-anonymized augment data result in more significant improvements. While not the top-performer, the McAdams method is shown to be a reliable augmentation strategy, especially for the openSMILE features. Overall, these findings suggest that anonymization has the potential to be used as a data augmentation approach to improve COVID-19 diagnostics accuracy when tested on anonymized data.

The study’s principal aim was to validate the effectiveness of anonymization methods within and across datasets in the context of assessing voice-based COVID-19 diagnostic accuracy. While we investigated three anonymization methods, other methods are emerging continuously (e.g., [89, 90]); thus, the findings reported herein should be validated with more recent methods. In the present study, the ASR anonymization method developed on English speech was applied to the multilingual CSS dataset. The finding that GAN-based anonymization had the lowest cross-dataset performance results may suggest challenges in applying this method in multilingual datasets and non-English speaking populations. In the future, multilingual GANs should be explored to avoid unfair outcomes [91] due to certain languages or cultural settings being excluded from the training and testing datasets. Moreover, while the injection of anonymized external data showed to be a useful data augmentation strategy, the final results were still at times lower than those achieved in the classical “unprotected” setting. This suggests that health-related information is being discarded during the anonymization process, thus future work could explore the development of diagnostic-aware anonymization methods that keep such discriminatory information intact.

Beyond tackling these limitations mentioned above, future work into voice-based diagnostics should be mindful of potential biases during data collection that could lead to confounds for both the anonymization and diagnostic steps. These confounders, if not properly dealt with, can reinforce the systemic nature of biases, for instance in relation to gender and racioethnic groups, that already exist within the healthcare system, thus transferring them to automated diagnostic systems. While [35] already showed some impact of sampling rate on diagnostic accuracy, several other potential biases may exist at the methodological level. For example, sociodemographic biases may emerge if age is not taken into consideration, as cognitive limitations (e.g., difficulty in speech planning or lexical access) associated with aging could alter speech patterns that could affect overall diagnostic accuracy. Recent work has shown that socioeconomic status could serve as a bias in COVID-19 detection [92]. For example, as data was collected from participants from home, those living in crowded conditions could have resulted in increased background noise levels that negatively affected anonymization and diagnostic efficacy. Moreover, disadvantaged populations have been shown to have more chronic respiratory diseases [93] and higher levels of mood disorders and psychological distress [94]. As such, anonymization processes affecting para-linguistic features associated with depressed mood may disproportionately affect those with a low socioeconomic status. Addressing biases in automated voice anonymization and diagnosis systems is beyond the scope of this paper and is left for future studies.