A Benchmark for Multi-speaker Anonymization

Following VPC baseline models, SSA approaches can be categorized into digital signal processing- and disentanglement-based methods. Digital signal processing methods perceptively modify speech to conceal the original speaker’s identity. For instance, VPC provided one baseline where the speaker attributes are altered using the McAdams coefficient to shift the spectral envelope [15, 16, 17]. However, this approach may distort the content, even though the speaker’s privacy is being protected, making it less effective than disentanglement-based methods [15, 16, 17].

Two hypotheses underlie disentanglement-based approaches. First, speech can be explicitly decomposed into content, speaker identity, and prosodic representations, where speaker identity is a time-invariant representation across the utterance, while content and prosody are time-varying. This enables the modification of content and other paralinguistic attributes independently. Second, the representation of speaker identity carries most of the speaker’s private information; modifying the original speaker representation can hide the majority of speaker identity information. The modified speaker embeddings are finally combined with the original content and prosody representations to generate anonymized speech.

Because disentanglement-based SSAs show superior effectiveness in preserving speaker privacy and maintaining utility, the majority of speaker anonymization studies have adopted a similar disentanglement-based framework, with improvements made from several perspectives.

Several works argue that the disentanglement of linguistic information extracted from the content encoder and prosody still contains original speaker information. To address this, modifications are made to the content and prosody. For instance, [23] removed speaker information from pitch and linguistic information by introducing differentially private noise. [24] modified pitch and speech duration to remove the speaker identity information residual in pitch contours and their dynamics. [25, 26] enhanced the disentanglement by removing speaker information from the acoustic model using vector quantization. [35] proposed a neural audio codec-based approach, known for generating high-quality synthetic speech when combined with language models, effectively bottlenecking speaker-related information to enhance privacy protection.

A widely-used selection-based speaker anonymizer [20, 21] replaced an original speaker vector with a mean vector (pseudo-speaker vector) of a set of randomly-selected speaker vectors from an external pool of English speakers. Previous research [15, 16] has demonstrated that anonymized voices generated by selection-based anonymizers have limited variability due to the average operation on speaker vectors. There have been recent attempts to preserve speaker distinctiveness. One of the top SSA systems [27] submitted to VPC 2022 utilized a generative adversarial network (GAN) to generate artificial speaker embeddings. This GAN was trained to turn random noise into artificial speaker embeddings that follow a similar distribution as the original speaker vectors. Another interesting work [28] is based on an orthogonal householder neural network (OHNN), which rotates the original speaker vectors into anonymized speaker vectors while ensuring they maintain the distribution over the original speaker vector space to preserve the naturalness of the original speech. The most recent work [29] models a matrix related to speaker identity and transforms it into an anonymized singular value transformation-assisted matrix to conceal the original speaker identity, generating more natural and distinctive anonymized speech.

SSA, as defined by VPC, focuses solely on removing speaker identity from the original speech while leaving the linguistic content and other paralinguistic attributes unchanged like age, gender, emotion, and dialect. However, these paralinguistic attributes, which also contain sensitive personal information, could potentially disclose a speaker’s privacy, including geographical background, social identity, and health status [30, 31]. Researchers have explored techniques aimed at protecting voice privacy by concealing various privacy-related characteristics, such as age, gender, and accent from speech signals.

Another aspect of research aims at a SSA solution that can be applied to speech for unseen language. Self-supervised learning (SSL)-based speaker anonymization has been proposed [32, 28, 30], which utilizes an SSL-based content encoder to extract general context representations regardless of the input speech language. The entire system requires no text labels or other language-specific resources, enabling the system to anonymize speech data from unseen languages.

Applying SSA techniques directly in conversations would convert multi-speaker interactions into a single pseudo-speaker, thereby completely erasing the turn-taking information crucial for MSA. However, most SSAs can be extended to handle the multi-speaker conditions with additional components. In this work, we select SSL-based SSA as the backbone, as it has been verified to maintain good intelligibility, naturalness, and applicability across multiple languages [32, 33, 28]. This will be detailed in Section IV-B.

To evaluate the effectiveness of speaker privacy protection in anonymized speech, we take the viewpoint of the attacker who uses an ASV model ($ASV_{\text{eval}}$) to guess the speaker identity from the anonymized speech. Given an unanonymized (original) reference utterance from a targeted user, the attacker uses the ASV model to measure how similar an anonymized utterance is to the reference in terms of speaker identity. The similarity (i.e., the ASV score) should be low for a well anonymized utterance. However, if there is some leakage of the speaker identity in the anonymized utterance, the ASV score may be higher than the ASV threshold, leading the attacker to accept the hypothesis that the anonymized utterance and the reference are uttered from the same speaker. The ‘success’ rate of the attacker’s guess is equivalent to the ASV false accept rate (FAR). The pair of an anonymized utterance and unanonymized reference is considered to be negative data, while that of an unanonymized utterance and reference from the same speaker is considered to be positive data. Similar metrics have been used in other security fields, e.g., membership inference attack [36, 37].

To further explain the FAR, we first define three types of enrollment and test pairs that are used to compute the FAR. Given $M$ original (O) and anonymized (A) conversations, $C_{o}^{m}$ is the $m$-th original conversation with ${N_{m}}$ speakers and $m\in[1,M]$. $\mathbf{x}_{\text{o}}^{mn}$ is the aggregated single-speaker segment for speaker $n$ in original conversation $C_{o}^{m}$. Similarly, $C_{a}^{m}$ is the $m$-th anonymized conversation. $\mathbf{x}_{\text{a}}^{mn}$ is the aggregated single-speaker segment for speaker $n$ in anonymized conversation $C_{a}^{m}$.

• •

  O-O positive pairs: $\mathbf{x}_{\text{o}}^{mn}$ is split in half, denoted by $\mathbf{x}_{\text{o}}^{mn(1)}$ and $\mathbf{x}_{\text{o}}^{mn(2)}$, to form positive pairs for each conversation, and then traverses all conversations. This traversal process uses the union of sets $\bigcup$ to encompass all positive pairs:

  $$P_{\text{positive}}=\bigcup_{m=1}^{M}\bigcup_{n=1}^{N_{m}}\{(\mathbf{x}_{\text{o}}^{mn(1)},\mathbf{x}_{\text{o}}^{mn(2)})\},$$

  (1)

• •

  O-O negative pairs: different speaker segments from each conversation that traverse all conversations form negative pairs :

  $$P_{\text{negative}}=\bigcup_{m=1}^{M}\bigcup_{\begin{subarray}{c}n=1\\ k\neq n\end{subarray}}^{N_{m}}\{(\mathbf{x}_{\text{o}}^{mn},\mathbf{x}_{\text{o}}^{mk})\}.$$

  (2)

• •

  O-A pairs: the original single-speaker segment and its corresponding anonymized segment form as O-A pairs:

  $$P_{\text{O-A}}=\bigcup_{m=1}^{M}\bigcup_{n=1}^{N_{m}}\{(\mathbf{x}_{\text{o}}^{mn},\mathbf{x}_{\text{a}}^{mn})\}.$$

  (3)

$ASV_{\text{eval}}$ is then utilized to select a threshold corresponding to the equal error rate (EER), where the FAR and the false rejection rate (FRR) are equivalent, using $P_{\text{positive}}$ and $P_{\text{negative}}$ from the original conversation. Subsequently, $P_{\text{O-A}}$ is input to $ASV_{\text{eval}}$ to calculate cosine similarity and compute the FAR. This FAR is determined as the number of false acceptances (under the threshold identified using original segments) divided by the total number of $P_{\text{O-A}}$.

A lower FAR indicates that $ASV_{\text{eval}}$ identifies anonymized test segments as dissimilar to their corresponding original enrollment segments, suggesting that most anonymized conversations conceal speaker identities, thereby safeguarding the speakers’ privacy.

All the MSAs examined in this work are based on cascaded structures, utilizing SD and SSL-based speaker anonymization with various speaker anonymizers, all of which are well-pretrained models. The proposed conversation-level speaker anonymizer is a selection procedure with specific conditions, eliminating the need for additional training.

For SD, we apply the efficient and robust spectrum clustering-based approach, which is implemented with the WeSpeaker Toolkit ⁶⁶6https://github.com/wenet-e2e/wespeaker/tree/master [49]. It first performs Silero-VAD ⁷⁷7https://github.com/snakers4/silero-vad to remove the silent segments, then splits the long audio into the 1.5-second segments. Note that these segments have 0.75 seconds of overlapping. Each segment is fed into the speaker recognition model, which is a context-aware masking-based structure [50] pre-trained on the VoxCeleb2 dataset [51]. After that, spectral clustering [39, 40] is performed to aggregate the segments into several speakers by analyzing the similarity metrics. The diarization results are saved into the RTTM file, which summarizes the timestamp of each speaker’s active speech segments.

Table II lists the notations for the different MSA approaches that were examined. $A_{OHNN}$ and $A_{Select}$ are the cascaded MSAs using speaker-level speaker anonymizers designed for SSA. $A_{DS}$ and $A_{AS}$ are those using the proposed conversation-level speaker anonymizers. Specifically, all approaches share the same backbone, utilizing the YAAPT algorithm [46] to extract the fundamental frequency (F0); the ECAPA-TDNN architecture, with 512 channels in the convolutional frame layers [47], provides 192-dimensional speaker identity representations; the HuBERT-based soft content encoder [52] uses a convolutional neural network (CNN) encoder along with the first and sixth transformer layers from the pre-trained HuBERT base model. It downsamples a raw audio signal into a continuous 768-dimensional representation, subsequently mapped to a 200-dimensional vector using a projection layer to predict discrete speech units. These units are derived by discretizing the intermediate 768-dimensional representations through k-means clustering⁸⁸8https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit [53, 54]. The configuration of HiFi-GAN is consistent with [53]. Additional training procedures are detailed in [32].

$A_{OHNN}$, the state-of-the-art speaker-level speaker anonymizer that achieves good speaker distinctiveness for single-speaker utterances, uses an OHNN-based anonymizer trained on authentic VoxCeleb2, utilizing random orthogonal Householder reflections with a random seed of 50 for parameter initialization. An additive loss function is used, combining weighted angular margin softmax and cosine similarity. Training details are available in [28].

$A_{Select}$, the commonly-used speaker anonymizer, uses a selection-based strategy that identifies the 200 farthest same-gender speaker vectors from an external speaker vector pool (specifically LibriTTS-train-other-500 [55]), subsequently averaging 10 randomly-selected ones as the pseudo-speaker vector⁹⁹910 vectors instead of the default setting of 100 used in VPC, as it has been demonstrated that averaging 100 (large number) vectors significantly reduces the distinctiveness of anonymized speakers [17]..

Both $A_{AS}$ and $A_{DS}$ are based on the same external speaker vector pool as $A_{Select}$. $L_{\text{far}}=200$, $L_{\text{prune}}=10,000$. The gender recognition model used before selecting an anonymized speaker vector from the pool is a fine-tuned version of wav2vec2-xls-r-300m¹⁰¹⁰10https://huggingface.co/facebook/wav2vec2-xls-r-300m on Librispeech-clean-100. It achieves 99% accuracy on the LibriSpeech test-clean subset.

$ASV_{\text{eval}}$ is the publicly available ECAPA-TDNN model¹¹¹¹11https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb, trained on VoxCeleb1 [60] and 2 [51]. $ASR_{\text{eval}}$ is a model fine-tuned on LibriSpeech-train-960 [56] from wav2vec2-large-960h-lv60-self¹²¹²12https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self, using a SpeechBrain [61] recipe¹³¹³13https://huggingface.co/speechbrain/asr-wav2vec2-librispeech. $SD_{\text{eval}}$ is the same speaker diarization model used in MSA. For $MOS_{\text{eval}}$ [38], is a model fine-tuned on the Blizzard Challenge for TTS [62] and the Voice Conversion Challenge [63] from wav2vec2-base¹⁴¹⁴14https://huggingface.co/facebook/wav2vec2-base by mean-pooling the model’s output embeddings, adding a linear output layer, and training with L1 loss. We utilized the predicted MOS instead of human perception-based MOS from listening tests due to time and cost constraints. The predicted MOS is reasonably well-aligned with human perception [38]. In our previous work [28], we demonstrated that the ranking of the predicted MOS of original and anonymized speech, generated by different speaker anonymization systems, is consistent with those from listening tests conducted by VPC [15]. This observation holds for clean datasets, and therefore we only computed PMOS for clean simulated datasets in the following experiments.

Note that except for the DER computation, where $SD_{\text{eval}}$ takes the conversation as input, the other metrics’ computations require RTTM to split the original and anonymized conversation into single-speaker segments. In real MSA applications, we assume there is no RTTM available. MSA produces predicted RTTM using the SD model and then uses this RTTM to split and reconstruct the audio. In our experiments, we also provide the results using real RTTM (ground truth) as the upper baseline.

First, we examine whether $A_{AS}$ and $A_{DS}$ learn as the optimization objectives. Fig. 4 plots the sum of similarities for all speaker pair combinations per conversation, with each pair consisting of two different speakers, for the original data, $A_{AS}$, and $A_{DS}$ using predicted RTTM on clean simulation datasets. Overall, the sum of similarities for $A_{AS}$ is the lowest across all scenarios with different numbers of speakers, ensuring better speaker distinctiveness of anonymized speakers. $A_{DS}$ mimics the original conversation distribution, and the sum of similarity for each conversation matches that of the original conversation. However, as the number of speakers increases, it becomes more difficult to obtain a similar sum, resulting in more fluctuations and outliers.

Tables X and XI list the FARs and DERs for the VoxConverse test dataset, respectively¹⁵¹⁵15We omit WER and PMOS computations since VoxConverse dataset lacks a text transcript and includes real-world noise.. The trends of FARs and DERs for both original and anonymized speech generated with different MSA systems are remarkably similar to those observed on the simulation test sets. Specifically, using $A_{DS}$ and $A_{AS}$ achieves less than 1% FARs, showing almost perfect privacy protection ability. For DERs, $A_{AS}$ achieves about 13.7% whether predicted or real RTTM is used, and is close to the original conversation, which achieved 10%, showing good speaker distinctiveness preservation.