Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling

In our previous work, we proposed an ASV-model-free anonymization framework [14]. This framework consists of a feature extraction module, an acoustic model and a vocoder. The feature extraction module extracts F0, bottleneck feature (BN), and speaker ID from input speech by the YIN algorithm [15], WeNet toolkit [16], and look-up-table (LUT), respectively. Speaker ID is utilized to generate speaker embedding by a speaker encoder and controls the speaker identity. An encoder-decoder architecture is utilized as the acoustic model. We use the CBHG module [17], which does not contain an attention mechanism, as the encoder because the BN and the acoustic features are initially aligned. The decoder utilizes an autoregressive model to improve the quality and intelligibility of the reconstructed Mel-spectrogram. As for the vocoder, we modify HifiGAN with multi-band processing [18, 19]. The modified HifiGAN generator takes Mel-spectrograms as input and produces sub-band signals instead of full-band signals. In the upsampling module, 4 sub-bands are predicted simultaneously through 3 upsampling layers with 2x, 5x and 5x factors respectively, resulting in a 200x upsampling finally.

As for the anonymization strategy, the previous framework combines two types of speaker embeddings which are generated by the speaker encoder. The first one is a pseudo speaker embedding generated using a reserved pseudo speaker ID and does not correspond to any real speaker. The other one is an averaged embedding which averages the randomly selected speaker embeddings. The averaged speaker embedding and the pseudo speaker embedding are weighted concatenate to generate the anonymized speaker embedding. The averaged speaker embedding guarantees that each trial utterance from different speakers is anonymized by different anonymized speaker embeddings, while all trial utterances from a given speaker are anonymized by the same anonymized speaker embedding.

Although our previous anonymization framework ranked first place in all conditions of the VoicePrivacy 2022 Challenge, there is still an apparent drawback compared with a satisfactory anonymization system, that is, distinguishability performance is poor. The main reason is that each speaker has their own voice distinctiveness and the averaged speaker embedding introduced by the anonymized embedding may severely erode such distinctiveness. What’s more, the capacity of the previous framework to produce distinctive voices is also limited by the size of the look-up table and the number of speakers in the datasets.

Our proposed framework consists of three major parts: (1) a feature extraction module, (2) an acoustic model, and (3) a vocoder. A speech signal is anonymized by three steps. First, the feature extraction module extracts BN, formant and F0 from the original speech signal. Then, the acoustic model predicts anonymized Mel-spectrogram by BN, scaled formant and F0. Finally, the vocoder reconstructs the anonymized Mel-spectrogram to signal.

Feature extraction. We use the PRAAT toolkit ¹¹1https://github.com/praat/praat to extract F1-F5 formants and F0. The purpose of the ASR encoder is to extract speaker-invariant representation. The same tool, WeNet [16], is used to extract BN as in our previous work.

Acoustic model. In addition to using the same CBHG encoder and autoregressive decoder as in our previous framework, the formant encoder and F0 encoder are also added to the acoustic model. The formant encoder and F0 encoder architectures are shown in Fig. 2 (a) and (b), they have the same architecture of a stack of two convolutional neural networks with a kernel size of 3. Differently, the output of the formant encoder is the distributions of the formant, while the high level representations are the output of the F0 encoder. A dropout layer is also utilized at the beginning of the convolutional block to make the training process more stable. We adopt the L1 loss as the reconstruction loss to optimize the predicted Mel-spectrogram.

Vocoder. Since the output of the acoustic model is the same as the previous framework, the same vocoder and training losses are used as our previous framework.

Since most voice information is preserved in the F0 and formant frequency ranges [20], whereas linguistic information is preserved in the relative formant frequency ratios [21]. Therefore, uniformly scaling the formant and F0 is able to suppress the original speaker information while retaining the linguistic information.

We devise two anonymization strategies for scaling the formant and F0 of the source waveform, which can effectively preserve privacy and speaker distinguishability. The first strategy is gender-independent scaling, which scales the formant and F0 using the same factor directly for different genders. Suppose $\alpha$ is the scaling factor, the formant and F0 from the source speech are denoted as $\mathbf{f}_{\texttt{src}}$ and $\mathbf{p}_{\texttt{src}}$, respectively. The first scaling strategy is as follows:

$$\mathbf{f}_{\texttt{anon}}=\alpha\times\mathbf{f}_{\texttt{src}},$$

(1)

$$\mathbf{p}_{\texttt{anon}}=\alpha\times\mathbf{p}_{\texttt{src}},$$

(2)

here, $\mathbf{f}_{\texttt{anon}}$ and $\mathbf{p}_{\texttt{anon}}$ are the anonymized formant and F0, whose original speaker identity is suppressed.

The second strategy is gender-dependent scaling, which takes the differences between male and female in formant frequency and F0 into consideration. In general, female has higher formant frequency and F0 than male [22], so we use different scaling factors to adjust the source formant and F0 for different genders as follows:

$$\mathbf{f}_{\texttt{anon}}=\left\{\begin{array}[]{lr}(1+\alpha)\times\mathbf{f}_{\texttt{src}},\quad if\ \ male,&\\ (1-\alpha)\times\mathbf{f}_{\texttt{src}},\quad if\ \ female,&\end{array}\right.$$

(3)

$$\mathbf{p}_{\texttt{anon}}=\left\{\begin{array}[]{lr}(1+\alpha)\times\mathbf{p}_{\texttt{src}},\quad if\ \ male,&\\ (1-\alpha)\times\mathbf{p}_{\texttt{src}},\quad if\ \ female,&\end{array}\right.$$

(4)

The formant and F0 of the male speakers are enlarged by a factor of $(1+\alpha)$, and the female speakers are reduced by a factor of $(1-\alpha)$. These factors enable the formant and F0 are not adjusted beyond the minimum or maximum range.

We use the same datasets as we employed in the VoicePrivacy 2022 Challenge [4]. That is, the ASR model is trained on LibriSpeech-clean-100 and LibriSpeech-other-500 datasets [23], while the LibriTTS-clean-100 and LibriTTS-other-500 datasets [24] are used to train the acoustic model and vocoder. The evaluation of the systems is performed on VoicePrivacy 2022 challenge LibriSpeech-test and VCTK-test datasets [25], whose details are described in [4].

Following the VoicePrivacy 2022 challenge configuration, the privacy-utility metrics are the equal error rate (EER) and word error rate (WER), which rely on the official ASV and ASR systems²²2https://github.com/Voice-Privacy-Challenge/Voice-Privacy-Challenge-2022. The higher EER represents better privacy protection and the lower WER represents better intelligibility. It is essential to emphasize that anonymization should not only lead to the speaker being de-identifiable but also preserve intonation and speaker distinguishability. Thus, the intonation is measured using the Pearson correlation [26] between the pitch sequences of original and anonymized utterances, denoted as $\rho^{F0}$. The range of pitch correlation is from 0 to 1, and the higher is the better. The gain of voice distinctiveness ($G_{vd}$) metric is used to measure the preservation of speaker distinguishability, which relies on voice similarity matrices [27]. The voice distinctiveness remains the same when $G_{vd}=0$ and an average increase or decrease in voice distinctiveness are indicated by a gain above or below 0.

Two anonymization strategies are proposed in Sec 3.2. In this subsection, we investigate the effect of scaling factor for the different formant and F0 scaling strategies, respectively. The first one is to test the validity of anonymization using the same scaling factor for different genders, and the second is to test the use of different scaling factors for different genders.