Non-Parallel Voice Conversion for ASR Augmentation

It is critical for automatic speech recognition (ASR) to be robust to speaker differences. This is typically addressed by training on speech from a wide variety of speakers. In this paper, we show that despite being trained on  1000 hours of speech, augmenting speech with diverse speaker characteristics can improve speech recognition performance.

Voice conversion (VC) models convert input speech from its source speech to some target speaker, transferring the speaker timbre and other characteristics while retaining the lexical content of the source speech. While there has been successful approaches to voice conversion in recent years (Section 2), VC performance is typically evaluated on clean, in-domain speech. That is, the same style of speech used in training the VC model is used for evaluation. We have found that this can lead to VC systems that fail to generalize to speech and speakers that are substantially different than the training material (Section 5.3).

Since ASR training data is typically more diverse than VC training data, constructing a robust VC system is essential for ASR augmentation. ASR systems have to interact with speech from a variety of recording conditions and a broad range of speakers including speakers with less common accents.

In service of this robustness we make three important VC design choices. First, we pursue a non-parallel voice conversion approach (Section 3). By not requiring parallel data for training, we can train on much more data than we could otherwise. Second, we use a non-autoregressive decoder. We found that autoregressive models were susceptible to attention failures which were catastrophic for their application as a data augmentation technique. Third, we initialize the VC encoder with a pre-trained ASR encoder. This choice substantially improves the robustness of the VC model to diverse inputs (Section 5), while moderately limiting VC naturalness. For ASR augmentation, we find that this substantial improvement to robustness is more important than a marginal improvement to naturalness.

The main contribution of this paper are as follows:

• •

  We show that voice conversion can be used to successfully augment speech recognition training data. We find relative word error rate (WER) improvements up to  $6\%$ when augmenting LibriSpeech which contains 2,456 speakers.

• •

  We show that we can substantially improve the robustness of voice conversion systems to unseen corpora and different speaker characteristics by initializing the VC encoder with an ASR encoder.

• •

  We demonstrate VC metrics that can be used for model selection prior to subjective listening tests.

Non-parallel voice conversion can be broadly categorized into two major approaches, the first using Phone posteriogram (PPG) features [1, 2]. PPGs acts as speaker-invariant representations that can be easily utilized to achieve conversion. The other main approach is with auto-encoding style training approaches, utilizing both variational auto-encoder (VAE)[3, 4, 5, 6, 7, 8] and vector-quantized VAE (VQ-VAE) [9, 10, 11, 12] approaches. The auto-encoding style training approaches try to disentangle speaker speaker information from the content of source speech, using various methods from bottlenecking to adversarial training to prevent speaker leakage in the content or speech encoder. In [7], instead of adversarial training, carefully designed bottlenecks were coupled with auto-encoding loss to train the VC system. In [12], this is achieved by VQ-VAE of the speech encoder outputs.

Recent approaches have utilized pre-trained speech encoder to extract speaker agnostic representation. In [13], ASR trained to predict PPGs are used on source speech to achieve accent conversion. In [14], a trained ASR encoder is used as the speech encoder, along with explicit prosody modelling to achieve speaker and prosody in VC. In [15], accent and speaker disentanglement and control in VC was achieved in stages, where first stage is accent-agnostic ASR training, from which the trained speech encoder is used to further conduct VC training and disentangle accent and speaker.

Previous work on voice conversion and morphing augmentations to help augment ASR training include Vocal Tract Length Perturbation (VTLP) [16, 17]. More recent approaches use neural VC models to augment speech. In [18], a VC model is used to augment adult speech into children speech to help with children speech ASR. However in this work, the VC model was a Cycle-GAN[19] architecture trained on the same in-domain data on which the ASR training was also conducted with. In [20], a VC system was utilized as augmentation for low resource languages. This system utilizes representations from a pretrained encoder with CPC[21] objective, instead of one trained on ASR targets. The VC model augments speech in the very low data regime.

For our proposed VC model, we utilize an architecture consisting of an encoder, a bottleneck layer, and a non auto-regressive decoder. Figure 1 shows the architecture of the VC model.

Encoder: The VC encoder is similar to the encoder in the Conformer[22] ASR model. Specifically, the encoder consists of sub-sampling convolution layer that reduces the time dimension of the input speech by 4x. This is then followed by a stack of conformer layers (16 layers), followed by a output layer norm layer. We study two versions of encoders, the first being an conformer encoder that is trained from scratch jointly with the rest of the system. The second version being a frozen encoder, where the weights are obtained from an ASR model trained on 960 hours of Librispeech.

Speaker Embedding: Speaker information is provided in the form of one-hot vectors, which are passed through a learned speaker embedding that maps from one-hot vectors to a dense 256-dim embedding vector.

Decoder: The decoder is a non-autoregressive decoder composed of RNN layers followed by convolution layers. Speaker embedding is provided along with the encoder output to the decoder. Specifically, the dense speaker embedding vector is copied $T$ times across time and then concatenated together with encoder outputs to be fed to the decoder. The decoder consists of 2 bi-directional LSTM layers of size 1024 followed by a stack of convolution layers. During development, we also explored an auto-regressive, attention-based decoder. However, we found that during inference even within-domain speech would sometimes show attention failures leading to extremely poor conversion [23]. While there are ways to mitigate these failures, since robustness is necessary for ASR augmentation we pursued the less error-prone, non-autoregressive approach.

VQVAE Bottleneck: A Vector Quantized Variational Auto Encoding (VQVAE) Bottleneck [24] is used to quantize encoder outputs into discrete codebook entries. This restricts information through the VC system during auto-encoding training. The VQVAE bottleneck typically contain 3 loss terms, the reconstruction loss, codebook loss and commitment loss.

Reconstruction loss is computed on the decoder, thus we simply add codebook and commitment loss from VQVAE to the overall loss objective during training. We fix VQVAE codebook size to 128 and group size to be 2.

Adversarial Speaker Classification: To disentangle speaker information, we apply adversarial speaker classification loss on the VQVAE codebook outputs to further ensure no speaker leakage. Given encoder outputs $m_{enc}(x)$ and speaker labels $l_{speaker}$, we combine reverse gradient operation $R_{grad}$ along with softmax cross entropy loss as follows:

In effect, the training will try to maximize $L_{adv}$, which in effect will remove as much speaker information from the VQVAE output as possible. We scale adversarial gradient by weight, and find that this adversarial gradient weight term requires careful tuning, too small and it does nothing, too large and it overwhelms the reconstruction loss, see section 5.5 for details.

Objective Function We train on non-parallel data, and only utilize speech features and speaker embedding for the loss computation. To enable the VC models to be used as ASR data augmentation, we train our VC model to predict ASR log-mel features directly. During VC training, the same log-mel features are used as input and target. We use Huber loss for reconstruction loss. The model is jointly optimized by minimizing $L_{total}$, with all loss weights set to 1.0.

Frozen ASR Encoder In addition to training the full VC model (Figure 1) from scratch, we replace the encoder network with the encoder of an ASR model. ASR models implicitly normalize out speaker differences. ASR outputs are not (explicitly) dependent on speaker characterstics so they are less likely to be retained in the internal representations of the ASR network. We use an ASR encoder as the VC encoder, and freeze its weights, training only the decoder, VQVAE and speaker embedding. By freezing the encoder, it retains the representations learned during ASR training which leads to improved robustness.

VC Corpora: Three TTS corpora were used for training the VC model. The first LS is the publicly available LibriSpeech Corpus containing 960 Hours of speech with 2,456 speakers [25]. The second EN-VC is an in-house corpus containing professional English speakers from 6 English locales, including United States (en/us), Britain (en/gb), India (en/in), Singapore (en/sg), Nigeria (en/ng) and Australia (en/au). This corpus contains 58 speakers, totaling around 400 hours of studio quality speech intended for Text-to-speech (TTS) training and research and hence was recorded at 48KHz. The third corpus SV-VC is, similar to the second, an in-house Swedish (sv/se) dataset collected for TTS training and research, which contains 21 hours of across 6 speakers. The data distribution between speakers is skewed, with speaker hours ranging from 0.5 hours to 40 hours. All datasets are downsampled to 16kHz, and we use 80-dim mel spectrum as features, with a frame size of 25ms and a 10ms frame shift.

ASR Corpora: For ASR, three training datasets were used in this work. The first is LS the public LibriSpeech[25] corpus. The other two datasets are in-house data sets comprising of anonymized short utterances representative of Google’s voice search (VS) traffic in two languages, English EN-ASR and Swedish SV-ASR. While the choice of English allows us to analyze two very different domains, namely, audio books and voice search with similar amounts of training data( 1000 hours), the choice of Swedish( 6500 hours) allows us to explore some transfer effects. While the LS corpus used in VC and ASR training is identical, the in-house VC (EN-VC and SV-VC) and ASR corpora (EN-ASR and SV-ASR) are completely disjoint.

We demonstrate that even when trained on more than two thousand speakers, LibriSpeech ASR performance can be improved by Voice Conversion augmentation. We find similar ($\sim 6\%$) relative gains on an in-house English ASR task, but smaller improvements ($\sim 1\%$) on a comparable Swedish task. It is necessary for VC models to be robust to noisy inputs and diverse speakers to achieve these result. We show that using a pre-trained ASR Encoder to serve as the encoder in a non auto-regressive, non-parallel VC model is a successful approach to promoting the necessary VC robustness.