Unsupervised Data Selection via Discrete Speech Representation for ASR

Data selection improves ASR performance and data efficiency by identifying the most informative training examples. Concretely, the task of data selection for self-supervised learning is, given labeled data from a target domain $T$ and unlabeled data from a general pool $G$, selecting the best subset of $G$ for self-supervised pre-training, such that the model pre-trained on that subset and then fine-tuned on $T$ achieves the best performance for the target domain. We assume that $G$ is much larger than $T$.

Confidence filtering is a classic method in data selection for ASR [28, 29], and has been successfully applied to end-to-end models [12, 30, 31]. However, confidence methods focus on data of good transcript quality, which might not be useful for self-supervised pre-training. Moreover, training a confidence model requires supervised data, which is not often feasible.  [19] proposes to do frame-level data selection and utterance-level reweighting, which is complementary to our task.

We propose to do data selection by applying contrastive data selection on discrete speech representations. It selects data that is acoustically similar to the target domain, and does not require any labeled data in the loop.

An emergent trend in self-supervised learning for speech is to learn discrete tokens from the continuous speech signals [32, 33]. The discrete representations make it applicable to NLP methods that require discrete inputs, for example BERT-style pre-training algorithms. Empirically, it leads to better results compared to without quantization [4, 3].

A quantizer maps continuous features into discrete tokens from a learnt codebook. It can be placed at different depths of the network, which leads to latent codes of different semantic level. In wav2vec 2.0 family of models [3, 5], the quantization is immediately after the feature encoder and before any transformer/Conformer representation learning layers. Therefore the discrete tokens is of low semantic level, and could contain information like pitch, background noise, and other cofounding details of the audio signal. On the other hand, in the generative VQ-VAE, the quantization is at the top of all representation layers, where the discrete code is more abstract.  [32] shows the VQ-VAE quantized code is predictive of the phonetic content of the utterances. Moreover, the discrete token is used differently in the training objectives. W2v-BERT uses the token ID as the target label in the cross-entropy loss of the masked language model task; VQ-VAE uses the discrete token to look up a 1-of-K embedding vector, which is then used to reconstruct the spectrogram. It’s an open research question on how the quality of the quantization affects the self-supervised learning [20]. In this work, we experiment with both w2v-BERT and VQ-VAE discrete token sequences as input to the contrastive data selection module described next.

Contrastive data selection selects examples from a source corpus that are matched to a target domain. It was originally proposed on text data [21], and has since been successfully applied in other domains such as machine translation and ASR. We extend this method to work on discrete speech tokens.

More specifically, we train two language models (LM) on corpora of the target domain $T$ and of general domain $G$ using the discrete speech tokens as input. For each utterance, we compute the log probability difference [21, 26] between the two LMs normalized by the number of tokens in the utterance. Assume $\bm{q}$ is the vector quantized representation of the utterance, the domain relevance score is $\frac{\log P_{T}(\bm{q})-\log P_{G}(\bm{q})}{\text{length}(\bm{q})}$, where $P_{T}$ and $P_{G}$ are the probabilities of the target and general domain LM respectively. We select utterances with the top domain relevance scores. See Fig. 1 for an illustration.

In the data selection for pre-training experiments, we use LibriSpeech and MLS as the target dataset, and YT-U as the general pool. We first pre-train the encoder on YT-U and then fine-tune on the LibriSpeech or MLS labeled set. The quantizer for speech tokenization is trained on YT-U. In the data selection for supervised learning experiments, we use WSJ and CHiME-6 as the target datasets, and the People’s Speech dataset as the general pool. We add the selected People’s Speech data to the in-domain training set for supervised learning. We reuse a quantizer trained on SpeechStew for these experiments. See Table 1 summarizes the datasets in each experiment.

In the pre-training experiment, we use both the w2v-BERT [5] and the VQ-VAE [8] as the quantizer. In the supervised learning experiment, we use a VQ-VAE quantizer. In w2v-BERT quantizer, the codebook vocab size is 1024 and the codebook dimension is 1024. In VQ-VAE, the vocab size is 8192 and the dimension is 16. For both models, we apply Gumbel-Softmax technique [41] to the quantization operation, which makes the argmax differentiable through a temperature parameter in the backward pass. The temperature is gradually annealed towards a small but non-zero value during training.

We use the same set of contrastive data selection parameters across all experiments. We build $N$-gram language model (LM) with $N=5$ and back-off. We use the Kneser-Ney LM interpolation algorithm [42] to estimate LM probabilities. We use a few hundred to a few thousand hours of data as input. In the pre-training experiments, we use the in-domain training set for the target LM, and a random subsample of a few hundred hours YT-U for the general LM. In the supervised experiment, we use WSJ or CHiME-6 training set to build the target LM, and the People’s Speech clean set to build the general LM. As we will show in the experiment, the method is not sensitive to the amount of data in LM training.

In all our experiments, we use 80-dimensional log-mel filter bank coefficients as the acoustic inputs, computed with a 25ms window and shifted every 10ms. For transcript tokenization, we use a 1024-token WordPiece model [43] for all English experiments, and 4096-token WordPiece models for French, German, and Spanish experiments.

In the pre-training experiments, we follow the w2v-BERT XL setup in [5] which is a 24 layer Conformer. The feature encoder has two 2D-convolution layers with strides (2,2). We use the 12-th Conformer block hidden features to compute the contrastive loss and the last (24-th) layer features to compute the masked prediction loss. We use a RNN-T model for fine-tuning. The decoder is a two-layer LSTM with a hidden dimension of 640. The model has 600 million parameters. In the supervised learning experiments, we use a 17 layer Conformer model of 119 million parameters, the same as ConformerL in [44].

In the pre-training experiments, we use w2v-BERT as the pre-training recipe. We use a masking length of 400ms with masking probability of 0.065. We use Adam optimizer [45], and a transformer learning rate schedule [46] with 1e-3 peak learning rate and 25,000 warm-up steps. The batch size is 4096. In the fine-tuning, we use different learning schedules for the pre-trained encoder and the decoder. The encoder has a peak learning rate of 3e-4 and 5,000 warm-up steps, while decoder has 1e-3 and 1,500 steps. The batch size is 256.

For the supervised learning experiment, we use Adam optimizer with a chedule of 0.002 peak learning rate and 10,000 warm-up steps. The batch size is 256 for WSJ experiment, and 2048 for CHiME-6 experiment. We find a larger batch size is essential for CHiME-6, otherwise the model may fail to train.

Table 2 compares WERs on LibriSpeech dev and test sets when the models are pre-trained on YT-U-En with and without data selection, and then fine-tuned on LibriSpeech 960h or 100h. We compare with state-of-the-art performance from [5] in the first row, where the pre-training data is the in-domain Libri-light 60k hours. For our models, we pre-train the encoder for up to 800k steps. Since there is no golden rule in picking pre-training checkpoints, we fine-tune at every 100k checkpoints. We pick the best one in terms of fine-tuning WERs on dev sets for each method to report. Comparing the YT-U with YT-U select row, we reduce WER by 5% and 11% relative on the test clean and other set respectively, and reduce the amount of pre-training data to 6%. In the last row, by combining the selected YT-U data with Libri-light, we are able to slightly improve the state-of-the-art WER on the test-other set.

Table 3 shows WERs on MLS French, German, Spanish test sets. For our monolingual models, the pre-training data is YT-U and the fine-tuning data is MLS-full. To the best of our knowledge, the state-of-the-art (SoTA) WERs on these test sets are achieved by multilingual models in [27, 47], as detailed in the first row. We pre-train for 100k steps for fast experimentation. Comparing YT-U with YT-U select, the WER reduction is 18%, 15%, and 26% for French, German and Spanish. And the amount of data is less than 6%.

In Table 4 we compare the proposed selection method with random subsample and confidence filtering. We provide WERs with no data selection for reference. For the confidence filtering method, we use a strong RNN-T model of 183 million parameters trained from around 150k hours of labeled YouTube data. We interpret RNN-T loss as the confidence measure following [30]. All methods pre-train for 100k steps and fine-tune on 960h. Both confidence filtering and our method are better than no data selection. Note the confidence method requires labeled YouTube data. In contrast, our method does not require any paired data, yet still outperforms the confidence method.

We conduct ablation studies on the choice of the quantizer, and hyper-parameters of the contrastive selection method. For all variants in this section, the pre-training data is 60k select YT-U-En, and the fine-tuning data is LibriSpeech 960 hours. We pre-train for 100k steps, as the relative comparison stays the same for more steps.

Table 5 compares WERs when different quantizers are used to extract the discrete tokens for data selection. Both w2v-BERT and VQ-VAE quantizers perform similarly. The proposed method is not sensitive to the choice of the quantizer.

Table 6 compares WERs when different amount of data is used for LM training in the contrastive selection module. The quantizer is w2v-BERT. Our method is not sensitive to the amount of data used in LM training, once in a reasonable range.

While our work is mainly motivated to perform data selection for pre-training, the proposed method is also applicable to labeled data. We can select transcribed speech close to the target domain and add it to the training set to improve the ASR performance. This is useful for low resource domains.

To demonstrate the performance in this setting, we apply data selection to the People’s Speech dataset to improve ASR performance on WSJ and CHiME-6. We select around 800 hours of data from People’s Speech and add that to the in-domain training data for supervised learning. In Table 7, we compare three data selection methods: random sample, text contrastive LM selection, and our unsupervised method. We do not include confidence method because the ground-truth transcript is already given. We also present WERs when we use the original training set only from [40], and when we add the full 11.8k hours People’s Speech data.

Adding extra labeled data from People’s Speech greatly improves WERs on WSJ and CHiME-6. Our selection method outperforms random sample and text contrastive selection method. It reduce the amount of labeled data to 7%, and the WER is 0.1% worse compared to using the full People’s Speech data. This result shows that we can apply the proposed method to select unlabeled data to send for transcription, and achieves good WER with small amount of annotation.

Lastly, we compare the contrastive score ranking from quantized codes with contrastive score ranking from text. There is almost zero rank correlation between the two ordinal variables, measured by Kendall $\tau$ coefficient. It suggests that the two data selection methods can be complementary, and we leave empirical studies on this as future work.