Guided contrastive self-supervised pre-training
for automatic speech recognition

The left part of Figure 1 gives an overview of conventional CPC representation learning approach. Given frames of audio features $\mathbf{x}_{t}\in\mathcal{X}$, we first apply the feature encoder network $f_{enc}:\mathcal{X}\mapsto\mathcal{Z}$ to map the input sequence to a sequence of latent feature representations $\mathbf{z}_{t}\in\mathcal{Z}=f_{enc}(\mathbf{x}_{t})$. An autoregressive context network $f_{ar}:\mathcal{Z}\mapsto\mathcal{C}$ summarizes all $\mathbf{z}_{\leq t}$ in the latent space and produces a contextual latent representation $\mathbf{c}_{t}=f_{ar}(\mathbf{z}_{\leq t})$

Both the feature encoder network and the autoregressive context network are trained to optimize the contrastive loss defined in Equation 1 based on Noise-Contrastive Estimation (NCE) [25] for each step $k$, which equivalently maximizes the mutual information between $\mathbf{c}_{t}$ and the latent representation $\mathbf{z}_{t+k}$ that is $k$ steps in the future [11].

where $\mathbf{\tilde{z}}$ is a set of negative samples sampled from the same audio example to represent the imposter distribution, $h_{k}(\mathbf{c}_{t})=W_{k}\mathbf{c}_{t}+\mathbf{b}_{k}$ is a step-specific affine transformation applied to $\mathbf{c}_{t}$ for each step $k$, and $\kappa$ is the temperature. We optimize the final contrastive loss $\mathcal{L}_{C}$ by averaging $\mathcal{L}_{k}$ over the next K steps:

CPC learns representations from the complete data distribution, which may not be optimal for the downstream ASR task. In this paper, we propose to provide weak guidance for the contrastive loss. This weak guidance is provided in the form of posteriors from a prior-knowledge model learned from a small labeled dataset, and we use a monophone classifier for experimentation in the paper. As shown in the right part of Figure 1, we use an additional encoder network $g_{enc}:\mathcal{P}\mapsto\mathcal{Q}$ to map the sequence of unnormalized posteriors (logits) $\mathbf{p}_{t}$ to a sequence of latent representations $\mathbf{q}_{t}\in\mathcal{Q}=g_{enc}(\mathbf{p}_{t})$, and then optimize the guided contrastive loss $\mathcal{L}_{C}^{guided}$ defined in Equation 4.

During training, parameters of the prior-knowledge model are fixed. We hypothesize that representations $\mathbf{c}_{t}$ learned through this new technique could capture more phone discriminative characteristics since optimizing the guided contrastive loss helps maximizing the mutual information between $\mathbf{c}_{t}$ and transformation of phone posteriors $\mathbf{q}_{t+k}$. Thus, $\mathbf{c}_{t}$ might be more aligned with the downstream ASR task and serve as a better initialization point.

We use an RNN-T [24] based ASR system for our experiments. The RNN-T model consists of an encoder, a prediction network and a joint network as shown in Figure 2(a). Let $\mathcal{D}=\{(\mathbf{X},\mathbf{Y})\}$ denote a single example from a training corpus where $\mathbf{X}=\{\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T}\}$ is a sequence of speech features and $\mathbf{Y}=\{y_{1},y_{2},...y_{U}\},y_{u}\in\mathcal{V}$ is a sequence of tokens from the vocabulary $\mathcal{V}$ (e.g. word pieces) representing the labels. The encoder maps each frame of the input speech features $\mathbf{x}_{t}$ to a hidden state $\mathbf{h}_{t}^{enc}$. The prediction network takes the embedding vector of the previous non-blank token $y_{u-1}$ and generates the hidden state $\mathbf{h}_{u}^{pred}$. The joint network is a feed-forward network that combines the outputs of the encoder and the prediction network to predict the conditional distribution over the next possible token $\tilde{y}_{i}\in\mathcal{V}\cup\langle blk\rangle$, where $\langle blk\rangle$ denotes the blank symbol. The RNN-T loss is computed by marginalizing over all possible blank-augmented token sequences $\mathbf{\tilde{Y}}=\{\tilde{y}_{1},\tilde{y}_{2},...,\tilde{y}_{T+U}\}$ aligned with each original token sequence $\mathbf{Y}$ and feature sequence $\mathbf{X}$:

where the index $i$ in $\mathbf{\tilde{Y}}$ is mapped to the index $u_{i}$ in $\mathbf{Y}$ and the index $t_{i}$ in $\mathbf{X}$.

After pre-training with the CPC or GCPC approach, The feature encoder network $f_{enc}$ and the autoregressive context network $f_{ar}$ are used to initialize encoder of the RNN-T ASR model in our experiments as shown in Figure 2(b). The remaining parts of the RNN-T model are randomly initialized before the final supervised training stage. When initializing the RNN-T encoder, we experiment with initializing the entire RNN-T encoder as well as initializing part of the RNN-T encoder. During the RNN-T supervised training stage, we additionally experiment with joint training combining the supervised RNN-T loss and the self-supervised contrastive loss.

We explore our approach on three datasets in different languages; two in-house far-field corpora of de-identified utterances in German and French from a voice assistant, and one public dataset, Librispeech [26] in English. When experimenting with the Librispeech data, we use the Libri-Light [27] dataset for self-supervised pre-training. The numbers of hours for each dataset are summarized in Table 1 below.

The test sets for the German and French ASR tasks consist of de-identified utterances from a voice assistant, similar to the training data. For the English ASR task, we report results on the Librispeech test-clean and test-other test sets. The amount of test data is also summarized in Table 1.

The encoder of the RNN-T model for our experimentation contains 3 feed-forward (dense) layers of size 512 with ReLU non-linearity, followed by 8 LSTM layers with 1024 units for the German and French ASR tasks, and 6 LSTM layers with 1024 units for the Librispeech ASR task. For experiments that initialize the RNN-T encoder using CPC/GCPC pre-training methods, the feed-forward layers are initialized from the feature encoder network $f_{enc}$ of the CPC/GCPC pre-trained model illustrated in Figure 1, and the LSTM layers are initialized from the autoregressive context network $f_{ar}$. The RNN-T model also contains a single-layer LSTM prediction network with 1024 units, and a single-dense-layer joint network. When training the RNN-T ASR model, we use a sentence piece model [28] containing 4000 sentence pieces which are trained with the corresponding monolingual dataset for each task.

The acoustic features used are 256-dimensional log short-time fourier transform (log-STFT), computed on a 25ms window with a frame shift of 10ms. The input log-STFT features from 3 consecutive frames are stacked for an effective frame size of 30ms, so that the final RNN-T input feature is of the dimension 768. There is no external language model used for any of the ASR tasks.

When computing the guided contrastive loss, extra dense layers $g_{enc}$ are added on top of phone logits, and are updated during the pre-training stage. Our ablation studies described in Section 4.3.3 will demonstrate the impact of these extra feed-forward layers. The number of steps $K$ (in Equation 2) used for pre-training is 4.

The phone classification model is a 5-layer LSTM model with 768 units in each layer. It is trained with the standard cross-entropy loss. Frame-level monophone targets for the internal datasets are generated from the 1-best decoding output of a hybrid LSTM-HMM ASR model. For Librispeech, we obtain the frame-level phone targets using the Montreal forced aligner¹¹1https://github.com/CorentinJ/librispeech-alignments. Frame-level accuracy for these phone classifiers are shown in Table 2. These phone classifiers are then used as the prior knowledge models for the guided contrastive loss based pre-training.

The results for each ASR task (German, French and English) with different encoder pre-training methods are reported in Tables 3 and 4. The baseline RNN-T ASR models ($B_{G_{1}}$, $B_{F_{1}}$, $B_{L_{1}}$) are trained from scratch. Encoders of $M_{G_{2}}$, $M_{F_{2}}$ and $M_{L_{2}}$ are initialized with the encoder pre-trained with standard CPC method explained in Section 2.1. Encoders of $M_{G_{3}}$, $M_{F_{3}}$ and $M_{L_{3}}$ are initialized with our proposed Guided CPC (GCPC) method explained in Section 2.2. For a more comprehensive comparison, we also pre-train the encoder with the cross-entropy loss using phone posteriors (PCE) obtained from the phone classifier described in Section 4.1, and these results are reported for models $M_{G_{1}}$, $M_{F_{1}}$, $M_{L_{1}}$. Note we only report relative WER reduction (WERR) on tasks using the internal data shown in Table 3, but we do report the absolute WER on the Librispeech task shown in Table 4.

Overall, the standard contrastive pre-training on the RNN-T encoder reduces WER on our internal German and French ASR tasks by 2.96% and 1.01% respectively relative to the baseline, while our proposed guided contrastive pre-training method brings higher relative WER reductions (4.44% and 6.55% on the German and French ASR tasks respectively). The standard phone cross-entropy pre-training leads to a worse WER than both the CPC and GCPC pre-training, which indicates the importance of the contrastive term in the pre-training loss function. On the Librispeech task, we observe similar trend under the Test-Clean condition. However, under the Test-Other condition, we don’t see benefits of the GCPC pre-training method. From all our experimental results, we observe that for the tasks with larger labeled training dataset, the gain with the pre-training techniques are smaller.

In order to identify the best training scheme, we perform ablation studies on the internal German dataset. We only show results for the German ASR task, but the WER trends for the ASR models for all 3 languages on both the development and test data were similar. The training scheme was tuned on a held out development set, but we show our results on the test set so the numbers in sections 4.2 and 4.3 are comparable.

To better understand the gains from using the guided contrastive loss, we visualize the output of the RNN-T encoder pre-trained using various methods with the German data. We take the frame level representations for a subset of the development set from the pre-trained RNN-T encoder and apply the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm [29] to reduce the embedding dimension from 1024 to 2. We plot these 2-dimensional embeddings obtained from CPC and guided CPC per monophone.

Figure 4(a)-4(d) show the t-SNE visualizations for four different monophones for the German ASR task. We use the X-SAMPA phone-set for the experiments. For the visualization, we choose two vowels, one plosive and one nasal phone. From these figures, we observe that the representations for each of the phone classes are more closely clustered when using the guided CPC pre-training as compared to using traditional CPC pre-training. Figure 4(e)-4(f) show the t-SNE visualization for three different phones on the same plot from CPC and GCPC pre-trained encoders. Although there is overlap between the different phone clusters, we observe better separation between frames from the different phones with the GCPC pre-trained encoder as compared to the CPC pre-trained encoder. The separation is more pronounced between voiced frames compared to frames with speech like noise (SPN). Table 9 breaks down the relative reduction by substitution, deletion and insertion errors on the German ASR task. The breakdown shows that CPC pre-training degrades the insertion error rate by 3.45% relative, while GCPC pre-training improves the insertion error rate by 3.39%. The better separation between frames with speech like noise and other speech frames explains this improvement in insertion errors.