Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces

Unlike when the Transformer[9] was first proposed as an encoder-decoder architecture for machine translation task, we only use the encoder part for acoustic modeling. Specifically we follow the setup in [10] and use VGG layers[34] in lieu of the original sinusoid positional encoding, since we have seen that for ASR tasks, convolutional positional encoding performs the best. Iterated loss[35] was also applied to the intermediate embedding of transformer layers to help convergence and improve accuracy. Also different from the original transformer, we apply layer normalization [36] before multi-head attention (MHA) and feed-forward network (FFN) and we have an extra layer normalization operator after the residual connection. This is necessary to prevent bypassing the transformer layer entirely and helps with model convergence.

Unidirectional recurrent neural networks such as LSTMs base their predictions solely on the history they have already seen, hence the prediction accuracy is worse than bi-directional LSTMs that have access to the full context of the input, including future frames. However for streaming applications such as live captioning, we cannot wait for the full context to arrive because the ASR output needs to be made available within a certain latency budget of the audio stream being fed into the ASR system.

To strike a balance between recognition latency and accuracy, LC-BLSTM was first introduced in [5]. Unlike BLSTM that cannot produce any hypothesis until the whole audio input is processed, LC-BLSTM only utilizes a limited number of right context ($RC$) frames to make predictions, which controls the latency. Similar to BLSTM, each LC-BLSTM layer also has two LSTMs, one left-to-right LSTM and one right-to-left LSTM. The difference is that the the input sequence is first divided into overlapping chunks of chunk size ($CS$) frames. The amount of overlap between chunks is equal to $RC$ frames. When forwarding left-to-right LSTM, hidden states and cell states are carried over between chunks, so that we have unlimited left context as in BLSTM. We forward right-to-left LSTM on each chunk and only keep $CS-RC$ frames of output activations, so that each input frame has at least $RC$ frames of right context to produce its activation. This mechanism enables LC-BLSTM to generate better acoustic embeddings than LSTM, without delaying the generation until the encoder has seen the whole audio.

Chenone was first introduced in [6] as an alternative to traditional senone [27] unit. Chenone not only eliminated the need for a phonetic lexicon, usually generated by linguists, but also showed WER improvement relative to phonetic modeling units in some cases. In order to use chenone as modeling unit in CTC training, we shift the labels after force alignment by one (all labels +1), and use label $\#0$ as the blank label in CTC. Essentially the neural network is now modeling distribution over all chenones plus one blank symbol. We then squeeze same adjacent labels in the alignment sequence into one label, let the encoder trained with CTC criterion to learn the implicit alignments.

After a CTC model is trained, the next step is to construct a decoding graph. We first build an $\rm{H\circ C\circ L\circ G}$¹¹1Note that L here is simply a mapping between word and its graphemes. Examples could be found in [6] graph following the standard procedure for CD-HMM (here, we always assume every HMM only has 1 state); then we transform it to a new graph which can consume an extra blank symbol. For each FST state $s$ in the decoding graph (shown in Figure 1), we split that state into two FST states, $s$ and $s^{\prime}$; we moved the outgoing edges ($e2$ and $e4$) to start from $s^{\prime}$; A self loop edge with blank label as input and $\epsilon$ as output symbol with weight one (in the semi-ring’s sense) is added; $s$ and $s^{\prime}$ are then connected by a $\epsilon:\epsilon$ edge. Self-loop edges ($e3$ in this example) and incoming edge ($e1$) in this case are not changed. The transition model is on-the-fly converted to a new mapping function that shifts the output units by one. Once we change the transition model and decoding graph, the standard Kaldi decoder can be used to decode chenone-CTC model.

Subword unit representations such as byte pair encoding (BPE)[37] and wordpiece model[38] have been proposed with improved performance in many natural language processing (NLP) tasks. This approach chooses to divide words into a limited set of subword units, e.g. the word “hello” may be encoded as “_he ll o”, where the underscore indicates word start. In this work, we train wordpiece model using unigram language model word segmentation algorithm[39], and use the generated wordpiece vocab plus blank symbol as modeling units.

Since wordpiece is context independent, we only need to build an $\rm{H\circ L\circ G}$ graph for decoding, where: $H$ transduces $n+1$ symbols to $n$ wordpieces, i.e. absorbing the blank symbol; $L$ maps the sequence of wordpieces to the sequences of words, e.g. “_he ll o” to “hello”, and is done by the trained wordpiece model; $G$ is the standard $n$-gram word level LM. Once these FSTs are constructed, a standard procedure is used to compose $H$, $L$ and $G$ together.

Compared to alignment-based training which first need to train a bootstrap model and build decision tree; then use the bootstrap model to perform force alignment and finally CE training, we only need to train a wordpiece model using text-only data and followed by one-stage CTC training. The whole training pipeline is greatly simplified.

The LibriSpeech corpus contains about 960 hours of read speech data for training, and 4 development and test sets ({dev, test}-{clean,other}), where other sets are more acoustic challenging. We use the official 4-gram language model (LM) with 200K vocabulary for all first-pass decoding and n-best generation for neural LM rescoring.

The German and Turkish datasets are our in-house video datasets, which are sampled from public social media videos. The datasets are completely de-identified before transcription; both transcribers and researchers do not have access to any user-identifiable information (UII). The training, validation and test set sizes are shown in Table 1. All hyper-parameter tuning is done on validation set. The video datasets contain a diverse array of speakers, accents, video categories, and acoustic conditions, and are more challenging than the LibriSpeech dataset explored in this work.

We use the same bootstrap model obtained from decision tree building stage to segment the training split of both LibriSpeech and video dataset to up to 10 seconds ²²2 This is achieved by force aligning the whole audio against the reference using the LC-BLSTM acoustic model. . We also segment the dev and test split of video dataset to up to 10s, while no segmentation is performed on LibriSpeech dev and test sets in order to be comparable with other published results. The benefit of segmenting training data was shown in [10].

We follow [6, 10] to use context- and position-dependent graphemes (i.e., chenones) for CE baselines and CTC experiments. We bootstrap our HMM-GMM system using the standard Kaldi [41] LibriSpeech recipe. We use 1-state HMM topology with fixed self-loop and forward transition probability (both 0.5). 80-dimensional log Mel-filter bank features are extracted with a 10ms frame shift and 25ms FFT windows. Speed perturbation and SpecAugment [42] (LD policy without time warping) are applied to all experiments unless specially noted.

Since the focus of this work is not on network architecture searches, we use our previously found best setup across all following experiments. For acoustic transformers, we use a 24-layer transformer encoder architecture with embedding dimension 512 and 8 attention heads; the FFN dimension is 2048. Iterative loss was applied to intermediate output embeddings of layer 6, 12 and 18. We use three VGG blocks [43] to encode acoustic features before feeding into transformer layers: each VGG block contains 2 consecutive convolution layers with a 3-by-3 kernel followed by a ReLu non-linearity and a pooling layer; 64 channels are used in the convolution layer of the first VGG block and increase to 128 for the second block and 256 for the third block. Max-pooling is performed at a 2-by-2 grid, with optional stride choice from 1 to 3 in each block. This model has about 81M parameters and we note the model as vggTrf. For LC-BLSTMs, we follow [6] and use 5 layers with 800 hidden units per layer per direction. Optional subsampling by a factor of 2 or 3 can be applied after the first hidden layer. This model has about 83M parameters, similar to the transformer model. Dropout is applied in all experiments: 0.1 for transformers and 0.2 for LC-BLSTM.

All neural network training is performed using an in-house developed PySpeech framework that is built on top of the open-sourced PyTorch-based fairseq[44] toolkit. Adam optimizer [45] with (0.9, 0.999) betas and $1e^{-8}$ epsilon is used in all experiments. We apply the tri-stage[42] learning rate (LR) scheduler. The hold stage LR is $1e^{-3}$. For experiments on transformer models, we follow a schedule of (48K, 100K, 200K) steps with a batch contains up to 20,000 frames. For LC-BLSTM models we use schedule of (16K, 32K, 64K) steps with a batch contains up to 50,000 frames. All models are trained on 32 Nvidia V100 GPUs for 200 epochs in total. Training is usually finished between 2 to 4 days.

Test set WERs are obtained using the best model based on evaluated WER on the development set. The best checkpoints for both LibriSpeech test-clean and test-other are selected using the dev-other development set. For video, the best checkpoint was selected using the valid set.

In the first set of experiments, we investigate what is the best stride for different modeling units. For vggTrf model, striding is achieved by setting stride of max-pooling layers, e.g. a total stride of 8 can be achieved by setting stride equals 2 for all three VGG blocks. For LC-BLSTM, we apply subsampling factor of 2 for activation of the three consecutive hidden layers right after the first layer. We follow the setup in [6] and use stride 2 for the chenone-CE baseline. We use 2K wordpiece size in this study. Results on LibriSpeech test-other dataset shown in Table 2.

The results show that for chenone-CTC, the best stride is 4 and it out performs CE baseline. While for WP-CTC system, we find that we can use stride as high as 8 without losing much accuracy. Because wordpiece model is trained based on word frequency, some common whole words, e.g. ’welcome’ and ’information’, made into the vocab. The acoustic model can use larger stride since the corresponding input frames will span longer. On the other hand, although context dependent, chenones are essentially still characters and have a shorter corresponding time span in input sequence. So we observed that WER degrades quickly when larger stride is used.

In the second set of experiments, we explore the effect of different wordpiece sizes. Here, all models use a total stride of 8. We report results on LibriSpeech test-other dataset in Table 3.

The results show that on LibriSpeech, smaller wordpiece vocab size tends to work better. It’s worth exploring vocab sizes below 1K in future study. We also explored subword regularization during training following [39]. We tried the setup of $(l=64,\alpha=0.1/0.5)$ but results turn out slightly worse. During decoding we also tried building decoding graph with multiple pronunciations in the lexicon with corresponding pronunciation probability for the lexicon entries. The results turn out on-par. We also explored sMBR and mWER training after CTC stage. It provides very marginal gain so we didn’t include the results in this study.

To achieve best WERs, we trained a 36-layer stride-4 vggTrf model with chenone-CTC(about 124M parameters), and changed time masking of SpecAugment LD policy to $(T=30,mT=10)$. Its performance and those of some other published LibriSpeech systems can be found in Table 4. The new system outperforms our previous best hybrid system[10] by 11% and 14% respectively on test-clean and test-other. We also trained a 42-layer transformer LM following the setup in[46] with the LibriSpeech transcriptions and 800M-word text-only data. The transformer LM achieved perplexity 52.35 on the dev set (a combination of dev-clean and dev-other). We then perform n-best rescoring on up to 100-best hypotheses generated by first pass decoding. The oracle error rate of the n-best hypotheses are 1.0% and 2.2% on test-clean and test-other respectively. Our final WERs (2.10%/4.20%) are the best results among all hybrid systems on this widely used benchmark.

Finally, we tested vggTrf with CTC criterion on the more challenging and larger scale internal VideoASR tasks, as described in Section 5.1. Stride 2 is used for CE, 4 for chenone-CTC and 8 for WP-CTC training. Results are shown in Table 5. We find that both CTC systems outperform the baseline CE system. It’s also consistent that chenone-CTC outperforms WP-CTC in terms of WER. On the other hand, WP-CTC system is significantly better in terms of real-time factor (RTF) due to the larger stride used. Blank frame skipping during decoding also contributes to the speedup. We skip frames if blank label posterior is greater than 99%. Empirically, we found that over 20% of the frames in chenone-CTC and over 50% of frames in WP-CTC were skipped.