On Comparison of Encoders for Attention based End to End Speech Recognition in Standalone and Rescoring Mode

The end-to-end speech recognition models directly convert audio features or raw waveform into output text. These models provide easy training and inference setup as compared to traditional models with separate acoustic, pronunciation, and language components [1]. These end-to-end models are based on CTC, RNN-T, or LAS architectures [2, 3, 4]. One of the important aspects influencing the choice of these models is the streaming capability. The streaming models transcribe the audio on the go whereas the non-streaming models wait for the entire audio before starting the transcription.

The streaming models are desirable for consumer-facing applications like voice search [5, 6]. Immediate feedback in the form of interpreted text makes such applications more user-friendly. The non-streaming models can be utilized for applications like automated customer experience support. Although non-streaming models have limited applications it provides superior performance as compared to streaming models [7]. The streaming models make use of only past context to generate current text. The non-streaming models look at the entire audio context before producing text and hence provide better results as compared to limited context streaming models.

Recently, non-streaming models have been used in second pass re-scoring mode to rank the text hypothesis generated using the streaming models. The re-scoring happens once the entire audio is available. This combination is shown to improve the results of streaming models [8]. These second pass models are commonly based on Listen-Attend-Spell (LAS) architecture. The LSTM and Transformer based variants have been explored in literature. The LAS model attends to encoder output to score the target text. The successor to these re-scoring systems are the recently proposed deliberation networks [9]. These networks encode the candidate hypothesis using a separate text encoder and the decoder attends to both audio and text representations. This allows the network to utilize bidirectional context with both text and audio. However, this further complicates the architecture and hence is not considered in this work.

In this work, we describe the LAS-based non-streaming models and evaluate them on Flipkart Voice Search task [10, 11]. The models are evaluated in standalone text generation mode and second pass rescoring mode. The two-pass system uses hypotheses generated from the CTC-based streaming model. We present a comparative analysis of different encoder architectures based on LSTM, Transformer, and Conformer. These architectures have provided state-of-the art results over time and hence been considered in this work. We also explore a shared first-pass system encoder instead of an independent audio encoder. We study these variations in re-scoring mode and first-pass standalone mode and make some important observations. To the best of our knowledge this is the first work to perform such a comparison for LAS based ASR systems.

The most basic form of second pass re-scoring involves the usage of language models. The language models (LM) trained on large text-only corpus have even been shown to improve the accuracy of end-to-end ASR systems [12]. The text-only corpus is easily available as compared to parallel audio-text pairs and can be leveraged to train language models. The language models are used during the decoding of text as well as to re-score the final hypothesis [13]. The importance of using more powerful BERT-based language models has been shown in [14]. However, the BERT-based re-scoring has not been popular due to latency requirements. A more usable approach incorporating discriminative loss during BERT training was proposed recently in [15].

A class of text correction models has been proposed to correct the generated ASR text [16, 17]. The idea is similar to having a second pass system however it takes a step further to correct the text instead of just re-scoring the candidate hypothesis.

On similar lines, a second pass ASR model can be used to re-score the first pass hypothesis. The full second pass ASR model performs better than language models as it also considers the full audio context. However, both forms of re-scoring provide complementary benefits and are often used together. A two-pass model using streaming RNN-T architecture in the first pass and a non-streaming LAS in the second pass was proposed in [8]. They showed that RNN-T + LAS architecture outperformed the conventional streaming models while providing acceptable latency. The LSTM based LAS decoder was replaced with a Transformer decoder in [18] to enable efficient utilization of resources. The deliberation models were used for second pass rescoring in [9]. The deliberation models attend to both audio representation and text representations and are shown to work better than LAS rescoring which attends to audio representations only. Further, the LSTM decoder in the deliberation network was replaced with a Transformer decoder in [19].

We evaluate the non-streaming LAS models in standalone and re-scoring mode. The rescoring mode follows a two-step process. The first pass top 100 hypothesis generated using streaming CTC based system is re-scored using non-streaming LAS model. In the standalone mode, we do a beam search with beam size = 10. The process for rescoring mode is shown in Figure 1 and Figure 2. The rescoring mode is evaluated in the shared encoder and independent encoder setup. The shared encoder setup utilizes the trained encoder of the first pass system with some extra layers. The encoder of the independent encoder system is different from the first pass system and hence trained from scratch.

The LAS model follows an encoder-decoder architecture. The decoder attends to the encoder hidden representation and generates the output text sequentially. During re-scoring mode entire candidate hypothesis is processed in parallel to generate the score [18]. The log probability of the input token sequence is taken as the LAS score. The LAS score is combined with the other scores using the following formula:

The values of $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, and $\lambda_{4}$ are fixed using grid search on the held-out validation data. The $ctc\_score$ indicates the score from first-pass model, $lm\_score$ is the language model (LM) score from a domain LM and $len$ is the number of subword tokens.

The decoder architecture is fixed to a 2 layer transformer. We explore different encoders for the LAS model based on LSTM, Transformer, and Conformer in both modes. We also explore the integration of the CTC model encoder in the LAS setup. This setup is similar to the method outlined in [9]. The LSTM encoder used in the first pass model is frozen and used as the LAS encoder.

In this work, we report word error rates for different LAS models in standalone and rescoring mode. For each mode, we evaluate LSTM, Transformer, and Conformer-based encoders. Another efficient approach is to re-use the first pass encoder for second pass re-scoring. The results for standalone mode and re-scoring are shown in Table I. In the standalone mode, the Bi-LSTM pyramidal encoder and Transformer encoder with VGG frond-end perform the best. This shows the importance of the VGG frontend with the Transformers model. Although Conformer has been shown to produce state-of-the-art results on Libri Speech data, the numbers are slightly on the lower side when used as a LAS encoder on our VS dataset. The Conformer model evaluated in this work uses 10 encoder layers as opposed to 17 used in the original work. This allows us to train the model on commodity GPU systems. Finally, sharing the encoder with the first pass streaming system reduces the performance even further. We add BiLSTM and Transformer based additional small encoders on top of the frozen CTC encoder. The BiLSTM based system performs better than the Transformers counterpart in standalone mode.

In re-scoring mode, the Conformer encoder performs the best but has the highest latency requirements. In general full encoders perform better than the shared encoder setup. However, the difference in WERs for different encoders is much less in re-scoring mode. Therefore the choice of encoder architecture is not much relevant and all the encoders perform more or less the same. The latency requirements for these encoders are also listed in Table I. The latency was computed on a Intel Xeon Skylake system with Nvidia A100 GPU. The transformers provide significant benefit in latency as compared to LSTM based models. Overall we report a relative WER improvement of 16-17% over the first pass streaming system. The latency requirements of the shared encoder setup are the lowest as we only consider computations in the additional encoder part. The shared encoder setup provides comparable WERs with minimal latency requirements.

In conclusion, we evaluate different variations of non-streaming LAS models on the Flipkart Voice Search Task. The end-to-end speech recognition models were evaluated in standalone and re-scoring mode. We compare different encoders based on Bi-LSTM, Transformer, Conformer, and shared uni-LSTM. The Transformer model with the VGG frontend is shown to provide the best accuracy with reasonable latency requirements. In the re-scoring mode, different encoders provide very competitive performance as the hypothesis is generated by the first pass model. The lowest latency approach with an overhead of 4.25ms provides a relative WER improvement of 16% over the first pass system.