Transfer Learning Approaches for Streaming End-to-End Speech Recognition System

Speech enabled applications are increasingly gaining popularity across the world. This has initiated a need to build accurate automatic speech recognition (ASR) system across different languages. Also, End-to-End (E2E) ASR systems are emerging as a popular alternative to conventional hybrid ASR systems. They replace the acoustic model (AM), language model (LM) and pronunciation model with a single neural network [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. Recurrent neural network transducer (RNN-T) [1] is one such E2E system that allow streaming input and is suitable for real-time ASR applications. Therefore there is a lot of interest in building accurate RNN-T models for different languages spoken across the world.

There is often disparity in the availability of transcribed data for different languages. In most cases, a lot more data is available for American English than other languages. The quality of ASR model depends on a number of factors including, the training data quantity and diversity, acoustic model structure, and optimization algorithm. Furthermore, training data diversity spans a number of factors in adults, kids, speaking rate, accents, near-field, and far-field acoustic conditions. A low-resource locale has limited ASR training data, and may not meet the acoustic diversity needed to train a robust model that can generalize to above acoustic factors. To overcome the low-resource constraint, transfer learning has been widely used in the hybrid ASR system to transfer the knowledge from a well trained source locale to a low-resource target locale that bring significant acoustic robustness for the target locale. In our recent work, we applied TL from a large scale en-US conventional hybrid model to the corresponding models in en-IN and it-IT locales, and achieved over 8% word error rate relative reduction (WERR). Motivated by the success of the TL methods in the hybrid ASR system, we explore TL methods to improve low-resource RNN-T models.

Besides improving the target model acoustic robustness, TL is also crucial for training large and complex deep learning architectures. RNN-T models are difficult to train [12] and also require significantly large amount of data to jointly train the acoustic as well as language model attributes. In our study we have noted weaker convergence or significant parameter tuning requirements for desirable E2E training outcome for low-resource locale. Therefore we expect TL techniques to be even more relevant for E2E systems to stabilize training and improve ASR accuracy.

In the hybrid ASR system, transfer learning is typically done by initializing the target AM with the source AM. In the RNN-T framework, several transfer learning strategies exist depending upon the choice of the initialization model for the encoder and prediction networks. In this paper, we compare different transfer learning strategies in the RNN-T framework. We propose two-stage TL, by first training a target initialization model bootstrapped with a pretrained source model. Subsequently, this model is used to initialize the target RNN-T model. The two-stage TL approach shows  $17\%$ WERR reduction and faster convergence in the training loss as compared to randomly initialized RNN-T model. We also study the effect of TL with different amount of training data and show the importance of transfer learning in the case of low-resource languages.

Several methods have been proposed to improve the performance of low-resource ASR models [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. Successful strategies include transfer learning [17, 18], that leverage a well trained AM from high-resource language to bootstrap the low-resource AM; multi-task training [19, 20] and ensemble learning [21, 22] that aim to utilize multi-lingual data and share the model parameters. However, most of these methods are studied in the context of hybrid ASR system.

A few multi-lingual approaches are recently proposed in the E2E framework [23, 24, 25]. Authors in [23] propose the multi-lingual RNN-T model with language specific adapters and data-sampling to handle data imbalance. Audio-to-byte E2E system is proposed in [24] where bytes are used as target units instead of grapheme or word piece units, as bytes are suitable to scale to multiple languages. A transformer based multi-lingual E2E model, along with methods to incorporate language information is proposed in [25]. Although multi-lingual methods are attractive to address the problem of low-resource languages, the transfer learning methods, besides being simple and effective, have the benefit of not needing the high-resource language data, but only the models trained on them. In many practical scenarios, trained models are available, however the original corpus is not. Given the simplicity and effectiveness of TL, we explore transfer learning approaches to improve the performance of low-resource RNN-T models.

The rest of this paper is organized as follows: In Section 3, we briefly discuss the RNN-T model. The transfer learning methods for RNN-T are described in Section 4 and experimental setup in Section 5. Next, we discuss results in Section 6, followed by conclusions in Section 7.

The RNN-T model was proposed by Alex Graves [1]. The RNN-T model architecture has three components; an encoder, prediction network and joint network as shown in Fig. 1. The encoder maps the input acoustic feature, $x_{t}$, to a high level representation, $h{{}_{t}}^{enc}$, where $t$ represents the time index. The prediction network receives the previously predicted non-blank symbol, $y_{u-1}$ and maps it to a high level representation, $h_{u}^{pred}$. The joint network is a feed forward network that combines the encoder and prediction network outputs. The posterior probability over all the targets, $p(y|t,u)$ is obtained after softmax operation on the output of joint network. The whole network is trained jointly to minimize the RNN-T loss [1]. In our implementation, the encoder consists of $6$ long short-term memory (LSTM) [26] layers and the prediction network has $2$ LSTM layers along with the input embedding matrix. During inference, beam search decoding is used to find the most likely label sequence.

The RNN-T models are difficult to train and are often initialized with the pretrained models. Initializing the encoder with connectionist temporal classification (CTC) model [2] or cross entropy (CE) model [12], and the prediction network with LSTM language model (LM) is proven to be beneficial [27]. Transfer learning can also be used to overcome the RNN-T training difficulty by initializing the low-resource (target) RNN-T models with the models trained on high-resource (source) languages.

A number of choices exist in selecting the initialization model for encoder and prediction network in the context of TL the RNN-T model. Authors in [12] have shown that CE initialized RNN-T models perform better than CTC initialized models, and hence, we only explore CE models for initialization. The following choices exist for encoder/prediction network initialization of the target RNN-T model: a) Source RNN-T encoder/prediction networks b) Pretrained networks used to initialize the source RNN-T model c) Pretrained models trained only on the target language. Therefore several combinations are possible depending upon the choice of the initialization model for encoder and prediction network.

In this paper, we explore TL methods in the context of Hindi as the target language and American English as the source language. The goal is to improve Hindi RNN-T model by leveraging models trained on American English, which has approximately ten times more data than Hindi. ‘en-US’ prefix is used to refer to models trained with American English and ‘hi-IN’ prefix is used to refer to models trained with Hindi data. We next discuss different transfer learning strategies in detail.

The hi-IN models are trained with approximately $4$ million utterances amounting to few thousand hours of speech data. The speech data is distorted by noise to achieve robustness to noisy conditions. The en-US models were trained with $65000$ hours of data. The hi-IN test set contains $17619$ utterances consisting of five different scenarios including phrasal, conversational and code-mixed utterances. Training and test utterances are anonymized to remove any personally identifiable information.

We use $80$-dimensional log Mel filter bank features computed every $10$ milliseconds (ms). Eight vectors are stacked together to form $640$-dimensional acoustic features fed to the encoder. The frames are shifted by $30$ms. The hyper-parameters such as number of layers, layer dimension, frame size was tuned for en-US model [12] and we adopted the same parameters for Hindi. All encoders have six LSTM layers with $1600$ hidden dimension and $800$ projection dimension. All prediction networks have two LSTM layers with same cell dimension as encoders. Such a model setup follows the en-US work in [28]. All models are evaluated after training for $6$ sweeps of the training data.

The hi-IN grapheme targets contain all the unique graphemes in the Hindi native script. We also include grapheme targets with $B\_$ prefix to be able to segment the grapheme sequence into word sequence. Some research works use <space> symbol, however, we observed better accuracy with $B\_$ prefix based grapheme targets. A total of $130$ grapheme targets are obtained by combining the the original Hindi graphemes, graphemes with $B\_$ prefix and <blank> symbol. The word piece targets for en-US model is obtained by using byte pair encoding [29] algorithm as described in [12].

We also report the word error rate (WER) on hybrid ASR model trained with same amount of data. The AM consists of $6$ layers of latency-controlled bidirectional LSTM [30] with $1024$ hidden dimension and $512$ projection dimension. AM is CE trained followed by EMBR training. The softmax layer has $9212$ senone labels. $80$-dimensional log Mel filter bank features are computed every $10$ms. Frame skipping [31] is done by a factor of $2$. Run-time decoding is performed using a 5-gram language model.

Table. 1 shows WER for different transfer learning methods on hi-IN test sets. en-US CE initialization outperforms random initialization with $15.6\%$ relative WER (WERR) reduction. en-US RNN-T initialization is better than random, while is slightly inferior to en-US CE initialization. This could be because the en-US RNN-T encoder representations are influenced by en-US prediction network representations, as they are trained jointly. However, en-US CE model is trained in isolation and could serve as a better initialization model for the encoder. Two-stage transfer learning with grapheme targets performs better than the rest of the methods with $17.4\%$ WERR reduction over random initialization. The pretraining method, hi-IN CE + hi-IN LM initialization improves over random initialization showing the importance of pretraining. The en-US CE + hi-IN LM initialization is better than pretraining, however, is inferior compared to en-US CE initialization, contrary to our expectation. The reason for such a behaviour is not known and we look to investigate this further in the future. With the improvements obtained from transfer learning, the WER of hi-IN RNN-T model (with transfer learning) is in parity with the hybrid model.

In this paper, we explore transfer learning methods for RNN-T models. Our motivation is to leverage well-trained en-US models to bootstrap hi-IN RNN-T models and also to stabilize the hi-IN RNN-T model training. We evaluated the following transfer learning methods: a) en-US CE initialization b) en-US RNN-T initialization c) Two-stage transfer learning and d) Encoder and prediction network initialization. Based on the WER gains and training convergence, we propose Two-stage learning approach with grapheme targets as the preferred transfer learning strategy. The experiments on smaller data-sets and training loss convergence reveal the importance of transfer learning for low-resource RNN-T models. The methods discussed in this paper can be generalized to other low-resource languages as well. In future, we plan to explore other transfer learning methods and its extension to multi-lingual RNN-T models.