Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning

A LSTM-based encoder-decoder architecture [1], denoted as $A_{1}$ in the rest of this paper, consists of a Bidirectional LSTM encoder and a LSTM-based decoder which are shown in Fig. 1. Let $<$X, Y$>$ be a training utterance, where X is a sequence of acoustic features and $\textbf{Y}=\{y_{1},y_{2},...,y_{|\textbf{Y}|}\}$ is a sequence of output units. The encoder acts as an acoustic model which maps acoustic features X into an intermediate representation h. Then, the decoder, which consists of an embedding, a LSTM and a projection layers, generates one output unit at each decoding step $i$ as follows,

	$\displaystyle s_{i}$	$\displaystyle=LSTM(s_{i-1},embedding(y_{i-1}))$		(1)
	$\displaystyle c_{i}$	$\displaystyle=attention(\textbf{h},s_{i})$		(2)
	$\displaystyle P(y_{i}\mid\textbf{X},y_{<i})=\$	$\displaystyle softmax(proj(s_{i})+proj(c_{i}))$		(3)

where $c_{i}$ is the context vector, $s_{i-1}$ and $s_{i}$ are output hidden states at time step $i-1$ and $i$ respectively, $embedding()$ and $proj()$ are embedding and projection layers respectively.

From Equation (1), the LSTM is only conditioned on the previous decoding hidden state and previous decoding output. In other words, the LSTM acts as an independent language model that can be easily updated with text-only data [1].

Transformer has been proposed in [6] for sequence-to-sequence modeling in natural language processing tasks, then adopted to the ASR task in [12, 13]. The model architecture, denoted as $A_{2}$, is shown in Fig. 2.

The encoder is shown in the left half of Fig. 2. It consists of $N_{e}$ encoder-blocks, each of them has two sub-blocks: self-attention and position-wise feed-forward network (FFN). The self-attention employs multi-head attention (MHA) which is a function of three inputs: query $Q$, key $K$ and value $V$. It includes multiple heads which can be processed in parallel. Each head employs a dot-product attention ($DotProdAtt$) as follows,

$$DotProdAtt(Q,K,V)=softmax(\frac{QK^{T}}{d_{k}})V$$

(4)

where $d_{k}$ is the hidden dimension. Besides, layer normalization [14] and residual connection [15] are introduced to each encoder-block for effective training.

The decoder is shown in the right half of Fig. 2 which consists of $N_{d}$ decoder-blocks. Different from the encoder-block, each decoder-block has one more sub-block, i.e. the cross-attention. Its first input comes from the previous sub-block output while another input is from the output of the encoder.

In this section, we first compare approaches presented in Section 2.1 and 2.2. Then, based on the comparison, we propose a novel architecture that takes advantage of both approaches.

Previous work [16] showed that the Transformer not only produces better encoding representation but is also faster than the LSTM counterpart in training. First, the Transformer encoder uses dot-product attention (see Equation (4)) which allows each position has access to information from all other positions in the same sequence regardless of their distance. In contrast, although in theory LMST can model long-range dependence, in practice it faces difficulty to capture dependencies of far-distance elements [17] which limits its modeling capacity for long sequences such as acoustic signal. Second, by relying entirely on feed-forward components, the Transformer model avoids any sequential dependencies, and hence can maximize parallel training. In contrast, training a LSTM-based network is slow due to the recurrence property of the LSTM.

Despite being highly effective, the Transformer decoder is not easy to be improved using text-only data. Specifically, the decoder includes the cross-attention sub-block which is conditioned on the encoder output. In contrast, the LSTM-based decoder (in Section 2.1) can be easily boosted using the text data. Another issue of the Transformer decoder is slow inference [18]. Specifically, to generate an output $y_{i}$, the decoder needs to process all previous decoding units $y_{1:i-1}$. On the other hand, the LSTM-based decoder has faster inference since it only needs the last output unit $y_{i-1}$ to generate $y_{i}$.

Based on the above comparisons, we propose a hybrid architecture that takes advantage of both Transformer and LSTM architectures. Specifically, our encoder is from Transformer, while the decoder is taken from the LSTM architecture. The benefits of the proposed architecture lie in two aspects. First, it has high modeling capacity, as well as faster training and decoding. Second, the LSTM-based architecture allows us to easily leverage text data to boost the decoder, yielding improved ASR performance. We denote the proposed architecture as $A_{3}$ in the rest of this paper.

To tackle the low-resource training problem, we first perform cross-lingual transfer learning. We start with training E2E models using a source language. We then replace the language-dependent components of the decoder (i.e. the embedding and output projection layers) of the source language by those of the target language. Finally, the models are fine-tuned using labeled data of the target language. Although transfer learning is not our focus, to achieve the best performance, we carefully examine various transfer settings as presented in Section 5.2. More importantly, we aim to boost the transferred model of the proposed architecture using extra text data of the target language. Fig. 3 describes our process.

From Fig. 3, the entire process is implemented with two main steps. In the first step, we merge the extra text and the labeled data together to fine-tune the transferred model. This avoids a so-called catastrophic forgetting problem as mentioned in [1]. Specifically, at each training iteration, we mix a batch of labeled data consisting of $B_{labeled}$ utterances with a batch of text data consisting of $B_{text}$ utterances to fine-tune the transferred model with the following loss function:

$$L_{total}(\theta)=(1-\lambda)L_{ASR}(\theta)+\lambda L_{LM}(\theta_{d})$$

(5)

where $\lambda$ denotes an interpolation factor, $\theta$ and $\theta_{d}$ denote entire E2E parameters and decoder parameters respectively, $L_{ASR}(\theta)$ and $L_{LM}(\theta_{d})$ denote the ASR loss and LM loss generated by the labeled data and text data respectively. In the second step, the model is further fine-tuned with the labeled data of the target language. Similar to [1], we empirically found that the second step is necessary to improve overall performance.

We conduct experiments on our in-house corpus of Malay language which consists of limited labeled data plus extra text. We split the labeled data into three sets for training, validation and evaluation. Detailed division is shown in Table 1. We perform speed perturbation based data augmentation [19] on the training data. For extra text data, we examine two sets: the first set has over 8 million sentences, denoted as $T1$; while the second set is a subset of the first one which consists of 2 million sentences, denoted as $T2$.

For the source language, which is English, we use two subsets of the National Speech Corpus (NSC) [20] to train source models. The first subset, denoted as $S_{200h}$ consists of 200 hours, where we also apply speed perturbation based data augmentation. The second subset consists of 1000 hours, denoted as $S_{1000h}$.

ESPnet toolkit [21] is used to train our E2E architectures. We use 80 mel-scale filterbank coefficients with pitch as input features, and 500 Byte-Pair Encoding (BPE) units are used as output units. For all E2E architectures, the acoustic features are processed by the VGG network [22]. Detailed setting of architectures can be seen in Table 2. Each BLSTM layer has 320 cells, while each LSTM layer of the decoder has 256 cells. In each self-attention and cross-attention sub-blocks, we use 4 heads with hidden dimension of 2048. The FFN consists of two ReLu activation functions and two affine transforms with size of 2048. To allow transfer learning, we use the same settings for both source and target languages. During the fine-tuning process in Section 3.2, we set $B_{labeled}=30$ and $B_{text}=90$.

This section presents the results of different E2E architectures, i.e. $A_{1}$, $A_{2}$ and $A_{3}$ trained using only the labeled data of the target language. The word error rate (WER) results are reported in the first three rows in Table 3. We also report the decoding speed of these models in Table 4. As we can see, $A_{3}$ not only outperforms $A_{1}$ by 24.2% relative WER but also offers 1.5 times faster decoding. This indicates that employing the Transformer for the encoder is very effective. $A_{2}$ achieves best WER, but has much slower decoding speed compared to $A_{3}$.

In this section, we examine the ASR performance of different transfer learning settings. Firstly, we examine the effect of using different amounts of source language data on the target language’s performance. Secondly, we analyze the different level of transfer learning: (1) only $l$ (e.g. $l$ = 3, 6, 9) bottom layers of the encoder is transferred; (2) entire encoder is transferred; (3) both encoder and decoder (except embedding and projection layers) are transferred. These experiments were conducted only for $A_{2}$ and $A_{3}$, since $A_{1}$ produced worst results. The experiment results are given in Table 5.

We observed that transferring the entire encoder achieves the best results for both $A_{2}$ and $A_{3}$. Additionally, transfer with 1000 hours of source data is noticeably better than that of 200 hours with data augmentation. The best results are summarized in rows 4 and 5 in Table 3. It can be seen that cross-lingual transfer learning leads to significant improvement for both $A_{2}$ and $A_{3}$. For example, the result in row 5 outperforms that of row 3 by 25.4% relative WER (from 13.8% to 10.3%).

We first present the ASR performance on development data when using extra text data $T1$ and $T2$ to fine-tune $A_{3}$ after cross-lingual transfer learning. The results are shown in Fig. 4. We observed that using extra text data is very effective and $T1$ produces substantial improvement over $T2$. We also observed that Step 2 (see Fig. 3), i.e. to use labeled data to fine-tune the E2E network, is essential. Finally, $\lambda=0.7$ yields the best results in most of the cases.

We then employ $T1$ with $\lambda=0.7$ on the test set and the result is reported in the row 6 of Table 3. Using extra text data significantly improves $A_{3}$ by 13.6% relative WER (from 10.3% to 8.9%). With the help of the extra text data, the proposed architecture outperforms the Transformer baseline by 11.9% relative WER (from 10.1% to 8.9%).

We now investigate the effect of an external language model on the proposed architecture $A_{3}$. We train a Recurrent Neural Network LM (RNN-LM) as a 1-layer LSTM with 1024 cells using both transcriptions of the training data and the extra text data $T1$, then integrate the RNN-LM into inference process of $A_{3}$ (row 5 and 6 in Table 3). Results are reported in Table 6. As we can see, after fine-tuning with $T1$, $A_{3}$ still benefits from the external RNN-LM and we observed 34.8% relative WER reduction (from 8.9% to 5.8%).