An Empirical Study of Language Model Integration for Transducer based Speech Recognition

The RNN-T model [2] consists of an acoustic encoder (a.k.a. the transcription network), a prediction network (PN) and a joint network. With given acoustic features $X=(\mathbf{x}_{1},...,\mathbf{x}_{T})$ and the token sequence $Y=(y_{0},...,y_{U})$, the posterior probability is defined as:

	$\displaystyle P_{\text{RNN-T}}(Y|X)$	$\displaystyle=\sum_{\tilde{Y}\in\mathcal{B}^{-1}(Y)}P(\tilde{Y}|X)$		(3)
		$\displaystyle=\sum_{\tilde{Y}\in\mathcal{B}^{-1}(Y)}\prod_{i=1}^{T+U}P(\tilde{y}_{i}|X,y_{0:u})$

$$P(\tilde{y}_{i}|X,y_{0:u})=\text{Softmax}\left[J\left(\mathbf{g}_{u},\mathbf{f}_{t}\right)\right]$$

(4)

Here $\tilde{Y}=(\tilde{y}_{1},...,\tilde{y}_{T+U})$ represent the alignment sequence including the blank label <blk>. $t\in[1,T]$ and $u\in[0,U]$ denote the number of speech frames and that of non-blank labels up to emitting $\tilde{y}_{i}$. $\mathcal{B}(\cdot)$ represents the alignment mapping of $\tilde{Y}$ to $Y$ by removing <blk>. The token-level probabilities are computed by the joint network as in Eq. (4), where $\mathbf{g}_{u}$ and $\mathbf{f}_{t}$ denotes the output hidden features of the PN and encoder. $J(\cdot)$ denotes the joint network, which consists of linear layers and non-linear activations in the common RNN-T architecture.

The shallow fusion (SF) method takes the linear combination of log scores of E2E models and the ELMs, with one extra parameter $\beta$ as reward for the sequence length $|Y|$. The hypothesis $\hat{Y}$ is selected as follows:

$$\hat{Y}=\mathop{\arg\max}_{Y}\left[\log P_{\text{E2E}}(Y|X)+\lambda\log P_{\text{ELM}}(Y)+\beta|Y|\right]$$

(5)

The density ratio (DR) method [17] is proposed as an extension of SF, and makes the assumption that the model distribution $P(Y|X)$ captured by RNN-T can be factorized into acoustic and linguistic parts as in the hybrid system:

$$P_{\text{RNN-T}}(X|Y)=P_{\text{RNN-T}}(X)\frac{P_{\text{RNN-T}}(Y|X)}{P_{\text{RNN-T}}(Y)}\vspace{-1mm}$$

(6)

where $P_{\text{RNN-T}}(X)$ models the marginal distribution of data. The DR method further assumes: (1) the acoustic models of the source domain (where the RNN-T is trained) and the target domain are consistent, both modeled by $P_{\text{RNN-T}}(X|Y)$; (2) the linguistic distribution for the two domains can be separately estimated via LMs $P_{ILM}(Y)\approx P_{\text{RNN-T}}(Y)$ and $P_{ELM}(Y)$, respectively. According to [17], $P_{ILM}(Y)$ is estimated by a separately trained neural LM over source domain transcripts. $P_{ELM}(Y)$ can be trained over the target domain corpus. With these assumptions, decoding for recognizing utterances from the target domain can be derived the same as shown in Eq. (2). Introducing LM weights $(\lambda_{0},\lambda_{1})$ and length reward $\beta$, we have

	$\displaystyle\hat{Y}=\mathop{\arg\max}_{Y}$	$\displaystyle\left[\log P_{\text{RNN-T}}(Y|X)+\lambda_{0}\log P_{\text{ILM}}(Y)\right.$		(7)
		$\displaystyle\left.+\lambda_{1}\log P_{\text{ELM}}(Y)+\beta|Y|\right]$

When $\lambda_{0}=-1,\lambda_{1}=1$ and $\beta=0$, Eq. (7) is equivalent to Eq. (2), in a strict manner of Bayes rule.

Inspired by [13], ILME [14] applies Proposition 1 in [13, Appendix. A] to estimate the ILM via zeroing out the acoustic part in Eq. (4). The proposition claims that, if $J\left(\mathbf{g}_{u},\mathbf{f}_{t}\right)\approx J\left(\mathbf{g}_{u},\mathbf{0}\right)+J\left(\mathbf{0},\mathbf{f}_{t}\right)$ is satisfied, we have

$$P_{\text{RNN-T}}(y_{u+1}|y_{0:u})\propto\exp\left(J(\mathbf{g}_{u})\right)\vspace{-1.5mm}$$

(8)

where we omit the $\mathbf{0}$ to simplify the notations. So the ILM of RNN-T can be computed by applying Softmax to the token logits excluding <blk>, i.e.

$$P_{\text{ILM}}(Y)=\prod_{u=0}^{U}\text{Softmax}\left[J_{\setminus\verb|<blk>|}(\mathbf{g}_{u})\right]\vspace{-2mm}$$

(9)

The ILME decoding basically follows Eq. (7), except that the ILM is estimated from RNN-T itself. It is reported that the simple joint network described in Sec. 2.1 makes RNN-T possible to roughly satisfy the condition of the proposition [13].

As discussed in Sec. 1, previous works have shown that the ILM of RNN-T actually captures small amount of LM information from the transcripts [13, 14], and RNN-T only makes use of limited context and low-order information in the prediction network [13, 18]. In contrast, the DR method uses a well-learned LM with full context as the estimation of ILM. Inspired by the findings, we hypothesize that the original DR setting is inappropriate and may deteriorate the performance of integration.

To investigate the performance of DR using a low-order weak ILM, we separately train a low-order LM on the transcripts and follow the Eq. (7) to do the integration. In implementation, we train a bi-gram LM on the training transcripts, and prune the bi-grams except for the most frequent 20k ones. This would produce a very small model, typically around 250 kB on disk. Note that if the modeling units are in small granularity like alphabets, the number of bi-grams is probably not enough up to 20k; in that case, one may need to use relatively higher order of n-gram. In our experiments, we use 1024 word-pieces for English and around 5k characters for Chinese as the modeling units, where the numbers of bi-grams are both more than 100k. We leave further discussion to the Sec. 4.3.

As is shown in Eq. (5) and Eq. (7), SF, DR, ILME and LODR are all essentially conducting linear interpolation between scores of the RNN-T and the LMs, where the hyperparameters, i.e., the $\lambda_{0},\lambda_{1}$ and $\beta$, are shown to be important to the integration performance [14, 16, 17]. Since lacking of common hyperparameter tuning setup in the literature and grid-search becomes too expensive as the number of hyperparameters exceeds two, in this work, we detail our hyperparameter searching method, which is coordinate descent [21] combined with binary search:

1. 1.

   Initialize the searching ranges and minimum interval sizes for each hyperparameter.

2. 2.

   Tune one hyperparameter per iteration. In an iteration, fix other hyperparameters and only search for one. Binary search is used for tuning one hyperparameter, and is done when the searching range is smaller than the minimum interval size.

3. 3.

   Continue the loop until there is not any better combination. Once the tuned hyperparameters are at the boundary of the initial range, we extend the range and continue the loop.

The hyperparameters are first tuned on a held-out validation set, then evaluated on the test sets. In the experiments, all LM integration methods follow the same steps to tune their hyperparameters, except that there are two hyperparameters for SF as Eq. (5), while three for DR, ILME and LODR as Eq. (7).

The hyperparameters of LM integration are tuned on: dev for WenetSpeech; dev-clean+dev-other for LibriSpeech. We set the initial range [0, 1] and minimum interval size 0.1 for parameter tuning. Though the scales of ILMs $\lambda_{0}$ are suggested to take negative in [17, 14], we do not add such restriction. With the range extending mechanism descried in Sec. 3.2, $\lambda_{0}$ is still possible to be negative.

The RNN-T model in WenetSpeech experiment is of around 90M parameters, while the one in LibriSpeech is Conformer-L with 120M parameters, following [5]. The ELM used in LibriSpeech experiment is a Transformer LM with 87M parameters, trained on the 800M-word LibriSpeech corpus. No extra data is used, so our LibriSpeech results in Table. 1 are comparable to those in literature [5, 16]. WenetSpeech experiments take the 200M-char WenetSpeech corpus to train the ELM. The ELMs are shared for all methods on each dataset.

As Table. 1 shows, all methods bring significant improvement over the baseline (standalone RNN-T without LM). The three methods with ILM estimated (DR, ILME and LODR) are of close performance, and all outperform the SF method.

Note that all $\lambda_{0}$ in the Table. 1 are 0 or -0.125. Considering that the minimum interval size in hyperparameter searching is 0.1, this could reveal that for in-domain evaluation, the subtracted part might be unimportant. This somehow shows the SF method is sufficiently good for in-domain test.

Among the methods estimating ILMs, in LibriSpeech experiment, DR, ILME and LODR perform considerably close on average; in WenetSpeech experiment, our LODR method gains 1.2% relative CER reduction over DR and slightly outperforms ILME.

It is also shown in the Table. 1 that the estimated ILM of DR simulate the transcript best (with the smallest perplexities on transcripts, in both datasets), but the evaluation performance does not consistently surpass the ILME and LODR, which further verifies our analysis in Sec. 3.1.

In cross-domain evaluation, we follow most of the settings in in-domain test, except that the evaluation is on a new domain and the ELM is trained on the target domain corpus.

Table. 2 shows the performance of the integration methods in domain adaptation. When adapting the RNN-T trained on LibriSpeech to Tedlium-2, DR gets the best WER on both dev and test sets. DR, ILME and LODR obtain 1.4%, 1.2% and 0.5% relative WER reduction on the Tedlium-2 test compared to SF. This relative reduction is much smaller than the literature results [16], which reports the DR gains 8.5% (16.4 $\rightarrow$ 15.0) relative WER reduction and ILME gains 12.2% (16.4 $\rightarrow$ 14.4) compared to SF. We argue that, this is probably due to our significantly lower baseline (No LM: 11.41 vs. 20.3 [16]).

In the adaptation from WenetSpeech to AISHELL-1, our proposed LODR method obtains 6.2% and 4.4% relative CER reduction over DR and ILME methods, respectively. Overfitting of weights tuning is observed in the adaptation from WenetSpeech to AISHELL-1: compared to SF, the DR method obtains lower CER on dev set, but performs worse on test set with 1.6% relative CER increase; ILME gains 2.2% relative CER reduction on dev set, while only 0.2% on test. In contrast, our LODR method obtains 4.6% and 4.1% relative CER reduction on the two sets over SF, showing that LODR is a very promising method for cross-domain LM adaptation. When adapted to TV-news from our in-house Tasi dataset, LODR also outperforms all SF, DR and ILME methods.

As we explained in Sec. 3.1, LODR is driven by estimating the ILM for RNN-T with the cognition that the ILM is indeed a low-order one.

To further investigate whether LODR really benefit from the transcript information, we do an ablation study that train the low-order LM from an external corpus. Here the external corpus is randomly taken from the ELMs training corpus described in Sec. 4.1, keeping the size the same as the transcripts. It is interesting that the low-order LMs trained on transcript and external corpus have consistent performance in cross-domain adaptation on both Chinese and English tasks. An intuitive interpretation is that, under the low-order bi-gram modeling and with most of the bi-grams pruned, the LM only learns very basic statistics about the language itself, which are, to some extent, consistent across corpus.

Reviewing the results in Table. 1 and Table. 2, it seems LODR performs consistently better in Chinese tasks, but not as in English. This is possibly due to the smaller number of modeling units in English tasks, which may require more than 20k bi-grams in the pruned low-order LM. Note that the number “20k” is an empirically set value, may not be optimal.