Streaming end-to-end bilingual ASR systems with joint language identification

Multilingual automatic speech recognition (ASR) is an active area of research for two reasons: (1) it improves performance across languages, particularly the under-resourced ones, by allowing data sharing and cross-lingual knowledge transfer [1, 2, 3, 4, 5, 6]; and (2) it allows the same model to be used by two or more languages [7], thereby creating seamless multilingual experiences via simplified model training, maintenance, and deployment. In hybrid ASR systems with separate acoustic model (AM) and language model (LM) components, shared hidden layer training [8, 9, 10] and transfer learning [11, 12, 13] have been proposed to address the data scarcity problem for under-resourced languages. However, achieving the model unification objective is more challenging in hybrid systems given the AM-LM separation. With recent end-to-end ASR advancements such as listen, attend and spell (LAS) [14] and recurrent neural network transducers (RNN-T) [15], unified multilingual ASR systems are now possible. Among these popular frameworks, RNN-T is favored in online applications owing to its support for streaming ASR [16]. We mainly focus on the objective of delivering a seamless experience to multilingual users without them having to specify the language before each interaction.

A commonly adopted architecture for creating a live multilingual experience is to run parallel ASRs behind the scenes while a language detector identifies spoken language [17, 18]. Multilingual ASR systems can simplify such architectures by eliminating the need to run multiple monolingual ASRs, but previous research on multilingual end-to-end models suggests that language information (assumed to be known beforehand in most studies) plays a crucial role in achieving acceptable levels of performance [19, 20, 21, 22, 23]. While language information can be provided at runtime by preselecting the language, such an experience could be counter-intuitive because multilingual users often prefer to seamlessly switch between languages.

Streaming language detectors have been proposed as a solution for guiding multilingual ASR on-the-fly [24]. Also, recent studies [17, 18] demonstrate that language identification (LID) can be improved by combining conventional acoustic representations with textual cues from the ASR decoder. To leverage these ideas—and also the fact that RNN-T models the joint evolution of acoustic and lexical information—, we propose a multilingual RNN-T architecture that consumes embeddings from an acoustic LID classifier and predicts both the spoken content and the spoken language. Accurate LID can improve ASR performance via language-specific second-pass rescoring and also enable appropriate downstream processing by language-specific components like natural language understanding. Furthermore, multilingual joint ASR-LID models can simplify dynamic language switching by replacing several monolingual ASRs and a language detector with just one system. The key contributions of this paper are summarized below.

$\bullet$ Joint ASR-LID: We append ground-truth transcripts with language tags and include them in the model’s vocabulary. This approach has been explored for LAS [25], but, to the best of our knowledge, has not been studied in the context of RNN-T.

$\bullet$ Acoustic LID embeddings: Frame-level embeddings from an acoustic LID classifier (trained beforehand) are used to guide RNN-T training and inference. This is similar in principle to the idea of deriving language information from an auxiliary RNN-T [24], but it has certain advantages (see Section 2).

$\bullet$ Biasing joint network: Previous studies provide language information to RNN-T’s encoder or decoder [21]. In this work, we study the effect of influencing RNN-T’s joint network.

$\bullet$ Languages: Previous studies mostly focus on groups of related languages, e.g. Indic, Arabic and Nordic [21, 24]. Motivated by the real-world usage patterns in bilingual markets, this work explores two pairs of unrelated languages: English-Hindi and English-Spanish.

In the rest of this paper, multilingual and bilingual will be used interchangeably—the latter will mostly be used when referring to experimental details or results.

A typical RNN-T ASR architecture comprises a transcription network (encoder), a prediction network (decoder) and a joint network. The encoder is an RNN that converts an input acoustic feature vector, $\mathbf{x}_{t}$, to a hidden representation, $\mathbf{h}^{enc}_{t}$, and the decoder is an RNN that receives the last non-blank label observed ($y_{u-1}$) and outputs a hidden representation, $\mathbf{h}^{dec}_{u}$:

$$\mathbf{h}^{enc}_{t}=f^{enc}(\mathbf{x}_{t}),\text{ }\mathbf{h}^{dec}_{u}=f^{dec}(y_{u-1}).$$

(1)

The joint network is a feedforward network that combines the encoder and decoder outputs to produce logits, $\mathbf{z}_{t,u}$:

$$\mathbf{z}_{t,u}=f^{joint}(\mathbf{h}^{enc}_{t},\mathbf{h}^{dec}_{u}),$$

(2)

which in turn are passed through a softmax layer to produce a probability distribution for the next output symbol, $y_{u}$ (either blank or one of the ASR targets):

$$P(y_{u}|t,u)=\mathrm{softmax}(\mathbf{z}_{t,u}).$$

(3)

A naive approach to train multilingual ASR using RNN-T is to simply pool data from the languages of interest and define the output symbol space as the union of the individual symbol sets. As literature has shown, better results could be achieved by supplementing $\mathbf{x}_{t}$ with auxiliary language information, $\mathbf{l}_{t}$, to obtain an improved encoder representation, $\mathbf{g}^{enc}_{t}$:

$$\mathbf{g}^{enc}_{t}=f^{enc}([\mathbf{x}_{t};\mathbf{l}_{t}]).$$

(4)

In this paper, we also study the effect of supplying language information to the joint network. This enables us to determine if language information is more helpful at the input where lower-level features are extracted, or deeper inside the network where higher-level acoustic and lexical information are combined. Depending on whether $\mathbf{l}_{t}$ is provided to the encoder, joint network or both, the model’s logits can be computed using Eq. (5), (6) or (7), respectively. Further implementation details are presented in Sections 2.2 and 2.3.

	$\displaystyle\mathbf{z}^{E}_{t,u}$	$\displaystyle=$	$\displaystyle f^{joint}(\mathbf{g}^{enc}_{t},\mathbf{h}^{dec}_{u}),$		(5)
	$\displaystyle\mathbf{z}^{J}_{t,u}$	$\displaystyle=$	$\displaystyle f^{joint}(\mathbf{h}^{enc}_{t},\mathbf{l}_{t},\mathbf{h}^{dec}_{u}),$		(6)
	$\displaystyle\mathbf{z}^{B}_{t,u}$	$\displaystyle=$	$\displaystyle f^{joint}(\mathbf{g}^{enc}_{t},\mathbf{l}_{t},\mathbf{h}^{dec}_{u}).$		(7)

To train a multilingual joint ASR-LID model we extend the RNN-T’s output symbol space to include language targets. Further details related to training and inference are presented in Section 2.4. The proposed multilingual joint ASR-LID architecture is schematically summarized in Fig. 1.

We conduct experiments using two pairs of languages: English-Spanish, as spoken in the United States (languages denoted as en-us and esp-us, respectively), and English-Hindi, as spoken in India (languages denoted as en-in and hi-in, respectively). Table 1 captures the dataset statistics.

Of the two language pairs studied, English-Hindi shows a higher degree of within-utterance code switching owing to the colloquial nature of spoken Hindi (which involves frequent use of common English words such as “play”, “call”, “book”, “volume”, etc.) and the presence of named entities in a number of other Indian languages. Utterances in en-in are transcribed using Latin script, whereas utterances in hi-in are transcribed using both Latin and Devanagari scripts: the latter for Hindi words and the former for words in English and other languages. Since named entities such as Hindi movie titles, song names, etc. are common to both the languages, a bilingual model can probabilistically output several words in either Latin or Devanagari. These aspects make English-Hindi a challenging language pair to work with. Table 2 summarizes the remaining details pertaining to our experimental setup.

Table 3 summarizes the results obtained from all of our experiments. We report word error rate reductions (WERR) relative to monolingual ASR systems, and LID accuracies relative to acoustic LID classifiers.

This paper employs the RNN-T architecture to build streaming, end-to-end, bilingual systems for joint speech recognition and language identification. We use a lightweight acoustic LID classifier to provide on-the-fly language embeddings to different components of the RNN-T, and demonstrate that providing language information to the joint network performs best. By penalizing language tag emissions and making language predictions using utterance-final posteriors, we show that ASR performance can be improved without impacting LID accuracy. The modeling techniques proposed in this work are language agnostic and can be scaled to multiple languages.

For the English-Spanish language pair, the joint ASR-LID model achieves comparable performance relative to the monolingual ASR and acoustic LID systems. In the case of English-Hindi, we observe a slight performance degradation for English owing to code switching effects combined with the presence of dual (Latin and Devanagari) word representations. For Hindi, the joint model surpasses baseline ASR performance while almost matching LID accuracy. The experimental evidence from two significantly different language pairs indicates that joint ASR-LID training is a promising direction to pursue, given the modeling simplification and compute savings it offers.