Multilingual and code-switching ASR challenges for
low resource Indian languages

Misalignments: Each spoken tutorial came with transcriptions in the form of subtitles with corresponding timestamps. These timestamps were more aligned with how the tutorial videos proceeded, rather than with the underlying speech signal. This led to misalignments between the transcription and segment start and end times specified for each transcription. While this is an issue for training and development segments, transcriptions corresponding to the blind test segments were manually edited to exactly match the underlyinto remove valid Hindi words and fix any transliteration errors.

Inconsistent script usage: There were multiple instances of the same English word appearing both in the Latin script and the native scripts of Hindi and Bengali in the training data. Given this inconsistency in script usage for English words, the ASR predictions of English words could either be in the native script or in the Latin script. To allow for both English words and their transliterations in the respective native scripts to be counted as correct during the final word error rate computations, we introduce a transliterated WER (T-WER) metric along with the standard WER metric. While the standard WER will only count an edit as correct if it is an exact match with the word in the reference text, T-WER wil count an English word in the reference text as being correctly predicted if it is in English or in its transliterated form in the native script. We compute T-WER rates for the blind test audio files. To support this computation, the blind test reference text was manually annotated such that every English word only appeared in the Latin script. Following this, every English word in the reference transcriptions was transliterated using Google’s transliteration API and further manually edited to remove valid Hindi words and fix any transliteration errors. This yielded a list of English to native script mappings and this mapping file was used in the final T-WER to map English words to their transliterated forms.

Punctuations: Since our data is sourced from spoken tutorials on coding and computer science topics, there are many punctuations (e.g., semicolon, etc.) that are enunciated in the speech. This is quite unique to this dataset. There were also many instances of enunciated punctuations not occurring in the text as symbols but rather as words (e.g., ’slash’ instead of ’/’, etc.). There were many non-enunciated punctuations in the text too (exclamation, comma, etc.). We manually edited the blind test transcriptions to remove non-enunciated punctuations and normalized punctuations written as words into their respective symbols (e.g. + instead of plus).

Mixed words: In the Bengali-English transcriptions, we saw multiple occurrences of mixed Bengali-English words (which was unique to Bengali-English and did not show up in Hindi-English). To avoid any ambiguities with evaluating such words, we transliterated these mixed words entirely into Bengali.

Incomplete audio: Since the lecture audios are spliced into individual segments, sometimes a few of the segments have incomplete audio either at the start or at the end of the utterances. For the blind test audio files which went through a careful manual annotation, such words were only transcribed if the word was mostly clear to the annotator. Else, it was omitted from the transcription.

Merged English words: Since the code-switching arises mostly due to technical content in the audio, there are instances of English words being merged together in the transcriptions without any word boundary markers. For example, words denoting websites and function calls generally have two or more words used together without spaces. For the blind test transcriptions, we segregated such merged English words into their constituents in order to avoid ambiguous occurrences of both merged and individual words arising together.

Hybrid DNN-HMM: ASR model is built using the Kaldi toolkit with a sequence-trained time-delay neural network (TDNN) architecture optimized using the lattice-free MMI objective function [19]. We consider an architecture comprising 6 TDNN blocks with dimensionality of size 512.

Lexicon: A single lexicon is used containing the combined vocabulary of all six languages. For each language, the lexicon’s entries are obtained automatically, considering a rule-based system that maps graphemes to phonemes. For the mapping, we consider the Indian speech sound label set (ILSL2) [15].

Language model (LM): A single LM is built considering the text transcriptions belonging to the train set from all six languages. For the LM, we consider a 3-gram language model developed in Kaldi using the IRSTLM toolkit. Since the LM has paths that contain multiple languages, the decoded output could result in code-mixing across the six languages.

Hybrid DNN-HMM: The ASR model is built using the Kaldi toolkit, the same baseline architecture being used for both Hindi-English and Bengali-English subtasks. We use MFCC acoustic features to build speaker-adapted GMM-HMM models and similar to subtask1, we also build hybrid DNN-HMM ASR systems using TDNNs comprising 8 TDNN blocks with dimension 768.

End-to-end ASR: The hybrid CTC-attention model based on Transformer  [20] uses a CTC weight of $0.3$ and an attention weight of $0.7$. A $12$-layer encoder network, and a $6$-layer decoder network is used, each with $2048$ units, with a $0.1$ dropout rate. Each layer contains eight $64$-dimensional attention heads which are concatenated to form a $512$-dimensional attention vector. Models were trained for a maximum of $40$ epochs with an early-stopping patience of $3$ using the Noam optimizer from [20] with a learning rate of $10$ and $25000$ warmup steps. Label smoothing and preprocessing using spectral augmentation is also used. The top 5 models with the best validation accuracy are averaged and this averaged checkpoint is used for decoding. Decoding is performed with a beam size of $10$ and a CTC weight of $0.4$.

Lexicon: Two different lexicon files each for Hindi-English and Bengali-English subtasks are used. First, the set of all the words in the training set i.e. the training vocabulary is generated. If the word is a Devanagari/Bengali script word, the corresponding pronunciation is simply the word split into characters. This is because both languages have phonetic orthographies. For English lexicon mappings, an open source g2p package (https://github.com/Kyubyong/g2p) is used. This spells out numbers, looks up the CMUDict dictionary [21], and predicts pronunciations for OOVs as well. We also map punctuations to their corresponding English words since these punctuations are enunciated in the audio.

Language model: Two separate language models are built for each of the code switching subtasks. For LM training, we consider a trigram language model with Kneser-Ney discounting using the SRILM toolkit developed in Kaldi [22].

Comparing multilingual and monolingual ASR: Table 3 shows the WERs obtained for test and blind test sets for each of the six languages along with averaged WER across all six languages. We show that the WER obtained with multilingual ASR is lower for Tamil. Though the WER from the multilingual ASR system is higher in the remaining languages, it does not require any explicit language identification system. Thus, the performance of monolingual ASR is affected by the effectiveness of LID for the Indian context. Further, it is known that multilingual ASR is effective in obtaining a better acoustic model by exploring common properties among the multiple languages. However, the performance of the multilingual ASR also depends on the quality of the language model, which, in this work, could introduce noise due to code-mixing of words.