Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition

The performance of an automatic speech recognition system is contingent on its core components including acoustic feature extraction, mapping acoustic features to tokens using the Gaussian Mixture Model/Hidden Markov Model (GMM-HMM) or neural networks, and language model-based rescoring of the outputs from connectionist temporal classification (CTC), etc Graves et al. (2006). As for segmentation, either word-level or subword-level tokens can be modeled to build ASR systems. However, word-level modeling faces a challenge due to the vast number of words in a language, which exceeds the typical vocabulary size of an ASR system. As a result, word-level modeling is susceptible to the out-of-vocabulary (OOV) problem Livescu et al. (2012). An OOV word refers to a word that was not encountered during model training but appears during the inference phase.

Subword units are known as effective solutions to tackle the OOV issue in natural language processing (NLP) Sennrich et al. (2016). Examples of subword units include phonemes, characters, unigrams, and byte pair encoding (BPE) tokens, etc Kudo (2018); Gage (1994); Sennrich et al. (2016). Phonemes represent the smallest units of sound, and after training with phonemes, a model can infer new words. Creating a lexicon that maps each word to its corresponding phonemes, however, requires domain expertise as well as a substantial amount of time and effort for manual annotation. Character-based ASR models are easier to develop since mapping between words and their corresponding characters can be done using an automated way. Furthermore, training a model with a limited number of characters in a language is more computationally efficient than training a word-based model. Unigram language modeling is another segmentation technique that removes tokens based on language model perplexity, initially applied in machine translation Kudo (2018).

BPE subword tokenization is first utilized in neural machine translation (NMT) and has gained widespread usage due to their ability to handle OOV words effectively Gage (1994); Sennrich et al. (2016). Subsequent studies implement BPE-based subword modeling in the speech processing domain Synnaeve et al. (2020); Yusuyin et al. (2023). For BPE, the number of merge operations determines the number of generated tokens/subwords. From the work of Gutierrez-Vasques et al. (2023) on 47 diverse languages, it has been found that in languages characterized by extensive inflectional morphology, there is a tendency to generate highly productive subwords during the initial merging steps. Conversely, in languages with limited inflectional morphology, idiosyncratic subwords tend to play a more prominent role. Therefore, the characteristics of subwords and the required number of merges in BPE tokenization are contingent upon the morphological nature of the respective language. Moreover, an empirical study is conducted by incrementally increasing the BPE merges going from characters to words Gutierrez-Vasques et al. (2021). The authors found that around 200 BPE merges result in the most similar distribution across different languages.

Since the optimal number of BPE merges cannot be universally determined for different languages with varied types of morphology Gutierrez-Vasques et al. (2023), in this study, we empirically determine the number of BPE merges needed for a highly inflectional language—Bengali to achieve superior out-of-distribution ASR performance. Bengali is an Indo-Aryan language spoken in Bangladesh and India and poses challenges to the development of robust ASR systems due to its intricate morphological forms and inadequate research Ali et al. (2008); Samin et al. (2021). Subsequently, we compare the results of BPE-based tokenization with segmentation approaches based on characters and unigrams by performing a cross-dataset evaluation, aiming to understand their effectiveness for handling out-of-distribution data. This investigation exploiting different subword modeling approaches is conducted for the first time for Bengali ASR, to the best of our knowledge.

The rest of the paper is structured as follows: Section 2 provides a comprehensive background study on BPE and unigram language modeling, along with a review of related work in the speech processing domain. In Section 3, we describe the methodology of our experiments. Section 4 provides details about the experiment setup. Results are discussed in Section 5. Section 6 concludes the article and outlines the future directions.

We train a convolutional neural network (CNN) based acoustic model for performing the experiments. We extract 21 mel-frequency cepstral coefficients (MFCCs) from each frame of the input signal and feed it to the CNN. The frame length and stride are 30 ms and 15 ms, respectively. We implement the same CNN architecture following the work of Samin et al. (2021) except for introducing a batch normalization layer in each convolution block. Moreover, we use 20 convolutional blocks instead of 15. The objective of the acoustic model is to predict the subword units based on the CTC loss criterion given the audio signal Graves et al. (2006). We choose either character, BPE, or unigram tokens in individual experiments as our subwords.

We do not perform beam search decoding with a language model since it can have an impact on the final result. The goal of this study is to investigate different subword-based acoustic modeling for a morphologically rich language Bengali, so we exclude the language modeling part. Therefore, greedy search decoding is used to generate the output tokens.

We implement CNN-based acoustic models using the Flashlight toolkit Kahn et al. (2022). We train our CNNs using SUBAK.KO, an annotated Bangla speech dataset Kibria et al. (2022). SUBAK.KO is mostly a read speech corpus with 229 hours of read speech and only 12 hours of broadcast speech. We use the same 200-hour long training set, 20-hour long development (dev) set, and 20-hour long evaluation (eval) set following Kibria et al. (2022). Using standard train, dev, and eval sets enables us to compare our strategy to those of the past. For a comprehensive evaluation, we use a 20-hour subset of the large Bangla automatic speech recognition training data (LB-ASRTD) and the 20-hour long full SHRUTI corpus Kjartansson et al. (2018); Das et al. (2011). Our SUBAK.KO-based ASR model encounters OOV words from LB-ASRTD and SHRUTI out-of-distribution data. Therefore, cross-evaluation assures a more reliable evaluation of various subword modeling algorithms.

We train baseline ASR systems using character and unigram tokens, subsequently contrasting their performance with BPE-based ASR systems. For character-based modeling, we simply use Python programming language to segment a word into individual characters. To build the BPE and unigram-based lexicons, we use the Sentence-piece library Kudo and Richardson (2018). For unigram language modeling, a fixed token size of 1000 is used. As for BPE, we develop four acoustic models with 500, 1K, 2K, and 3K tokens and compare the results. SUBAK.KO train set is used as a text corpus to generate the BPE and unigram tokens.

For evaluating ASR models, we employ standard metrics such as the word error rate (WER) and the token error rate (TER). Eight graphics processing units (GPUs) with only 12 gigabytes of virtual random access memory each are used to train the models.

Table 1 presents the WERs and TERs for various subword modeling types and token sizes. On all three evaluation sets, BPE-based acoustic modeling outperforms both character-based and unigram-based modeling in terms of WERs. Unigram modeling achieves lower WERs than character-based segmentation. With the same number of tokens (1000 tokens), unigram modeling cannot surpass the BPE-based approach when dealing with both in-distribution (SUBAK.KO eval) and out-of-distribution (LB-ASRTD and SHRUTI eval) data.

The number of generated tokens is proportional to the number of BPE merge operations. As we increase the number of BPE tokens, the WERs continue to decrease on the SUBAK.KO eval set. Nevertheless, acoustic models trained with 1000 BPE tokens and 500 BPE tokens achieve reduced WERs on the LB-ASRTD and SHRUTI eval sets, respectively. Notably, BPE tokens are generated utilizing the SUBAK.KO train corpus. This implies that as the number of BPE tokens increases, the model becomes overfit on its eval set while performing poorly on the out-of-distribution data. Thus, a target BPE token size of 500 or 1000 is found to be suitable to achieve better generalizability for ASR. This finding conforms to the work of Gutierrez-Vasques et al. (2023), indicating that morphologically rich languages necessitate fewer BPE merges, leading to a reduced count of BPE tokens. Also, a higher number of BPE merge operations tends to generate longer-length tokens, which resemble words and present similar bottlenecks of word-based tokenization.

With regard to TERs, character-based acoustic modeling exhibits lower error rates than the other two approaches. Furthermore, with the BPE tokens, the TERs tend to continually increase when we increment the number of BPE tokens. Each character represents a token of length one, while BPE tokens undergo multiple merge operations, resulting in longer token lengths. We argue that predicting a token with a shorter length is a comparatively easier task for the acoustic model in contrast to mapping input signal frames to a token with a longer length. This discrepancy can contribute to higher TERs for unigram and BPE-based modeling. However, if a model can accurately predict the long-length tokens of a word, it increases the probability of achieving a lower WER because the number of unigram/BPE tokens in a word is typically fewer than the number of characters in that word.

Figure 1 demonstrates the performance of BPE-based subword modeling with various train corpus sizes. We can observe the positive impact of increasing the amount of acoustic model training data on all three evaluation sets for the BPE-based approach. However, it is worth noting that BPE-based speech recognition systems can still achieve satisfactory performance even in low-resource scenarios.

In this work, we determine the number of BPE merges in the context of ASR for a morphologically rich language - Bengali and provide intriguing insights into the relationship between BPE merge operations and ASR performance in the presence of OOV words. Furthermore, we provide a comparative analysis for three subword modeling approaches including characters, unigrams, and BPE for ASR. Our empirical study suggests that BPE is a better choice for subword modeling than characters and unigram tokens. Additionally, through cross-dataset evaluation, we find that targeting a token size of approximately 500 or 1000 yields improved generalization and robustness, while excessively high numbers of BPE tokens can result in overfitting. This outcome corresponds with prior linguistic research, suggesting that morphologically rich languages demand fewer BPE merges to yield highly productive BPE tokens, as discussed in Gutierrez-Vasques et al. (2023).

There are several potential directions for future research. Firstly, instead of generating BPE tokens from the text files of SUBAK.KO train set, a large-scale text corpus could be constructed specifically for this purpose, enabling the generation of BPE tokens from a more extensive and diverse dataset. We hypothesize that this approach could yield superior BPE representations, resulting in enhanced robustness across out-of-domain data, particularly for challenging morphologically rich languages. Secondly, we aim to explore additional morphologically rich languages from diverse language families, as well as languages like English that lack complex morphology, to further evaluate the effectiveness of subword modeling approaches. Lastly, although our study benchmarks convolutional neural networks (CNNs), it would be valuable to investigate state-of-the-art transfer learning algorithms, such as wav2vec 2.0 Baevski et al. (2020) and HuBERT Hsu et al. (2021), to determine if BPE subword modeling remains effective with these architectures.