End-to-End Bengali Speech Recognition

Traditional automatic speech recognition systems spearheaded by HMM-GMM based acoustic models, where HMMs were used for alignment and GMM mapped these aligned tri-phones to output characters, have dominated the field for quite some time, and only recently were surpassed by HMM-DNN systems, with deep neural networks replacing GMMs.

The advent of sequence-to-sequence models (Seq2Seq) paired with the development of Connectionist Temporal Classification (CTC) [6] revolutionized the end-to-end approach for training speech recognition systems by training a single network that directly mapped audio sequence to the output text, with recurrent neural network ([6], [7], [1]) and attention ([8], [9]) based end-to-end models setting state-of-the-art results on several benchmark datasets.

Most recent deep learning based end-to-end systems utilize log-spectrogram or MFCC features as inputs to the network. Given the innate capability of deep neural networks to learn features from raw inputs, multiple recent works have switched to learning features from raw speech signal. To this end, [10] recently proposed time-delay convolution based trainable filterbanks, observing comparable performance with melfilterbank based systems on the WSJ dataset.

Languages spoken in the Indian subcontinent are a topic of interest primarily due to the sheer number of speakers, but the lack of large-scale, publicly available datasets pose a challenge in the development and evaluation of deep learning based systems.

Most recently, a multilingual phone recognition system was proposed by [11], covering four Indian langauges, viz. Kannada, Telugu, Bengali and Odia. They proposed a traditional HMM-DNN based phone recognition system leveraging Articulatory Features (AFs). Their DNN subsystem utilized tanh activation function at the hidden layers, and was trained in a greedy layer-by-layer supervised setting. However, unlike [11], which proposed a phoneme recognition system evaluated on a very small dataset (19.2 hours total), we evaluate and propose CNN-RNN based end-to-end systems evaluated on a very large dataset.

Addressing the challenges involved in building robust speech recognition systems for Indian languages, [12] proposed training phoneme recognition systems using articulatory movements captured by an electromagnetic articulograph. They evaluated multiple phoneme recognition models, similar to [11], with the addition of HMM-RNN based acoustic model. However, they collected a small, 2 male speaker dataset for each language covered, which is an inconclusive sample size to comment on the genericity and applicability of the approach.

Fig. 1 depicts the general outline of the CNN-RNN based end-to-end speech recognition models that are popular in recent literature. ([1], [7]). The convolution block represents n sized stack of convolution layers. Fig. 2 shows the original 2-layer convolution stack as proposed in [1], where each convolution layer is followed by a Batch Normalization layer [13] and HardTanh non-linearity.

The initial layers of a CNN based deep neural network are of significant importance in terms of overall network complexity as well as performance. The initial layers consume the most memory and comprise of the most floating point operations [2] due to larger spatial dimensions of the input which is gradually downsampled with the depth of the network. Convolutional neural networks comprising solely of kernels with very small receptive fields (3x3 [2], 7x7 [3]) have significantly pushed the state-of-the-art on various computer vision tasks and challenges such as ImageNet and COCO, and are widely used in recent works on cnn design. ([3], [4], [14], [15], [16], [17], [18], [19]), and have various benefits over larger filters, viz,

1. 1.

   Smaller kernels are more parameter efficient. [2]

2. 2.

   Multiple smaller kernels stacked over each other effectively increase the discriminative power of the decision function [2], by increasing the total number of non-linearities.

3. 3.

   Larger convolution filters operated at a stride on the very first layer greatly reduce spatial resolution of filter maps, which might be too drastic and might hurt the performance of the network.

We propose 2 convolution blocks with 2 and 4 convolution layers, henceforth referred to as Block A and B, respectively (Fig. 3). Instead of using large rectangular filters similar to [1], we utilize smaller 3x3 filters for all but the first layer (which uses 7x3 filters) in all our layers.

Table.2 depicts the total number of floating point operations (FLOPs) in millions and the number of parameters for each proposed convolution block as well as the 2-layer convolution block as proposed in DeepSpeech 2. As evident from the aforementioned table, the proposed convolution blocks have significantly lower FLOPs and parameters.

We propose and evaluate 7 network configurations of varying complexities as listed in Table 3. 3 distinct RNN stacks are evaluated with 3, 4 and 5 bidirectional GRU layers respectively. Despite challenges in deployment in an online setting, bidirectional recurrent neural networks routinely outperform similar unidirectional models [1]. Column RNN Config depicts number of bidirectional layers, followed by the number of hidden units per layer. All configurations have 512 hidden units per bidirectional GRU layer, except B-5GRU-Large, which has 800 hidden units.

Following [1], we use normalized log-spectrogram calculated using a sliding window of width 20ms and stride of 10ms, followed by a 160 point FFT as inputs to the network. The network is trained end-to-end using the CTC loss function [6], which facilitates prediction of character sequences directly from input.

Training was performed on a single GTX 1080 Ti system with a batch size of 20 using Stochastic Gradient Descent, with an exponential learning rate decay schedule and an initial learning rate of $3e-4$. Early stopping [20] with a patience of 3 was used for regularization, and all the networks were trained for a maximum of 30 epochs.

Inference: At inference time, we pair the proposed models with a 4-gram Kneser-Ney based language model with a beam size of 100, trained on a large corpus of Bengali text collected from Bengali news sites and blogs, using the Kenlm toolkit [21]. Results without the language model are also provided for the test set.