Skit-S2I: An Indian Accented Speech to Intent dataset

Earlier intent classification pipeline systems transcribe the speech signal with an ASR model and then train a Natual Language Understanding(NLU) model to predict the intents using the text transcripts. These pipeline methods are prone to error propagation due to ASR transcript errors. The text-to-intent NLU models only make use of the semantic meaning of the speech and ignore other paralinguistic information in the speech which could convey the speaker’s intention. The ASR and NLU models are optimized for different metrics separately and may not be the best approach for the intent classification task. This pipeline method also increases the computational requirement and latency of intent classification. Building this pipeline method is also complex as it involves training and testing two different models and the collection of both labeled corpus for training.

End-to-End SLU methods attempt to predict the speaker’s intent directly from the speech signal with a single model without any ASR transcripts. In contrast to cascaded pipeline methods, SLU methods also take advantage of other acoustic features in speech signals, including prosody. The SLU model is optimized for the target metric of intent classification, making training much simpler. SLU models also have the advantage of lesser computational requirements and better latency during deployment. The lack of a large audio dataset with SLU labels tagged still challenges SLU research. There is a need for an SLU dataset across languages, accents, and multiple domains to make the SLU models robust. In this paper, we release the Skit-S2I dataset, the first Indian-accented speech corpus for intent classification tasks in the banking domain recorded via telephony in a conversational tone. We performed an experimental comparison of cascaded pipeline methods and SLU models based on pretrained speech encoders. Also, we attempted to analyze how different pretraining methods and speech features with and without prosody information affect the performance of the SLU intent classification model. We also tried to diagnose the errors in the dataset with datamaps [1].

Air Travel Information System (ATIS) [2] is an audio dataset with semantic labels related to air travel planning, but it is private and expensive to obtain. Snips SLU Dataset has both English and French languages, with only 2.9k and 1.2k samples in the voice assistant domain. Many previous studies have used transfer learning [3] or acoustic model pretraining [4] for intent classification and slot prediction on smaller SLU datasets. Recently there have been efforts in curating larger datasets for end-to-end SLU tasks like FSC [5], SLURP [6] and STOP [7]. However, these datasets are all based on the personal assistant domain, and no SLU datasets are available for the Indian-accented English or banking domain. In this paper, we introduce S2IDataset, which contains an Indian accented speech corpus for end-to-end SLU with multiple speakers recorded over the telephony in a conversational tonality.

We experimented with several baselines of both cascaded pipeline(ASR+NLU) and end-to-end SLU models on the Skit-S2I dataset. For the cascaded pipeline, we extract the ASR transcripts using the wav2vec2 [8] model finetuned on commonvoice dataset [9] and whisper small model [10]. Then we trained the pretrained XLMR [11] model using the ASR transcripts to predict the intents.

For the end-to-end SLU models, we experimented with four different pretrained speech encoders. The intents are predicted with a linear classifier after average pooling encoded speech representations. We used the wav2vec2 [8] and Hubert [12] models pretrained on the self-supervised learning tasks. We also experimented with two sizes of the whisper encoder, base and small. All SLU models encode the audio with the encoders of the pretrained models. The models are finetuned with a learning rate of $1e^{-5}$ with Adam Optimizer.

Table 3 shows the cascaded pipeline and end-to-end SLU baseline results. The XLMR NLU model trained on different ASR transcripts gives different test accuracy and F1 scores, as the error in the ASR-predicted transcripts can affect the performance of the NLU model, and these models are not optimized together for intent classification.

The Hubert-based end-to-end SLU model performed better than the wav2vec2 model, and both the whisper models outperformed Hubert and wav2vec models. In the pipeline and end-to-end SLU baselines, the whisper model outperforms the other baselines, as the whisper model is more robust and performs better in zero-shot tasks than other models. Table 4 compares the number of parameters of different baseline models. The pipeline baselines have the largest number of parameters as the pipeline depends on two different models, but the end-to-end SLU models are much faster with lesser computation. Whisper-based SLU models are the fastest with the lowest number of parameters as the model input is Mel-Spectrograms which requires much lesser computation than wav2vec2 or Hubert models, which input raw waveform.

Most end-to-end SLU methods use the pretrained ASR features [5] for intent classification or use distillation methods [13] to learn features from pretrained text encoders. The ASR representations will force the model to only use the linguistic/semantic information in the speech signal and ignore other important information, such as prosody. Stressing of different syllables of a word can lead to different meanings, and the overall intonation contour contributes to the speaker’s intention, so prosodic features can also help improve the intent classification task.

We compared the SSL pretrained, and ASR finetuned versions of both wav2vec2 and Hubert models for the intent classification tasks in Table 5. The SSL pretrained models slightly improve the test metrics than the ASR finetuned models. [14] also shows that SSL-trained wav2vec and Hubert models can generalize well on tasks related to learning prosody and semantic features from a speech signal, and ASR finetuning of these models lead to a loss of prosodic information in the learned representations. Based on the above results and observations, we can hypothesize that the performance of the SSL pretrained model is slightly better than ASR finetuned model, as the representation of the SSL model contains prosodic information.

We also trained the whisper(small.en) SLU model on the FSC and SLURP datasets to analyze the dataset as a benchmark. From Table 6, the whisper SLU model got an accuracy score of 0.996 on the FSC dataset and 0.765 on the SLURP dataset. The SLURP and Skit-S2I datasets are better intent classification benchmarks than FSC to compare the SLU models, as FSC test scores are much higher with simple baselines.

We performed the dataset cartography [1] analysis on the S2IDataset, to assess the quality and diagnose the dataset. Dataset Cartography leverage the training dynamics of a model trained on a dataset to create the datamaps, which split the dataset into three distinct regions, easy-to-learn, ambiguous and hard-to-learn. The ambiguous samples in the dataset contribute to the out-of-distribution generalization, the easy-to-learn samples play an essential role in the optimization, and hard-to-learn samples often correspond to labeling errors. Figure 2 shows the datamap generated by training the dataset with the Whisper(small.en) model. After generating the datamaps, we found that a few of the data points in the hard-to-learn samples had label noise, speechless audio files, and short speech utterances, which may be because of telephony noise. Figures 5 and 5 show the generated datamaps for the FSC and SLURP datasets, respectively. Most samples in the FSC dataset were in the easy-to-learn region, whereas the SLURP dataset has a good split of data samples across all the regions. Thus SLURP dataset can act as a better benchmark than FSC.

We also generated the datamaps for each speaker in the dataset to identify which speaker’s audio samples had the above issues. We examined the generated datamaps for all the speakers and found that most of the dataset errors were samples from only a few speakers. Figure 3 shows the datamaps generated for speakers 1, 5, 6, and 8. Speakers 6,8 had very few hard-to-learn samples, whereas speakers 1 and 5 had many hard-to-learn samples and data errors. Not all hard-to-learn samples are data errors, but they have a high chance of being an error. So we manually check all the hard-to-learn samples and change/remove them to make the dataset error-free. Based on the dataset cartography analysis, we will remove the errors in the future version of the S2IDataset dataset.

In this work, we released the Skit-S2I dataset, the first Indian-accented SLU dataset for the banking domain. We trained and compared multiple cascaded pipeline-based and end-to-end SLU baselines. We also compared the performance of different models and found that SSL representations perform better than ASR, as SSL representations contain prosodic information. We also compared the performance of the whisper-based intent classification model on FSC and SLURP datasets and found that the baseline model was able to achieve very good performance on the FSC evaluation. Skit-S2I and SLURP are better benchmarks than the FSC dataset.