Word-Free Spoken Language Understanding for Mandarin-Chinese

Spoken dialogue systems such as Siri and Alexa have demonstrated their effectiveness in various spoken language processing tasks including intent recognition and slot filtering. They provide great convenience to people’s everyday life. However, current spoken language understanding (SLU) systems face the challenge of insufficient annotated natural speech data. Specifically, a typical SLU pipeline consists of an automatic speech recognition (ASR) module followed by a natural language understanding (NLU) module. The ASR module requires a large amount of labeled data for each language and thus forms a bottleneck.

Gupta et al. (2021) proposes a novel system that replaces the ASR module with Allosaurus, which is a universal phone recognition system that provides phonetic transcript for the input audio. Inspired by this acoustic-only approach, we propose an acoustic-based SLU system using the Transformer model. Our SLU system is similar to theirs in that we both train our systems on phonetic transcriptions and does not depend on any data-demanding language-specific modules such as Automatic Speech Recognition (ASR) module. However, while their work converts text-based data to synthetic speech in order to generate the phonetic transcriptions, our work operates on the natural speech data directly. To the best of our knowledge, our work is the first attempt that uses discovered phonetic units from natural speech instead of synthesized speech. Meanwhile, we use a transformer-based architecture instead of a CNN+LSTM based one in our NLU module. We then verify the effectiveness of our Transformer-based SLU system on a small intent classification dataset in Mandarin Chinese. We conclude that our Transformer-based Spoken Language Understanding (SLU) system has slightly worse performance on phone input than the state-of-the-art architecture.

Nowadays, Spoken Language Understanding (SLU) systems, an emerging field in between the areas of speech processing and natural language processing, have been widely used to process spoken utterances for various downstream tasks. Current researches in high resourced languages focus on building end-to-end (E2E) SLU systems (Radfar et al., 2020; Lugosch et al., 2019) to eliminate errors through the pipeline. Traditional SLU pipelines include an automatic speech recognition (ASR) module followed by a natural language understanding (NLU) module. The language-specific ASR module converts spoken utterance into textual transcription, which is then fed into the NLU module for downstream tasks. Training ASR often requires a large amount of language-specific data and therefore poses an obstacle for applying traditional SLU pipeline on low-resourced languages.

In our work, we replace the ASR module in the traditional SLU pipeline with Allosaurus (Li et al., 2020), a universal phone recognition system. Allosaurus provides language-independent and fine-grained phonetic transcriptions for input natural speech, which enables it to generalize well to low-resourced languages and novel sounds (Gupta et al., 2021, 2020). In our system, we feed the phonetic transcriptions by Allosaurus to a Transformer-based language model, which is trained to understand the phonetic transcription and perform downstream tasks like intent classification.

To the best of our knowledge, there have been limited attempts to use the phonetic units for intent classification in a supervised fashion. While the previous work (Gupta et al., 2021) proposed an architecture based on CNNs (convolutional neural network) and LSTMs (long-short term memory), we present the first attempt to combine Transformer-based architecture with intent recognition task using discovered phonetic units from natural speech. In addition, we use a BERT-based pre-training method called RoBERTa (Liu et al., 2019). The Transformer-based architecture (Vaswani et al., 2017) provides a better-structured memory for handling long-term dependencies in contents, resulting in robust performance across diverse downstream language understanding tasks; and the BERT-based pre-training procedure (Devlin et al., 2019) gives bidirectional contextual representations of the phonetic units.

We evaluate our model with two metrics: accuracy and F1-score,

where $n_{c}$ is the number of correct classifications, $n$ is the number of total classifications,

where TP, FP, FN are the number of true positives, false positive, and false negative, respectively. Because of the imbalance of the intent classification dataset, we report the macro-averaged F1-score (the average of F1-scores for all classes) in this paper. We also present a detailed discussion on the selected datasets in Section 5.

In our system, there are two stages of training: pre-training and fine-tuning. For the pre-training stage, we use THCHS-30 dataset released by Tsinghua University (Dong Wang, 2015); after pre-training, we fine-tune the model to perform intent classification on the dataset from the first CATSLU Challenge (Zhu et al., 2019).

The CATSLU dataset contains speech recordings and text transcripts, annotated with semantic intent labels, between users (humans) and service assistants (robots). To obtain finite number of labels and phonetic forms of input, we conduct the following steps to prepare the data:

1. For each text transcript in the dataset, if its corresponding audio recording is available, input it raw audio clip into Allosaurus for phonetic transcripts; otherwise, discard it.

2. Map each semantic intent annotation to numeric class labels. For example, "DecreaseBrightness" = 0.

3. Review the most common labels as well as the balance of labels within each dataset, downsampling/upsampling until reasonable balance is achieved.

The baseline results on the test set are shown in Table 4. For the baseline model (CNN + LSTM), we obtain an accuracy 63% on the intent classification dataset. In addition, we perform various experiments on the baseline architecture on the development set to see if they will give accuracy improvement. Experiments include increasing the hidden dimension of LSTM (e.g. to 256, 512, 1024), increasing the number of LSTM layers, increasing the embedding dimension, and increasing the number of filters or the filter size for CNN. However, we found that they do not give much accuracy improvement and will degrade the performance in many cases.

We present a transformer-based language model with the masked language modelling pre-training objective used in RoBERTa. We summarize our results in the following sections.

In this paper, we propose a novel transformer-based acoustic-only SLU system. Our work is the first attempt to apply BERT-based model on phonetic transcripts. Our work is also the first attempt to use the phonetic units from natural speech for intent classification. We test both the previous method (CNN + LSTM) and our system on the CATSLU dataset, which contains natural speech instructions in smart light domain and smart assistant domain respectively. We found our system shows reasonable performance, though slightly worse than the SOTA model due to the data-demanding nature of BERT-based models. In addition, we found pre-training improves both the performances of both the previous method and our system. Future work may replicate our general SLU pipeline, and further fine-tune the architecture on larger intent classifications datasets as well as low-resourced language datasets.