End-to-End Spoken Language Understanding Without Full Transcripts

We used the standard ATIS training and test sets for our experiments: 4978 training utterances from Class A (context independent) training data in the ATIS-2 and ATIS-3 corpora and 893 test utterances from ATIS-3 Nov93 and Dec94 data sets. The entity labeled text data is found in LDC2019T04, but there are no pointers to corresponding audio files. Only 518 (out of 893) test utterance audio files were found by [7]. We managed to find all 893 test audio files and 4976 (missing 2) training audio files, in the variety of spontaneous speaking mode, recorded with the Sennheiser microphone.

The 4976 training utterances comprise $\sim$9.64 hours of audio from 355 speakers. The 893 test utterances comprise $\sim$1.43 hours of audio from 55 speakers. The data was originally collected at 16 kHz, but we downsampled to 8 kHz to better model telephony use cases and so we could use off-the-shelf ASR models trained on conversational telephone speech. To better train the proposed E2E models, additional copies of the corpus are created using speed/tempo perturbation. The final training corpus after data augmentation is $\sim$140 hours of audio data. To simulate an additional practical use case, we create a second noisy ATIS corpus by adding street noise between 5-15 db SNR to the clean recordings. This $\sim$9.64 hours noisy data set is also extended via data augmentation to $\sim$140 hours. A corresponding noisy test set is also prepared by corrupting the original clean test set with additive street noise at 5 db SNR.

We measure slot filling performance with the F1 score. When using speech input instead of text, word errors can arise. The F1 score requires that both the slot label and value must be correct. For example, if the reference is toloc.city_name:new_york but the decoded output is toloc.city_name:york, then we count both a false negative and a false positive. It is not sufficient that the correct slot label is produced: no “partial credit” is given for part of the entity value (york) being recognized. The scoring ignores the order of entities, and is therefore suitable for the “bag-of-entities” case we study. Our scoring script was tested on text-input systems and gave identical values as the standard scoring scripts.

To allow the SLU model to process entities and corresponding values independent of an external language model, we first construct a word CTC model on general purpose ASR data with the recipe steps presented in [26, 27] and using 300 hours of Switchboard (SWB-300) data. We then explore different training recipes to build CTC based SLU models.

Our first experiment assumes that we have both verbatim transcripts and entity labels for the SLU data and uses all three training steps. The ASR-SLU adapt step is performed as follows. The output layer of the ASR model, which estimates scores for 18,324 word targets and the blank symbol, is replaced with a randomly initialized output layer that estimates scores for 18,642 word/entity targets and the blank. The weights of the remaining 6 LSTM layers, each with 640 units per direction, and a fully connected bottleneck layer with 256 units are kept the same. The model is then trained on a combined data set of 300 hours of SWB GP-ASR data and 140 hours of clean ATIS data. Note that in this step, although the output layer has units for entity labels, the training targets are only words. In the joint ASR+SLU step, entity labels are introduced into the training transcripts and a joint ASR-SLU model is trained on the SWB+SLU data, starting from the final weights from the ASR-SLU adapt step. In the third and final fine tune SLU step, the joint ASR-SLU model is fine tuned on just the 140 hours of ATIS SLU data.

In experiment $[$1A$]$ of Table 1, we evaluate this model on the clean test ATIS data. Given that the SLU model is a word CTC model, we do not use an external LM while decoding; instead, a simple greedy decode of the output is employed. This initial model has an F1 score of 91.7 for correctly detecting entity labels along with their values. In experiment $[$2A$]$, we develop a similar SLU model with full verbatim transcripts along with entity labels, but we skip the ASR-SLU adapt and joint ASR+SLU adapt steps. We initialize the model with the pre-trained SWB ASR model and directly train the SLU model. This model also achieves 91.7 F1 score, suggesting that the curriculum learning steps may not always be required.

In the next set of experiments we investigate how important verbatim transcripts are for the training process. After the joint ASR+SLU step of experiment $[$1A$]$, in experiment $[$3A$]$, we train an SLU model that recognizes just the entity labels and their values in spoken order. We observe that the model learns to disregard words in the signal that are not entity values, while preserving just the entity values along with their labels. This model performs slightly better than full transcript model in $[$1A$]$. We extend this experiment in $[$4A$]$ by removing the use of transcripts entirely in the training process. This SLU model, after being initialized with a pre-trained ASR model, is trained directly to recognize entity labels and their values without any curriculum learning steps or verbatim transcripts. The model drops slightly in performance, but remains on par with the baseline systems. Finally, we train SLU systems on the much harder task of recognizing alphabetically sorted entity labels and their values. After the joint ASR+SLU step of experiment $[$1A$]$, in experiment $[$5A$]$ we train an SLU model that recognizes just the entity labels and their values, but now in alphabetic order. In experiment $[$6A$]$ a similar model is trained, but without any curriculum learning steps. On this task, the performance of the CTC model drops significantly as it is unable to learn from reordered targets. With the curriculum learning steps, the results in $[$5A$]$ are better, but still much worse than the baselines.

The attention models for SLU are initialized with a state-of-the-art ASR model developed for standard Switchboard ASR task. This model uses an encoder-decoder architecture in which the encoder is an 8-layer LSTM stack using batch-normalization, residual connections, and linear bottleneck layers [28, 29, 30, 31]. The decoder models the sequence of BPE units estimated on characters [32], and consists of 2 unidirectional LSTM layers. One is a dedicated language-model-like component that operates only on the embedded predicted symbol sequence, and the other jointly processes acoustic and symbol information. The decoder applies additive, location aware attention [33], and each layer has 768 unidirectional LSTM nodes. As has been shown in [34], exploiting various regularization techniques, including SpecAugment, sequence-noise injection, speed-tempo augmentation, and various dropout methods [35, 36, 37, 38, 39, 40], results in state-of-the art speech recognition performance using this single-headed sequence-to-sequence model.

To recognize entities, the ASR model is adapted similar to the CTC model, following the steps of Section 3. In contrast to the CTC model, which uses word units, the attention model uses a smaller inventory of 600 BPE units and relies on the decoder LSTMs to model longer sequences — the attention based model has an inherent long-span language model. After the initial ASR model is trained on Switchboard, the subsequent adaptation and transfer learning steps used only the ATIS data without any Switchboard data. Because the attention model operates at the sub-word level, and all new words appearing in the ATIS transcripts can be modeled using these sub-word units, no extension of the output and embedding layer is needed in the first ASR-SLU adapt step. We skip the joint ASR+SLU step and proceed directly to the fine tune SLU step, where the output and the embedding layers of the decoder must be extended with the entity labels. The softmax layer and embedding weights corresponding to the entity labels are randomly initialized, while all other parameters, including the weights which correspond to previously known symbols in the softmax and embedding layers, are copied over from the ASR model. Having no out-of-vocabulary words, sub-word level models are ideally suited to directly start the adaptation process with step 3 of Section 3. All adaptation steps use 5 epochs of training.

The last column of Table 1 shows the slot filling F1 score for attention based SLU models. In experiment $[$1A$]$, an attention based ASR model trained on Switchboard-300h is first adapted on the clean ATIS data to create a domain specific ASR model. On the test set, the word error rate (WER) using the base SWB-300 model is about 7.9% which improved to 0.6% after adaptation. This ASR model is then used as an initial model for transfer learning to create an SLU model. The F1 score is comparable to that of the CTC model. In experiment $[$2A$]$, we skip the ASR adaptation step and directly use the SWB-300 ASR model to initialize the SLU model training. In this scenario, there is no degradation in F1 score. There is no difference in SLU performance whether the model is initialized with a general purpose SWB-300 ASR model (WER=7.9%) or with a domain adapted ASR model (WER=0.6%).

We next consider the effects of training transcription quality or detail. Using transcripts that contain only entities in spoken order ($[$4A$]$), we obtain F1 scores that are almost the same as using full transcripts ($[$1A$]$). When training transcripts contain entities in alphabetic order (possibly different from spoken order) ($[$6A$]$), there is a 2% degradation in F1 score, from 92.9 to 90.9. This result is much better than that for the CTC model (73.5), reflecting the re-ordering ability of attention based models. As before, adding an extra step of ASR model adaptation ($[$3A$]$ and $[$5A$]$) with verbatim transcripts made little difference. This is encouraging, since we are assuming we are only given entities in the training transcripts.

Figure 1 shows the attention plots for the utterance “i would like to make a reservation for a flight to denver from philadelphia on this coming sunday” with three different attention models: (a) ASR model, (b) SLU in spoken order, and (c) SLU in alphabetic order. The attention for (b) is largely monotonic with attention paid on consecutive parts of the audio signal corresponding to BPE units of keywords in the entities. There are gaps reflecting skipping over non-entity words. In (c), the attention is piece-wise monotonic, where the monotonic regions cover the BPE units within a keyword. Since the entities are given in an order different from spoken order, the plot shows how the model is able to associate the correct parts of the speech signal with the entities. In addition, at around 2 seconds, attention was paid to the phrase “make a reservation” which is predictive of the overall intent of the sentence “flight.” (Intent recognition is left out of this paper for simplicity and due to lack of space.)

In a next set of experiments (Table 2), we use the noisy ATIS corpus as the SLU data set and repeat the CTC based experiments conducted earlier. This set of experiments introduces additional variability to the training procedure with realistic noise in both training and test. Further, it increases the acoustic mismatch between the transferred model and the target domain. The general trends for the CTC model observed in Table 1 are also observed in Table 2: (a) ASR transcript based curriculum training is effective; and, (b) entity labels can be recognized reasonably well in spoken order, but the performance is worse when the entity order is different. In experiments like $[$2B$]$, the mismatch between the SLU data and the ASR data affects the performance of models that are only initialized with mismatched pre-trained models and have no other adaptation steps. The noise distortion in general causes these systems to drop in performance compared to the performance results in matched conditions.

Looking at the results in Table 2 for attention based SLU models in more detail, we note that there is an absolute degradation of 4.3% in F1 score when we compare a model trained on full transcripts ($[$1B$]$ F1=92.0) to one trained on entities in alphabetic order ($[$6B$]$ F1=87.7%). While this is a significant drop in performance, it is much better than the CTC result of ($[$6B$]$ F1=68.5). Compared to the clean speech condition, we also come to a different conclusion regarding the utility of ASR adaptation. We see about 1% improvement in F1 score when we are able to use an adapted ASR model instead of the base SWB-300 model to initialize SLU model training. On the noisy test set, using the base SWB-300 model results in WER=60%, whereas the ASR model adapted on noisy ATIS data gives WER=5%. It is remarkable that using these two very different ASR models to initialize the SLU model training leads to only a 1% difference in F1 scores for the final models.

Table 3 shows how the amount of data used to train the ASR model for initializing SLU training affects the final F1 score. Here we show only results for attention-based SLU models trained on entities in spoken order for clean speech. We saw earlier that ASR adaptation on domain data does not always help. But here, using 2000h instead of 300h for the initial ASR model improves the F1 score by about 1%, most likely due to increased robustness of the model to unseen data: the unadapted WER on the ATIS test set is 3.1% (SWB2000h) vs. 7.9% (SWB300h). In contrast, when we directly train the SLU model from scratch, the best we could do was about F1=78.1. When SLU data is limited, these experiments demonstrate the importance of ASR pre-training on a broad range of speech data, not necessarily related to the final SLU task.