Calibrate and Refine! A Novel and Agile Framework for ASR-error Robust Intent Detection

SDCM aims to calibrate the distorted representations of ASR outputs and minimize the negative impacts brought by semantic drift. To achieve this, inspired by the great success of pretrained language models (PLM) and finetuning techniques, we propose to adopt two PLM finetuning strategies, namely confusion-aware finetuning and task-adaptive finetuning, which are transformed from [20] and [22].

For confusion-aware finetuning, we first use both minimum edit-distance (MED) and word confusion network (WCN) to extract acoustic confusion, which are introduced by [20]. Due to space limitation, readers could check the details from their paper. Taking the two different utterances $x_{1}$ and $x_{2}$ as an example, we use $C=\left\{c_{1},c_{2},\cdots,c_{|C|}\right\}$ to denote the set of all acoustic confusions, where $c=\left\{w_{t_{1}}^{x_{1}},w_{t_{2}}^{x_{2}}\right\}$ consists of two acoustically similar words $w_{t_{1}}^{x_{1}}$ and $w_{t_{2}}^{x_{2}}$. Finally, we propose a new confusion loss to minimize the mean square error (MSE) between the word-level representations and sentence-level representations generated by pretrained language model as follows:

$\displaystyle\mathrm{L}_{\mathrm{ca}}\!=\!\frac{1}{|c|}\sum_{c\in C}\sum_{i=0}^{1}\operatorname{MSE}\left(h_{t_{1},i}^{x_{1}},h_{t_{2},i}^{x_{2}}\right)+\operatorname{MSE}\left(h^{x_{1}},h^{x_{2}}\right)$

(1)

Task-adaptive finetuning is a widely used technique especially when domain mismatch problem happens, which could effectively adapt pretrained LM from general corpus to the target data. For example, given a pre-trained ELMo model and a sentence $x=\left\{w_{1},w_{2},\ldots,w_{|x|}\right\}$, we could directly use the pretraining loss of ELMo as the task-adaptive loss, which could be written as:

$\displaystyle\mathrm{L}_{\mathrm{ta}}=\frac{1}{|x|}\sum_{t=1}^{|x|}-\log p\left(w_{t}\mid w_{<t}\right)-\log p\left(w_{t}\mid w_{>t}\right)$

(2)

where $p\left(w_{t}\mid w_{<t}\right)$ and $p\left(w_{t}\mid w_{>t}\right)$ denote the probabilities of $w_{t}$ calculated from forward and backward directions.

Eventually, we jointly finetune the LM using above-mentioned two strategies in a multi-task learning manner and the final loss is as follows:

$\displaystyle L=L_{\mathrm{ta}}+\lambda L_{\mathrm{ca}}$

(3)

where $\lambda$ represents a balancing hyperparameter to control the contribution of each finetuning strategy.

Phoneme is the smallest pronunciation unit in speech and the phoneme sequence of each word can represent its acoustic information to some extent. Therefore, we design PRM to refine and enrich the calibrated representation by injecting phonemic information into the modeling process.

Firstly, each word $w_{t}$ in the ASR output $x_{asr}$ will be transformed into a phoneme sequence $P_{t}=\left\{p_{1},p_{2},\cdots,p_{N_{w_{t}}}\right\}$ via a grapheme to phoneme (G2P) conversion algorithm [23], which highly depends on the pronunciation dictionary. In this paper we adopt CMU pronunciation dictionary [24] constructed by Carnegie Mellon University, which includes 39 types of phonemes and covers more than 130,000 words as well as their corresponding pronunciation information. Figure 2 also shows the process of converting the word ”find” into a phoneme sequence ”F,AY1,N,D”. Note that for vowels like ”AY”, there is a stress marker behind them indicating which stress types it belongs to. Generally, ”0” represents no stress, ”1” and ”2” represent primary stress and secondary stress respectively, which could provide fine-grained acoustic information for intent detection process. Then, each phoneme sequence will be mapped into the embedding space and be further encoded by a BiLSTM layer as follows:

$\displaystyle H_{w_{t}}=$

$\displaystyle BiLSTM([e_{p_{1}},e_{p_{2}},\dots,e_{N_{w_{t}}}]),$

(4)

where $e_{p_{i}}$ denotes the embedding of the phoneme $p_{i}$ and $H_{w_{t}}$ represents the hidden representation matrix for word $w_{t}$.

In the end, average pooling method is conducted on these hidden representation matrices to obtain the final acoustic embedding of the whole sentence $x_{i}$:

$\displaystyle H_{x_{asr}}^{acoustic}=$

$\displaystyle[h_{w_{1}},h_{w_{2}},\dots,h_{w_{N}}],$

(5)

$\displaystyle h_{w_{t}}=$

$\displaystyle Average(H_{w_{t}}),$

(6)

As shown in Figure 2, the intent detection module is decoupled from other modules, making it possible to incorporate any existing text-based intent detection model to our proposed CR-ID framework. Therefore, we adopt a self-attentive intent classification model inspired by [8] as the ID module. The input of ID module is the concatenation of the calibrated word embedding generated by SDCM and the acoustic embedding generated by PRM. BiLSTM is used to model the long-term dependency in the utterance and the self attention mechanism is adopted to capture the key information from the calibrated and refined representations. Max pooling is utilized to obtain the final sentence-level representation, which is further fed to a softmax classifier to predict user’s intent.

In [20], the authors used three datasets, namely SNIPS, ATIS and Smartlight, for their experiments. However, both ATIS (with confusion words) and Smartlight are not available for public because of the copyright issue. Therefore, for fair comparison with the method proposed in [20], we directly use their released version of SNIPS dataset to conduct all the experiments. Different from the original SNIPS dataset, [20] extracted confusion words via the two strategies introduced in Sec2.1 and added them to the original dataset, which is convenient for researchers to reproduce their results and make improvements on it. The readers could check the details of this dataset in https://github.com/MiuLab/SpokenVec.

A number of ASR error robust ID models have been proposed in the past few years. We do not compare with all of them because many previous methods are not directly comparable due to the use of different model architectures. Hence, we select SpokenVec [20] and construct several baselines that are fair (use the same information, similar architectures, etc.) to compare with. Specifically, we use the intent detection module (introduced in Sec 2.3) as the base model, because the self-attentive intent detection model has already achieved comparative performance on SNIPS dataset according to [25]. Then we incorporate it with different word embedding techniques as the baselines.
Static Word Embedding. We use three pre-trained static word embeddings, Word2Vec [13], GloVe [26] and FastText [27], as the embedding matrix to help encode sentences. We also use a randomly initialized embedding matrix as a comparison.
Contextual Word Embedding. We evaluate two pretrained language models, ELMo and BERT, to obtain contextual word embeddings. And each LM is evaluated with fixed and unfixed parameters respectively.
Implementation Details. For the ID base model, the dimension of BiLSTM and self-attention layer are all set to 300, the number of heads is set to 8, the batch size is 64. All ID base models are trained on the manual transcribed training set for 50 epochs using Adam optimizer with learning rate as 3e-4, and then tested on the manual transcriptions and ASR outputs respectively. For the SDCM, for fair comparison with SpokenVec, we follow its setting and adopt ELMo as the pretrained LM. We train the SDCM for 10 epochs with the batch size of 32, learn rate of Adam set to 1e-4. For PRM, the dimension of the embedding and BiLSTM hidden layers are all set to 50 and the PRM is jointly trained with ID base model.

The overall performance of the baselines and CR-ID are summarized in Table 1. Note that as introduced in Sec 2.1 and Sec 3.1, the confusion word pairs could be generated by minimum edit distance (MED) or word confusion network (WCN). Hence, for SpokenVec and our proposed CR-ID, we also report the performance variations using different confusion extraction methods in Table 1. Here are some observations from the Table 1: when testing on the manual transcriptions, the performance scores of all methods are very close, and the method based on contextual word embedding is slightly better than the static counterparts. However, the performances of all baselines except for SpokenVec drops sharply on the ASR output, demonstrating the necessity to reduce the negative impacts caused by semantic drift problem. CR-ID (WCN) achieved the best performance in terms of both Accuracy and Macro-F1. Specifically, compared with the best static word embedding based baselines, the Accuracy and Macro-F1 of CR-ID (WCN) are increased by 12.39% and 11.96% respectively; compared with the best contextual word embedding based baseline model, the performance are improved by 9.5% and 9% respectively; even compared with the SpokenVec, which is a very strong baseline, the performance gains still achieve 1.99% and 1.86% respectively, demonstrating the effectiveness of our propose CR-ID framwork.

In order to figure out the contribution of different modules in our proposed CR-ID, we conduct ablation study for each plug-and-play module, as shown in Table.2: 1) CR-ID w/o acoustic embedding, which only use SDCM for calibration; 2) CR-ID w/o confusion-aware finetuning strategy, where only task-adaptive finetuning and PRM are reserved. 3) CR-ID w/o task-adaptive finetuning strategy, where only confusion-aware finetuning and PRM are reserved. We observe that all these components contribute to performance improvements when testing on the ASR outputs. Specifically, task adaptive fine-tuning strategy contributes the most to the robustness of ID module. When this strategy is removed from the CR-ID, the accuracy and Macro-F1 are decreased by 9.42% and 8.69% respectively. And the confusion-aware finetuning strategy take the second place. Without it, the accuracy and Macro-F1 are decreased by 2.94% and 2.82% respectively. Without acoustic embedding, the accuracy and Macro-F1 are decreased by 0.99% and 0.96% respectively. Therefore, the combination of SDCM and PRM could significantly improve the robustness of ID module to ASR errors.

In addition, we also explore the effect of different distance functions in the confusion loss on model’s performance. Here we select three classical distance functions (Equation 7,8,9) to subsitute the MSE in Equation 1, and the results are shown in Table 3. We observe that MSE achieves the best performance under most experimental settings, which is the reason why we finally choose MSE distance for the confusion-aware finetuning strategy.

$\displaystyle\mathrm{L}_{\text{cos}}\!=\!\frac{1}{|C|}\sum_{c\in C}\sum_{i=0}^{1}\left(1-\frac{h_{t_{1},i}^{x_{1}}\cdot h_{t_{2},i}^{x_{2}}}{\left\|h_{t_{1},i}^{x_{1}}\right\|\left\|h_{t_{2},i}^{x_{2}}\right\|}\right)+\left(1-\frac{h^{x_{1}}\cdot h^{x_{2}}}{\left\|h^{x_{1}}\right\|\left\|h^{x_{2}}\right\|}\right)$

(7)

$\displaystyle\mathrm{L}_{\mathrm{l1}}=\frac{1}{|C|}\sum_{c\in C}\sum_{i=0}^{1}\left\|h_{t_{1},i}^{x_{1}}-h_{t_{2},i}^{x_{2}}\right\|_{1}+\left\|h^{x_{1}}-h^{x_{2}}\right\|_{1}$

(8)

$\displaystyle\begin{gathered}\mathrm{L}_{\!\text{triplet}}\!=\!\frac{1}{|C|}\sum_{c\in C}\sum_{i=0}^{1}\text{triplet}\left(h_{t_{1},i}^{x_{1}},h_{t_{2},i}^{x_{2}},h_{t_{3},i}^{x_{3}}\right)\!+\!\text{triplet}\left(\!h^{x_{1}},\!h^{x_{2}},\!h^{x_{3}}\right)\\ \text{triplet}(a,p,n)=\max\left\{d\left(a_{i},p_{i}\right)-d\left(a_{i},n_{i}\right)+\text{margin},0\right\}\\ \quad d\left(x_{i},y_{i}\right)=\left\|x_{i}-y_{i}\right\|_{p}\end{gathered}$

(9)

In this section, we aim to analyze the effect of the balancing hyperparameter $\lambda$ (in the Equation 3) on the performance of CR-ID. The results are illustrated in Figure 3. It can be observed that for both CR-ID (MED) or CR-ID (WCN), when $\lambda$ increases from 0.1 to 10, the model performance is slightly improved, but when the $\lambda$ gets larger (e.g. larger than 15), the performance of the model begins to decline. Therefore, for all the CR-ID related experiments, we set $\lambda$ to 10 to better balance the impact of task-adaptive finetuning and confusion-aware finetuning on model optimization.