Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning

Figure 1 illustrates the architecture of the proposed pre-training approach. In this study, we use the manual transcripts of the Librispeech 960hrs data [23] and the Fisher corpus¹¹1Fisher English Training Speech Part 1 LDC2004S13 and LDC2004T19; Part2 LDC2005S13 and LDC2005T19 as the pre-training data set (denoted $\mathcal{D}_{pt}$) for our proposed pre-trained model. We use phoneme labels to represent high-level speech features. Given each sentence $W=w_{1},w_{2},\ldots,w_{m},W\in\mathcal{D}_{pt}$ (where $w_{i}$ denotes the $i$th word), we construct its phone label sequence $P=p_{1},p_{2},\ldots,p_{n}$ by looking up the phoneme sequence for each word $w_{i}$ in a pronunciation dictionary. We use the CMU pronunciation dictionary²²2http://www.speech.cs.cmu.edu/cgi-bin/cmudict. For words that are not covered by the CMU dictionary, we use a special token $<$UNK$>$ to represent its phone sequence. 1.3% words in $\mathcal{D}_{pt}$ receive a $<$UNK$>$ label. The phone sequence for words with missing pronunciations can be generated with a grapheme-to-phoneme (g2p) model. In future work, we plan to explore phone labels generated through forced alignment [9].

Each pair $<$W,P$>$ is one input training sample to the pre-trained model. The input is embedded through the input representation layer as the element-wise sum of token embedding, position embedding, and segment embedding. These embeddings are then fed to a multi-layer bidirectional transformer encoder to learn the joint contextualized representations for the textual and phonetic sequences.

We design three pre-training tasks for learning joint textual-phonetic semantic representations:masked language modeling conditioned on the phone sequence (condMLM), masked speech modeling conditioned on the word sequence (condMSM), and word-speech alignment (WSA), inspired by UNITER [24] which is a single-stream image-text representation model.

Masked language modeling conditioned on the phone sequence. We randomly mask $M\%$ tokens in the text sequence $W$ and replace the masked tokens with a special token $[$MASK$]$. Same as BERT, $M\%$ masking is carried out as 80% $[$MASK$]$ substitution and 10% random word substitution and 10% unchanged. The model learns to use the remaining unmasked tokens in $W$ and the entire unmasked phone sequence $P$ to predict the masked tokens, and the goal is to minimize the cross-entropy loss, denoted $\mathcal{L}_{condMLM}$.

Masked speech modeling conditioned on the word sequence. We randomly mask $N\%$ tokens in the phone sequence $P$ and replace them with $[$MASK$]$. The model then uses the remaining unmasked phones and the entire unmasked sentence $W$ to predict the masked phones. The optimization is conducted by minimizing the cross-entropy loss of this prediction, denoted $\mathcal{L}_{condMSM}$. Previous studies show whole-word masking outperforms WordPiece based masking. In addition, to avoid introducing noise from aligning the phone sequence of a word to its WordPiece tokenization, we use whole-word masking for both words and phones. That is, we mask all tokens corresponding to a word at once and similarly, all phones corresponding to a word at once. Considering the asymmetric complexity of predicting words based on phones and vice versa, we investigate oneMod and twoMod masking strategies. In the oneMod strategy, for each input sample, we randomly choose masking the word sequence or the phone sequence, then conduct random masking on the chosen modality, but not masking both word and phone sequences. In contrast, in the twoMod strategy, both word sequence and phone sequence are randomly masked.

Word-speech alignment. We design a binary classification task to learn word-speech alignment between a text sequence $W$ and a phone sequence $P$. Given $<$W,P$>$ of sentences in $\mathcal{D}_{pt}$ and their phone sequences, we construct “[CLS] W [SEP] P” as positive samples and “[CLS] W [SEP] P_rand” as negative samples, where P_rand is randomly sampled from $<$W’,P’$>$ (W’ $\neq$ W). The hidden state for [CLS] is fed into a softmax classifier to decide whether the word sequence and phone sequence match or not. The training objective is minimizing the softmax cross-entropy loss, denoted $\mathcal{L}_{WSA}$.

The overall training objective is the combination of the objectives of these pre-training tasks. For each positive sample of the WSA task, loss is $\mathcal{L}_{condMLM}+\mathcal{L}_{condMSM}+\mathcal{L}_{WSA}$. For each negative sample of the WSA task, condMLM and condMSM fall back to MLM and MSM, respectively; and loss is $\mathcal{L}_{MLM}+\mathcal{L}_{MSM}+\mathcal{L}_{WSA}$.

We evaluate our proposed approach on two SLU benchmarks: Fluent Speech Commands (FSC) [9] and Snips [28]. FSC includes recordings of 248 unique English command phrases (e.g., “turn the lights off in the kitchen”) to a virtual assistant from 77 speakers. Following prior work [9], the three slot values annotated for each audio file (“action”, “object”, “location”) are combined as the intent of the utterance and we conduct IC on FSC. We decode 1-best for the validation and test sets of FSC with two off-the-shelf Kaldi ASR systems⁴⁴4Sys1: ASpIRE Chain Model: https://kaldi-asr.org/models/m1 and Sys2: Librispeech ASR Model https://kaldi-asr.org/models/m13 with trigram decoding and RNNLM rescoring. Both ASR systems use the CMU dictionary as used for phone label lookup..

The second dataset is Snips [28], collected from the Snips personal voice assistant. Since the original Snips release only comprises of text data without natural speech released, we use the ASR hypotheses for the Snips validation and test sets from [26, 27]⁵⁵5The authors synthesized audio from text using the Google TTS system and decoded with Kaldi Sys1.. Table 1 summarizes the statistics of the datasets.

For the baseline pre-trained model, we use English uncased BERT-Base⁶⁶6https://github.com/google-research/bert, pre-trained on the BooksCorpus [29] and English Wikipedia. We further pre-train BERT on different combinations of condMLM, condMSM, and WSA tasks, as well as masked language modeling (MLM) and next sentence prediction (NSP) tasks used in BERT pre-training. We then conduct SLU fine-tuning on the pre-trained models for evaluation. We also investigate efficacy of oneMod and twoMod masking strategies and different masking percentages for condMLM and condMSM. The maximum sequence length is 256, the batch size is 64, and the number of training steps is 100K. Adam [30] is used for optimization. The initial learning rate is optimized among {1e-4, 5e-5} for pre-training and among {3e-5, 5e-5} for SLU fine-tuning. The dropout probability is 0.1.

For our proposed pre-training model, we initialize with BERT-Base and then further pre-train the model by combining different pre-training tasks. For example, condMLM 100% + condMSM 100% (oneMod) denotes the configuration in which for each training sample, we first randomly choose the modality of speech or text to mask, then mask all tokens on the chosen modality and use the entire sequence of the other modality to recover the masked tokens. We compare SLU results on the validation set of each SLU task, from pre-training with different task combinations. We observe that the proposed model achieves significant improvement over the baseline on the validation set and also observe that the same performance rankings of pre-training configurations are retained on the SLU test set. As shown in Table 2, both condMLM 100% + condMSM 100% (oneMod) and condMLM 30% + condMSM 30% (twoMod) achieve the best ICAcc 89.2 on ASR 1-best with WER 19.2 after fine-tuning without adding phone embeddings, 1.6% absolute gain over the baseline. Adding phone embeddings into fine-tuning achieves additional gain 1.3%, overall 2.9% absolute gain (87.6 to 90.5) over baseline. On the test set with WER 15.2, similarly, pre-training with condMLM 100% + condMSM 100% (oneMod) obtains 1.8% absolute gain and adding phone embeddings into fine-tuning further improves 0.7% absolutely and raises the overall absolute improvement to 2.5% and the best gain is 2.7% absolute (92.4 to 95.1). These results demonstrate that the proposed pre-training and fine-tuning approaches significantly and consistently improve SLU performance on ASR hypotheses with different WERs.

Table 2 also shows ICAcc and semER results on ASR 1-best of the Snips test set (WER 42.3). The baseline ICAcc and semER from fine-tuning BERT-Base are 82.1 and 57.1. Pre-training with condMLM 100% + condMSM 100% (oneMod) achieves ICAcc 83.7 and semER 55.3, after fine-tuning without adding phone embeddings, 1.6% and 1.8% absolute gains over baseline. Adding phone embedding into fine-tuning achieves additional absolute gains of 1.7% and 1.9%, overall 3.3% (82.1 to 85.4) and 3.7% (57.1 to 53.4) absolute gains on ICAcc and semER. In all cases, adding phone features during fine-tuning achieves a solid improvement on top of pre-trained models. One possibility is that the pre-training models are designed towards learning generic joint textual-phonetic representations; whereas the proposed fine-tuning approach exploits phone features directly for SLU. Since computing WSA loss requires both $W$ and $P$ sequences, computing condMLM 100% + condMSM 100% (oneMod) and WSA losses on the same training samples is infeasible. We add WSA to condMLM 30% + condMSM 30% (twoMod) but it does not produce gain, probably because learning word and phone sequence alignment is relatively easy and a more difficult WSA task is required. We also evaluate using concatenation of $W$ and $P$ during fine-tuning (i.e., same as pre-training) instead of adding phone embedding, but did not observe consistent improvement over adding phone embedding.

Our proposed pre-train and fine-tune approaches are agnostic to the ASR systems in cascaded SLU. All of our SLU fine-tuning use the SLU train set manual transcripts and NLU labels, and the same fine-tuned models are used for inferring intents and slots on ASR 1-best of valid and test sets generated by different ASR systems. In contrast, both prior works of improving SLU robustness to ASR errors [26, 27] require training on the specific ASR system. [26] finetunes ELMo using combined language model loss and confusion-aware loss on the ASR 1-best and WCNs of the SLU train set. The finetuned ELMo embeddings are used by biLSTM for SLU and achieved 89.55% ICAcc on the Snips test set ASR output. The two-stage SLU approach [27] first pre-trains biLSTM on general domain text and then further pre-trains the model on ASR lattices of the SLU train set. Their approach achieves 95.37% ICAcc on the Snips test set ASR hypotheses. Both prior approaches learn a significant amount of information about the specific ASR system, which may contribute to the substantial boost of SLU performance. We confirm this hypothesis by fine-tuning BERT-Base on the SLU train set ASR 1-best with NLU labels and evaluating ICAcc on the Snips test set ASR 1-best. This Oracle setup achieves 93.9 ICAcc w/o phone embeddings and 94.3 with PE.
Model Analysis. To analyze the efficacy of the proposed pre-training model on aligning textual and phonetic representations, we select the most frequent 20 confusion word pairs from the FSC valid ASR 1-best (WER 36.7%) and exclude pairs containing words not covered by the BERT vocabulary. Each word pair is used for retrieval with the ASR hypothesized word as query and the reference word as reference. We evaluate mean reciprocal rank (MRR) for retrieval by computing the cosine distance between the trained input embeddings from the pre-trained models. Table 3 shows that compared to MRR 0.1012 from BERT-Base, the proposed pre-training model by further pre-training BERT-Base with condMLM 100%+condMSM 100% (oneMod) significantly improves MRR to 0.1591, confirming that the proposed pre-training model significantly reduces representation distance between acoustically confusable words.