Hallucination of speech recognition errors
with sequence to sequence learning

The architecture for the word level ASR prediction model is shown in Figure 1. An encoder takes a word sequence representation of the true text $X=x_{1},\dots,x_{n}$ as input, and embeds it into a sequence of 256-dimensional vector representations (combined with position embeddings) $E=e_{1},\ldots,e_{n}$. A stack of four residual CNN layers [19] transforms $E$ into a final hidden representation $H=h_{1},\ldots,h_{n}$. Both the hidden representation $H$ and the embedded input $E$ are provided to an attention mechanism.

The decoder is comprised of three residual CNN layers. The decoder takes as input the sequence of predicted words prior to the current timestep, and embeds them into a sequence of vector representations $G=g_{1},\ldots,g_{i-1}$, we use 256 dimensional embeddings here as well. Along with these embeddings, each decoder layer also conditions upon an attended representation from the encoder derived through the mechanism explained below. The output of the final layer is passed through a linear transformation followed by a softmax, to give a probability distribution over the target vocabulary at step $i$. Cumulatively, this model has 37M parameters.

For words and word-pairs where the model is unable to capture enough examples of possible recognitions or mis-recognitions, if we can make additional information about how each word sounds like (through the phonemes), the model could learn to “backoff” to the phoneme representation as needed. Thus, to improve generalizability and aid learning, we look at incorporating a phonetic representation of the true text as an additional input.

Accordingly, we propose a sequence to sequence model with two encoders and one decoder as shown in Figure 2. Encoder A takes a word sequence corresponding to the true text whereas encoder B takes the phoneme sequence corresponding to the same. The decoder attends to both encoders to produce predictions for the recognized word sequence.

In this model, we use the same four layer decoder architecture as in the word level ASR prediction model, but in the encoders we use wider kernels and increase the number of layers, so as to account for phoneme sequences being longer than word sequences, while keeping the number of parameters comparable. Each encoder comprises of three residual convolutional layers with 64 filters and a kernel size of 11, followed by two residual layers with 128 filters and a kernel size of 7, and finally one residual layer with 256 filters and a kernel size of 5. Cumulatively, this model has  38M parameters, which is comparable to the word level model.

To allow the decoder to look at both word and phoneme sequence encoders, we propose a dual attention mechanism detailed in III-B1 below, and to further encourage it to learn to incorporate both sources of information, we introduce an encoder dropout mechanism as detailed in III-B2. In limited experimentation, we also tried adding a second decoder with an auxiliary objective of predicting the phoneme sequence representation of the recognized text, but it did not seem to change the results much, as a result we did not explore it further.

Fisher is a conversational telephonic speech corpus in English containing 2000 hours of audio data paired with human annotated transcriptions [22], segmented into 1.8 million odd utterances. We transcribe Fisher using multiple ASR systems, in order to create pairs of “true text” (human annotated or corrected) and “recognized text” (ASR transcribed), used for training and evaluation of the error prediction system.

The virtual patient is a graphical avatar based spoken dialog system for doctors to practise interviewing patients (see figure 4). The virtual patient is designed to be capable of answering a limited set of questions (that have fixed and pre-written answers), creating the task of question classification based on user input. The following are different sets of virtual patient data we use:

The true text and recognized text are converted into word sequences using a tokenization scheme that mainly relies on lowercasing the text, removing punctuations, and splitting the on whitespaces. These word sequences are then deterministically transformed into corresponding phoneme sequence representations, by relying on a lexicon of word pronunciations, in conjunction with a grapheme-to-phoneme model to approximate pronunciations for unknown words. Following prior work [6, 2], we use the pronunciation lexicon provided as part of the Switchboard corpus [27], and use Phonetisaurus to train our grapheme-to-phoneme model on data from the same pronunciation lexicon. Special tokens such as noise, laughter, silence, end of utterance, were removed due to their absence in text data not of a spoken nature. A small number of examples ( 2.2%) in the Fisher data that contained zero words or phonemes in the “true text” as a result of this preprocessing were taken out prior to experimentation.

For the word level or single encoder model, we train the our network akin to a translation model using the Fairseq toolkit [28]. For each pair of true and speech-recognized word sequences, the encoder is fed the true word sequence, and for each $i\in 1\ldots m$, we feed the decoder the first $i-1$ words from the speech-recognized sequence and give as a target the $i$th word of the speech-recognized word sequence, with a cross-entropy loss. We train with a Nesterov accelerated gradient descent [29] optimizer for 60 epochs with a learning rate of 0.1 and a momentum of 0.99, with an optional 15 additional epochs in the finetune setting. To prevent overfitting, we employ a dropout of 0.2 on the input of every CNN layer, and on the output of the last CNN layer.

For the dual encoder model, we train our network similar to the word level model, except for two things. Firstly, Encoder A is fed the phoneme sequences corresponding to the true word sequence that is fed to Encoder B. Secondly, an Encoder Dropout of 0.5 is employed in addition to conventional dropout as used in the word level model i.e., on the input of every CNN layer, and on the output of the last CNN layer.

Following prior work [10, 6, 7], we use two metrics to evaluate the effectiveness of our models in hallucinating ASR errors, in addition to measuring the impact of our hallucinated ASR errors on the question classification model.

The first metric measures the percentage of real test set Error Chunks recalled in a set of “K best” simulated speech recognized utterances for each gold word sequence. The error chunks are again determined by aligning the gold word sequence with the errorful word sequence and removing the longest common subsequence. For example, if the gold sequence is “do you take any other medications except for the tylenol for pain” and the errorful sequence is “you take any other medicine cations except for the tylenol for pain,” the error chunks would be the pairs $\{medications:medicine\>cations\}$ and $\{do:\>\>\}$. Our detection of error chunks is strict — for an error chunk to qualify as predicted, the words adjacent to the predicted error chunk should be error-free.

The second metric measures the percentage of times the complete test set utterance is recalled in a set of “K best” simulated utterances for each gold text sequence (including error-free test sequences). We aimed to produce 100 unique simulated speech recognized utterances for each gold word sequence, so for both of these metrics, we evaluate the performance at K=100. These are both “hard” metrics since the possibilities of various kinds of errors is quite endless, and the metrics only give credit when the utterance/error chunk exactly matches what was produced.

In the in-domain evaluation setting, we measure our models’ ability to predict errors on audio from the same corpus, and transcribed using the same speech recognizer, as used to generate their training data.

Table I shows the results on the held out test sets from the Fisher corpus (IV-A1) for our word level and dual encoder end to end models, comparing with our prior reported results using a confusion matrix based model on the smaller test set. Both the end to end models greatly improve over the previous best reported results with sampled decoding on the confusion matrix, in terms of real error chunks recalled, as well as complete speech-recognized utterances recalled. The dual encoder model outperforms the word-level end to end model on both metrics on the full test set, corroborating previous observations about the usefulness of phonetic information in improving generalization to words with limited or no examples in the training set.

The sampled decoding mechanism does the best on the error chunk prediction metric, which agrees with previous observations about the peaky nature of errors made by modern neural network based speech recognizers. However, it also brings a slight penalty on the complete utterance prediction metric, compared with the beam search decoding, perhaps because we sample the final output words independently for each time step, whereas beam search scores the word sequences by cumulative weight.

In practice, we are hoping for error hallucination to help in the various scenarios where task-specific labeled speech data is limited or unavailable, and thus we also measure the quality of our models’ hallucinated transcripts in out-of-domain and limited-resource settings.

Table II shows the results on predicting recognition errors made by the cloud based ASR service from 2018 on the Virtual Patient read speech subset (IV-B2), for comparing results to prior work. We use the same models from Table I so this is a completely out-of-domain evaluation where the recognizer as well as audio domains are unseen at train time. All our end to end models again improve on the best prior reported results on both error chunk and complete utterance prediction metrics, although the improvements are more modest in this case compared to the in-domain setting. In preliminary published work [7], we reported how the output of the word level end to end model was different compared to the output of the phonetic confusion matrix model in this out-of-domain case, and the diversity of information gained from phonetics is again underscored again here by the gains seen due to use of the dual encoder model.

We also evaluate our models’ ability to predict recognition errors seen on audio from the Virtual Patient conversational speech set (VP_conv), made by a recent 2020 version of a cloud-based ASR service. Table III shows the results for predicting recognition errors on this set from our models trained in settings with zero as well as limited recognizer-specific ASR data available.

First, we evaluate base versions of our models i.e., the same as the ones evaluated in Tables I and  II, just trained on transcripts of the Fisher training set from an unrelated speech recognizer as compared to test time test-time (Fisher_base). Perhaps unsuprisingly, the results are comparable to what we see on the read speech data in Table II.

Further, we take the base versions of our models and train them further with the finetuning sets from the same speech recognizer as test-time, viz.: the Fisher finetuning set (Fisher_finetune) and the train portion of VP Conversational Speech Set (VP_conv) for up to 15 epochs. This results in an over 50% relative increase in error chunk recall on this test set, and approximately 9% absolute increase in complete utterance recall, showing a great benefit from the finetuning on recognizer-specific data including some domain-specific data.

As we see great benefit from finetuning,, we evaluate versions of our models that are train for 75 epochs only on the finetune sets i.e., data from the same recognizer as test time. While these models perform better than the base models trained only on unmatched recognizer data (Fisher_base), they are not as good as the finetuned versions of the base models.

Overall, in Table III, we find that our finetuned models that learn from both the larger but unmatched recognizer data (Fisher_base), as well as the smaller but matched recognizer data (Fisher_finetune and VP_conv), perform better than those only trained on either of them. The dual-encoder architecture still does the best, showing the continued benefit of the phonetic representation. Surprisingly, unlike what we see with the base version of the models Tables I, II, and the Fisher_base rows of Table III), the sampled decoding no longer helps improve error chunk recall on the finetuned models, in fact it hurts slightly. Our hypothesis for the cause behind this is that: with the Fisher_finetune and VP_conv sets, we are able to better model contextual errors resulting from the recent cloud based recognizer, and beam search’s ability to consider the likelihood of sequences of words in the output outweighs the benefits of sampling that we see in other scenarios.

We use a self-attention RNN [30] based question-classification model adapted for the Virtual patient task [25]. This model uses a single layer BiGRU as the RNN. For the attention mechanism, we use 8 attention heads. Each attention head produces a representation of the input attending to different parts of the text. The vector representations from the 8 heads are then concatenated and fed into a fully connected classification layer with softmax activations to predict a distribution over 376 classes. Unlike the originally proposed model [30], we do not impose an orthogonality constraint on the attention scores from different attention heads. We found that doing so hurt the classification performance.

We train our model to minimize the cross-entropy loss using the Adam optimizer [31] with a learning rate of 0.001, a dropout rate of 0.5, and early stopping with a patience of 15 epochs. In the baseline case i.e., without error hallucination, the training uses the gold or typed versions of the text as input along with corresponding class labels. In the settings with some real ASR training data, the speech recognized versions of the input from the Virtual Patient Conversational Speech set are added to the training set.

In the error hallucination case, we use a sampling strategy [2] wherein, at train time, the input text for the question classifier is randomly transmuted with a pseudo-speech-recognized utterance sampled from the output of our finetuned ASR error prediction model (best one from Table III), except that in the zero domain-specific ASR data case the VP_conv portion is excluded during finetuning. The sampling rate is treated as a hyperparameter and chosen from the set $\{5.0\%,10.0\%,25.0\%,50.0\%,75.0\%,100.0\%\}$ by tuning on the development set. A sample rate of $x\%$ means that the a training instance is replaced by a corresponding errorful alternative with a $x\%$ probability.

Table IV shows question classification performance with and without ASR hallucination, to measure changes in Accuracy and Macro-F1 scores averaged across 5 random seeds. We observe that our proposed approach for hallucination helps improve downstream performance in multiple WER settings whether real ASR training data for the NLU task is available or not. We also observe that with increase in WER, the benefit from improvements from performing ASR hallucination can even be higher than using real ASR data. In the 12.5% WER setting, adding hallucinated ASR in addition to some real ASR data, improved accuracy from 76.1% to 77.8% i.e., about twice as much as the improvement from real ASR data alone. In the 41.1% WER setting, even with zero ASR data, our hallucination approach allowed an absolute 2.3% improvement in downstream task accuracy, whereas real ASR data alone gave an improvement of 1.1%. Notably, this shows that our hallucination approach can improve the NLU system performance even more than by adding some real ASR data. We reason that this happens because the use of real ASR data provided at most one alternative transcript containing ASR errors per training example, whereas our hallucination approach allows the model to see multiple plausible errorful transcripts per training example, potentially even a different one every epoch.

It is worth noting that error hallucination also improves performance slightly on gold transcripts which suggests that it acts like a soft data augmentation as proposed in [32]. However, this improvement is not as high as that in noisy scenarios especially in higher WER settings.