Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation

Post-processing is a task that complements ASR systems by analyzing their output. Compared to tasks involved in directly manipulating components within the ASR system, the black-box approach of ASR post-processing yields several advantages. It is often intractable for the LM inside an ASR system to account for and rank all hypothesis presented by the acoustic model [ogrodniczuk_proceedings_2020]. A post-processing system is free from such constraints within an ASR system [1]. ASR post-processing also helps improve performance in downstream tasks [16] as well as in domain adaptation flexibility [17].

Interpolating existing data for data augmentation is a popular method for regularization. The mixup method suggested in [22] has been particularly effective, especially in the computer vision domain [23, 24, 25]. Given dataset $D=\{(x_{0},y_{0}),(x_{1},y_{1})\ldots,(x_{n},y_{n})\}$ consisting of (data, label) pairs, mixup “mixes” two random training samples $(x_{i},y_{i})$ and $(x_{j},y_{j})$ to generate a new training sample $(x_{m},y_{m})$:

$$x_{m}=x_{i}*\lambda+x_{j}*(1-\lambda)$$

(1)

$$y_{m}=y_{i}*\lambda+y_{j}*(1-\lambda)$$

(2)

where

$$\lambda\sim Beta(\alpha,\alpha)$$

$\alpha$ is a hyperparameter for the mixup operation that characterizes the Beta distribution to randomly sample $\lambda$ from.

[26] suggests manifold mixup to utilize mixup in the feature space after the data has been passed through a vectorizing layer, such as a deep neural network (DNN). Whereas mixup explicitly generates new data by interpolating existing samples in the data space, manifold mixup adds a regularization layer to an existing architecture that mixes data in its vectorized form. In essence, mixup and its variants discourage a model from making extreme decisions. A hidden space visualization in [26] shows that correct targets and non-correct targets are better separated within the mixup model because it learns to handle intermediate values instead of making wild guesses in face of uncertainty.

Because text is both ordered and discrete, mixup with text data is best applied at feature space. Magnitudes of each token id in a tokenized text sequence does not convey any relative meaning; whereas in an image, varying the degrees of continuous pixel values does gradually manipulate its properties.

While mixup is yet to be widely studied in speech-to-text or natural language processing domains, several studies have found success in applying the method to text data. [27] constructs a new token sequence by randomly selecting a token from two different sequences at each index. [28] suggests applying mixup in feature space, after an intermediary layer of a pretrained LM. New input data is then generated by reversing the synthetic feature to find the most similar token in the vocabulary. While these two works involve interpolating text inputs, our method differs significantly in that we do not directly generate augmented training samples; instead, we utilize mixup as a regularizing layer during the training process. Our method also does not require reversing word-embeddings or discriminative filtering using GPT-2 introduced in [28].

[29], a work most similar to ours, applies manifold mixup on text data to calibrate a model to work best on out-of-distribution data. However, the study focuses on sentence-level classification, which requires a substantially different feature extraction strategy from the work in this paper. Our paper suggests a token-level mixup method, and aims to gauge the impact of single task (ASR error detection) regularization on an auxiliary task (speech disfluency detection).

To utilize BERT for a sequence labeling problem, a token classification head (TCH) must be added after BERT’s final layer. There is no restriction on what constitutes a TCH. In [12], for example, a combination of conditional random field and a fully connected network was attached to BERT’s output as a TCH. In our study, a simple fully-connected layer with the same number of inputs and outputs as the original input token sequence suffices as a TCH because our aim is to measure the effectiveness of regularization that takes place between BERT and the TCH. We modify a reference BERT implementation [30] with a TCH.

A single data sample during the supervised training stage consists of input token sequence $\bm{{x}}={x_{0},x_{1},\ldots,x_{n}}$, and a sequence of labels for each token, $\bm{{y}}={y_{0},y_{1}\ldots,y_{n}}$. Each corresponding label has a value of 1 if the token contains a character with erroneous ASR output, and 0 if the token contains no wrong character. All input sequences are tokenized into subword tokens [31] before entering BERT. For the rest of this paper, boldfaced letters denote vectors.

Our proposed method adds a regularizing layer between BERT’s output and the token classification head. BERT produces a hidden state sequence $\bm{{h}}$, with length equal to that of input token sequence $\bm{{x}}$:

$$\bm{{h}}={h_{0},h_{1},\ldots,h_{n}}=BERT(\bm{{x}})$$

(3)

From $\bm{{h}}$ we obtain $\bm{{h_{shuffled}}}$, a randomly permuted version of $\bm{{h}}$. Each corresponding label in $\bm{{y}}$ is also rearranged accordingly, creating a new label vector $\bm{{y_{shuffled}}}$. The same index-wise shuffle applied to $\bm{{h}}$ applies to $\bm{{y}}$, such that each label $y_{i}$ originally assigned to an input token $x_{i}$ follows the new index for that value during the shuffling process:

$$\bm{{h_{shuffled}}}={h_{s0},h_{s1},\ldots,h_{sn}}$$

$$\bm{{y_{shuffled}}}={y_{s0},y_{s1}\ldots,y_{sn}}$$

where

$$s0\ldots sn=RandomPermutation(0...n)$$

Next, we apply the mixup process element-wise to $\bm{{h}}$ and $\bm{{h_{shuffled}}}$, according to equation (1) and (2):

$$\bm{{h_{new}}}=(\bm{{h}}\odot\lambda)+(\bm{{h_{shuffled}}}\odot(1-\lambda))$$

(4)

Similarly, we also mix the original label vector $\bm{{y}}$ with its permuted counterpart $\bm{{y_{shuffled}}}$:

$$\bm{{y_{new}}}=(\bm{{y}}\odot\lambda)+(\bm{{y_{shuffled}}}\odot(1-\lambda))$$

(5)

Where $\lambda\sim Beta(\alpha,\alpha)$. This process “softens” the originally binary labels into range $[0,1]$.

For the mixup hyperparameter $\alpha$, we empirically chose value a of $0.2$. Figure 3 shows the experimentally observed relationship between $\alpha$ and the normalized counts of the fine-tuned system erroneously tagging disfluent speech as ASR errors.

$\bm{{h_{new}}}$ is propagated to the token classification head, whose prediction loss is calculated using binary cross entropy against $\bm{{y_{new}}}$, instead of the original ground truth label $\bm{{y}}$.

To obtain ASR outputs on which to perform error detection, we process multiple speech corpora through black box ASR systems. Detailed descriptions of datasets used is provided in the next section. Within the decoded transcript, we tag subword token indices that contain ASR errors to be used as labels during the post-processing fine-tuning. We also tag indices in the transcription with speech disfluencies. To demonstrate the effectiveness of our method across various off-the-shelf ASR environments, we transcribe all available speech corpora with both a traditional ASR system and an end-to-end (E2E) ASR system. For English corpora, we utilize a pretrained LibriSpeech Kaldi [32] model (factorized time-delay neural networks-based chain model) for the traditional system, and a reference wav2letter++ (w2l++) [33] system for the E2E system. For Korean corpora, we use the same respective architectures trained from scratch on internal data.

To create error-labeled decoded transcripts, each decoded transcript is compared with corresponding ground truth reference transcription. We use the Wagner-Fischer algorithm [34] to build a shortest-distance edit matrix from the decoded transcript to its reference transcription. Traversing the edit matrix from the lowest, rightmost column to the highest, leftmost column yields a sequence of delete, replace, and insert instructions to convert the decoded transcript into the reference transcript. Each character instructed to be deleted and replaced in the decoded transcript is marked as ASR errors.

Disfluency labels in reference transcripts must be transferred over to corresponding positions within the decoded transcripts. Since the introduction of ASR errors causes the decoder transcripts to be mere approximations of its ground truth transcripts, mapping each character index from the reference transcript to the decoded transcript is not rigorously a defined process. Here we again utilize the Wagner-Fischer algorithm, mapping only the character indices in the decoded transcript without any edit instructions to their corresponding indices in the reference transcript.

We judge the effectiveness of our regularization method by two criteria: how much it helps in error detection (Error detection $F_{1}$), and how much it reduces the number of accurately transcribed disfluent speech tagged as errors (Disfluencies wrongly tagged).

We tested our proposed methodology on five different speech datasets (4 English, 1 Korean). All speech corpora were provided with reference transcripts hand-annotated with speech disfluency positions.

Results of applying our proposed method is presented Table 2. Corpus names in the first column are each marked with the name of ASR system used for transcription. The second and third columns of table 3 show error detection $F_{1}$ scores with and without applying proposed regularization, respectively. The fourth and fifth columns report the number of correctly transcribed disfluencies wrongly tagged as ASR errors.

The proposed regularization in this paper shows greater influence as the number of training data increases (AMI and ICSI). One surprising outcome is the unchanging scores on the CallHome corpus. Analysis of the corpus and high $F_{1}$ scores in error detection indicate the uniform scores are a result of distinct patterns in ASR errors during transcription; the fine-tuned LM in every setting learns to confidently and consistently pick out only certain tokens as ASR errors.