Interventional Speech Noise Injection
for ASR Generalizable Spoken Language Understanding

Previously proposed methods can be broadly categorized into three main approaches. The first, a Text-to-Speech (TTS)-ASR pipeline Liu et al. (2021); Chen et al. (2017), uses a TTS engine to convert text into audio, which is then transcribed by an ASR system into pseudo transcriptions. However, this method struggles due to different error distributions between human and TTS-generated audio, making the pseudo transcriptions less representative of actual ASR errors. The second approach, textual perturbation, involves replacing words in text with noise words using a scoring function that estimates how likely an ASR system is to misrecognize the words, often employing confusion matrices Jyothi and Fosler-Lussier (2010); Yu et al. (2016) or phonetic similarity functions Li and Specia (2019); Tsvetkov et al. (2014). The third method, auto-regressive generation, utilizes PLMs like GPT-2 or BART to generate text that mimics the likelihood of ASR errors in a contextually aware manner, producing more plausible ASR-like noise Cui et al. (2021).

We consider auto-regressive noise generation as our main baseline as it has shown superior performance over other categories Feng et al. (2022); Kim et al. (2021) However, auto-regressive noise generation is biased to ASR_i, limiting the SLU model used for ASR_i. Our distinction is generalizing SNI so that the SLU tasks can be conducted with ASR_∗. We provide a more detailed explanation of each category in Appendix 7.1.

As an alternative to SNI, ASR correction aims to denoise (possibly noisy) ASR transcription $T_{i}$ into $X$: Due to its similarity to SNI, similar methods, such as textual perturbation Leng et al. (2021) and auto-regressive generation Dutta et al. (2022); Chen et al. (2024), were used. Also, PLMs showed impressive results Dutta et al. (2022), as the ASR noise words can be easily detected by PLMs due to their semantic irrelevance. Using such characteristics, the constrained decoding method which first detects ASR errors, then corrects detected ASR errors is proposed Yang et al. (2022). However, the innate robustness of PLMs Heigold et al. (2018) makes SNI outperforms ASR correction Feng et al. (2022). In addition, it introduces additional latency in the SLU pipeline for ASR correction before the SLU model to conduct SLU tasks. In voice assistant systems like Siri, such additional latency is crucial as the SLU pipeline should deliver responses with minimal delay to ensure real-time interaction. Therefore, we focus on SNI for its effectiveness and minimal latency in SLU pipeline.

To robustify PLMs against ASR errors in SLU, the SNI model mimics the transcriptions of ASR_i which can be defined as a functional mapping $\textit{F}_{i}:X\rightarrow T_{i}$. Given the written GT $X=\{x^{k}\}_{k=1}^{n}$ with $n$ words, the SNI model $\textit{F}_{i}$ outputs the pseudo transcription $T_{i}=\{t^{k}_{i}\}_{k=1}^{n}$, simulating the transcriptions produced by ASR_i. These pseudo transcriptions, $T_{i}$, are subsequently used to train SLU models. For each word $x^{k}$, $z^{k}\in Z$ indicates whether ASR_i makes errors ($z^{k}=1$, hence $x^{k}\neq t^{k}_{i}$) or not ($z^{k}=0$, hence $x^{k}=t^{k}_{i}$). However, noises generated by this model differ from those of the other ASR system, ASR_j, and SLU model trained with the generated noises struggles with the errors from ASR_j. Therefore, we propose building an SNI model, $\textit{F}_{*}$, capable of generating “ASR_∗-plausible” pseudo transcripts $T_{*}$, which are plausible for any ASR system, ASR_∗.

To achieve this, we compare the underlying causal relations between the transcription process in SLU and SNI training data generation, which are depicted in Fig. 2(a). During ASR transcription, written GT $X$ is first spoken as audio $A$, which is then transcribed as transcription $T$ by ASR_i. Depending on the audio $A$, ASR_i may make errors ($Z=1$) or not ($Z=0$) in the transcription $T$. While transcribing, $X$ influences $Z$ through causal paths where every edge in the path is directed toward $Z$, where $Z$ acts as a mediator between $X$ and $Z$.

In SNI, the $X$ and $T$ transcribed by ASR_i are filtered based on $Z$ since they do not exhibit the error patterns required for training. However, such filtering induces a backdoor path and a non-causal relation in SNI. To further elucidate the causal relations in SNI training data generation depicted in Fig. 2(b), we outline the causal influences as follows:

• •

  $X\rightarrow T$: There is a direct causal relationship where the clean text $X$ influences the transcribed text $T$.

• •

  $Z\rightarrow T$: If $z^{k}\in Z$ is 1 (an error occurs), it directly affects the corresponding transcription $t^{k}_{i}$, causing it to deviate from the clean text $x^{k}$.

• •

  $Z\rightarrow X$: In the SNI training data collection process, $Z$ determines if $X$ is included. This means that only when the ASR system makes a mistake, indicated by any value $z^{k}\in Z$ being 1, the corresponding text is included in the training data. So, errors by the ASR system decide which clean texts are chosen.

The backdoor path $X\leftarrow Z\rightarrow T$, while ensuring that only instances where ASR_i made errors are included, introduces bias in previous SNI models based on conventional likelihood, defined as:

$$P(t^{k}_{i}|x^{k})=\sum_{z^{k}}P(t^{k}_{i}|x^{k},z^{k})P(z^{k}|x^{k}).$$

(1)

In contrast to ASR transcription where $z^{k}$ is a consequence of $x^{k}$ and the mediator between $x^{k}$ and $t^{k}$, $z^{k}$ conversely influences $x^{k}$ and acts as a confounder. Thus, the influence of $x^{k}$ to $t^{k}$ is drawn from $z^{k}$, not from $x^{k}$, thereby distorting the causal effect from $X$ to $T$. Such distortion skews the prior probability $P(z^{k}|x^{k})$ so that texts frequently incorrectly transcribed by ASR_i are also noised by SNI.

To mitigate biases in SNI toward the ASR_i, we propose interventional SNI (ISNI) where the non-causal relations are cut off by adopting $do$-calculus. Adopting $do$-calculus for conventional likelihood in Eq. 1, ISNI can be formulated as follows:

$$P(t^{k}|do(x^{k}))=\sum_{z^{k}}P(t^{k}|x^{k},z^{k})\cdot P(z^{k}).$$

(2)

Compared to Eq.1, the prior probability $P(z^{k}|x^{k})$ is replaced with $P(z^{k})$. This difference implies that the non-causal path $Z\rightarrow X$ is cut off, as the influence of $x^{k}$ on $t^{k}$ is not drawn from $z^{k}$ in Eq. 2 as prior probability $P(z^{k})$ is estimated independently from the non-causal path $Z\rightarrow X$. Thus, the influence of $x^{k}$ to $t^{k}$ is induced solely from $x^{k}$. We provide more detailed proof of Eq. 2 in Appendix 7.2.

In this section, we briefly overview how ISNI is implemented to generate ASR_∗-plausible noises of different error types, as illustrated in Fig. 3. To achieve this, ISNI implements two distinct terms in Eq. 2: $P(z^{k})$ and $P(t^{k}|x^{k},z^{k})$.

For debiased noise generation $P(z^{k})$, it is essential to cut off the influence of $x^{k}$ on $z^{k}$ so that $z^{k}$ becomes independent of $x^{k}$. ISNI explicitly controls $z^{k}$ and removes the dependence on $x^{k}$ by implementing SNI with constrained decoding, following Yang et al. (2022). Specifically, by utilizing do-calculus? to determine $z$, ISNI introduces corruption in tokens that ASR_i transcribes correctly but ASR_j may mistranscribe, thereby adding these as noise in the pseudo transcripts. For example, the token $x$ with $z=1$, such as ‘cue’ in Fig. 3, is fed to the constrained decoder. The constrained decoder then generates the noise word $t$ to replace $x$.

The noise word $t$ can be categorized as three conventional ASR error types: substitution, insertion, and deletion Jurafsky and Martin (2019), which is determined by the generation output of ISNI. Substitution errors occur when the generated output is a new token that replaces $x$. Insertion errors occur when the constrained decoder outputs multiple tokens for $x$. For example, in Fig. 3, the constrained decoder outputs two ‘the’ words for $x^{2}$, which means the insertion of the newly generated ‘the’ beside the given word ‘the’. Deletion errors occur when the constrained decoder outputs only the $eos$ token, representing the end of generation. If the generation ends without any tokens, as in $t^{5}$, the empty string replaces $x^{5}$. ISNI generates these error types synthetically, as demonstrated in the quantitative study in Appendix 7.6 and the generated examples in Appendix 7.7.

For ASR_∗-plausible generation $P(t^{k}|x^{k},z)$, we provide the phonetic information of $x$, such as ‘kju’ in Fig. 3, to ISNI to understand how $x$ is pronounced. The constrained decoder evaluates the generation probability based on phonetic similarity, recognizing that ‘c’ is pronounced as the phoneme ‘k’. Consequently, ISNI generates the ASR$*$-plausible noise word ‘queue’ which is phonetically similar to ‘cue’.

To mitigate bias from dependence on ASR_i, we propose an intervention on the non-causal effect $X\leftarrow Z$. This approach diversifies the noise words in the pseudo transcripts, including those rarely corrupted by ASR_i. By applying do-calculus as shown in Eq. 2, we derive a following equation for SNI:

$$P(t^{k}|do(x^{k}))=\sum_{z}P(t^{k}|x^{k},z)\cdot P(z).$$

(3)

This intervention ensures that the corruption of words $x^{k}$ in the pseudo transcripts does not depend on any particular ASR system. Consequently, it allows for the corruption of words that are typically less susceptible to errors under ASR_i.

For each word $x^{k}$, the prior probability $P(z)$ in Eq. 3 represents the probability that any given word $x^{k}$ will be corrupted. To simulate $P(z)$, we assume that the average error rates of any ASR system, ASR_∗ would be equal for all words and sample a random variable $a^{k}$ for each word $x^{k}$ from a continuous uniform distribution over the interval $[0,1]$. If $a^{k}\leq P(z)$, we set $z$ to 1, indicating an incorrect transcription of $x^{k}$ and generating its noise word $t^{k}$. Otherwise, $z$ is set to 0, and $t^{k}$ remains identical to $x^{k}$, mimicking an accurate transcription. We set the prior probability $P(z)$ as a constant hyperparameter for pseudo transcript generation.

To generate a noise word $t^{k}$ whose corruption variable $z^{k}$ is determined independently of any biases, we adopt a constrained generation technique that outputs $t^{k}$ to input $x^{k}$ Yang et al. (2022). To implement constrained generation, we use BERT Devlin et al. (2019) for encoding the written GT $X$ into the vector representation $E$ as follows:

$$E_{encoder}=\textit{BERT}(M_{word}(X)+M_{pos}(X)),$$

(4)

where $e_{encoder}^{k}\in E_{encoder}$ is the encoder representation of the token $x^{k}\in X$ and $M_{word}$ and $M_{pos}$ denote word and position embeddings.

The transformer decoder Vaswani et al. (2017) generates $t^{k}$ constrained on $x^{k}$. To encompass all ASR error types, the transformer decoder generates multiple tokens (see Section 3.4), denoted as $m$ tokens, $\{\tilde{t}^{k}_{l}\}_{l=1}^{m}$. These tokens replace $x^{k}$ in the pseudo transcripts. The error type of $t^{k}$ is contingent on $m$:

• •

  $m=1$ corresponds to deletion, generating only $eos$ is generated; $x^{k}$ is substituted by the empty string.

• •

  If $m=2$, one token in addition to $eos$ is generated; this token substitutes $x^{k}$.

• •

  When $m>2$, additional token will be inserted to $x^{k}$, to simulate insertion errors.

To generate $\tilde{t}^{k}_{l}$, the input of the decoder comprises the encoder representation of the written word $e_{encoder}^{k}$ and the special embedding vector $bos$ (beginning of sequence) and the tokens generated so far, $\{\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1}\}$ as follows:

$$E_{decoder}^{k}=H_{decoder}\cdot[e_{encoder}^{k};bos,\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1}],$$

(5)

where $H_{decoder}$ is the weight matrix of the hidden layer in the decoder. The transformer decoder then computes the hidden representation $d^{k}$ by processing the input through its layers as follows:

	$\displaystyle d^{k}=TransformerDecoder(Q,K,V),$		(6)
	$\displaystyle Q=E_{decoder}^{k},K,V=E_{encoder},$		(7)

where $E_{decoder}^{k}$ serves as query $Q$, and $E_{encoder}$ is used for both key $K$ and value $V$ in the transformer decoder. Finally, the probability of generating each token $\tilde{t}^{k}_{l}$ is calculated using a softmax function applied to the product of the word embedding matrix $M_{word}$ and the hidden representation $d^{k}$ added to the trained bias $b_{n}$:

$$P_{n}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1})=softmax(M_{word}\cdot d^{k}+b_{n}),$$

(8)

where $b_{n}$ are trained parameters. Here, we note that ISNI generates different error types depending on the output as we explained in Sec. 3.4.

We used the ASR transcription $T$ to train ISNI model. ISNI is supervised to maximize the log-likelihood of the $l$-th token $\hat{t}^{k}_{l}$ from the ASR transcription as follows:

$$\mathcal{L}_{n}=-\sum log(\textit{P}_{n}(\hat{t}^{k}_{l}|x^{k},\hat{t}^{k}_{0},...,\hat{t}^{k}_{l-1})).$$

(9)

The next step is to ensure the noise word generation $P(t^{k}|x^{k},z)$ ASR_∗-plausible. We adjust the generation probability $P_{n}$ with the phoneme-based generation probability $P_{ph}$ so that the ASR_∗-plausible noise words can be generated as followed:

$$\textit{P}_{gen}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1})=\textit{P}_{n}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1})\\ \cdot\textit{P}_{ph}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1}).$$

(10)

Our first proposal for $\textit{P}_{ph}$ is to provide the phonetic characteristics of each token via phoneme embedding $M_{ph}$. $M_{ph}$ assigns identical embeddings to tokens sharing phonetic codes, thereby delivering how each word is pronounced. Phoneme embedding $M_{ph}$ is incorporated into the input alongside word and position embeddings as follows:

$$E_{encoder}=BERT(\lambda_{w}\cdot M_{word}(X)\\ +(1-\lambda_{w})\cdot M_{ph}(X)+M_{pos}(X)),$$

(11)

where $\lambda_{w}$ is a hyperparameter that balances the influence of word embeddings and phoneme embeddings. The input for the decoder is formulated similarly to Eq. 5.

Fed both the word and phoneme embedding, the decoder then can understand the phonetic information of both the encoder and decoder input. Aggregating such information, the decoder would yield the hidden representation $d^{k}$ as in Eq. 7. Then, we feed the hidden representation $d^{k}$ to classification head to evaluate the phoneme based generation probability $\textit{P}_{ph}$ as follows:

$$\textit{P}_{ph}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1})=softmax(M_{ph}\cdot d^{k}+b_{ph}),$$

(12)

where classification head has the phoneme embedding matrix same as in Eq. 11 and bias $b_{ph}$. Using the phoneme embedding $M_{ph}$ instead of the word embedding $M_{word}$ in Eq. 8, $\textit{P}_{ph}(\hat{t}^{k}|x^{k})$ can be evaluated based on the phonetic information.

Our second proposal for $\textit{P}_{ph}$ is phonetic similarity loss $\mathcal{L}_{ph}$ supervising the phonetic information using phonetic similarity. This approach aims to generate ASR_∗-plausible noise words by assessing the phonetic resemblance of each token to $x^{k}$. $\textit{P}_{ph}$ should evaluate how phonetically similar each token is to the $x^{k}$. Phonetic similarity is advantageous as it quantifies the differences in phonetic codes, thereby allowing for objective supervision of $\textit{P}_{ph}(\hat{t}^{k}|x^{k})$. We utilize the phoneme edit distance $D$, as outlined in Ahmed et al. (2022), to calculate phonetic similarity. The phoneme edit distance $\textit{D}(w^{p},w^{q})$ measures the minimum edit distance between the phonetic codes of two words, reflecting how closely one word is pronounced to another. Notably, $\textit{D}(w^{p},w^{q})$ leverages articulatory features to compute similarity, which incrementally increases as the phonetic resemblance between the word pair enhances.

Phonetic similarity, $S$, is defined as follows:

$$\textit{S}(w^{p},w^{q})=max(|C_{p}|-\textit{D}(w^{p},w^{q}),0),$$

(13)

where $|C_{p}|$ is the length of the phonetic code of the word $w^{p}$. This formulation ensures that $\textit{S}(w^{p},w^{q})$ attains higher values when $w^{p}$ and $w^{q}$ are phonetically similar, and approaches zero when there is no phonetic resemblance.

To supervise $\textit{P}_{ph}$, phonetic similarity should be formulated as a probability distribution. For such purpose, we normalize phonetic similarity and compute the supervision $\textit{R}(\hat{t}^{k})$ as follows:

$$\textit{R}(\hat{t}^{k})=\frac{\textit{S}(t^{k},w)}{\sum_{w^{\prime}\in W}\textit{S}(t^{k},w^{\prime})}.$$

(14)

Then, $\textit{P}_{ph}$ is supervised by loss defined as follows:

$$\mathcal{L}_{ph}=KL(\textit{P}_{ph}(\tilde{t}^{k}_{l}|x^{k},\tilde{t}^{k}_{0},...,\tilde{t}^{k}_{l-1})|\textit{R}(\hat{t}^{k})),$$

(15)

where $KL$ is the KL divergence loss. Finally, ISNI is optimized to jointly minimize the total loss $\mathcal{L}_{tot}$ which is defined as follows:

$$\mathcal{L}_{tot}=\mathcal{L}_{n}+\lambda_{ph}\cdot\mathcal{L}_{ph},$$

(16)

where $\lambda_{ph}$ is the weight of $\mathcal{L}_{ph}$. Supervised to evaluate the generation probability based on the phoneme information, the phoneme generation probability $\textit{P}_{ph}$ ensures the ASR_∗-plausible noise words and $M_{ph}$ contains phonetic information.

In our study, we utilized diverse SNI models, including the GPT2-based auto-regressive SNI model, NoisyGen, as our primary baseline, given its proven efficacy in various spoken language tasks Cui et al. (2021); Feng et al. (2022). For the ASR GLUE benchmark, we also included NoisyGen (In Domain) Feng et al. (2022), another GPT2-based SNI model that, unlike our standard approach, uses the same ASR system for both training and testing, thereby not adhering to the ASR zero-shot setting. Also, PLMs trained only with written GT are used to show the necessity of exposure to ASR noises. Demonstrating that SNI models can match or surpass NoisyGen (In Domain) will confirm their ASR generalizability.

Additionally, for the DSTC10 Track 2, we incorporated the state-of-the-art TOD-DA Tian et al. (2021) as a baseline, selected for its inclusion of both TTS-ASR pipeline and textual perturbation techniques, which are absent in NoisyGen. We selected TOD-DA because it covers two distinct categories of SNI which were not covered by NoisyGen: TTS-ASR pipeline and textual perturbation.

We provide the ASR systems used for SNI training and SLU testing in Table 1.

To demonstrate the ASR generalizability of SNI models, we compare them with NoisyGen (In Domain) on ASR GLUE in Table 2. Unlike Noisy-Gen and our models tested in an ASR zero-shot setting, NoisyGen (In Domain) was trained and tested using the identical ASR system, which is incompatible with the ASR zero-shot setting.

Our findings indicate that auto-regressive SNI lacks generalizability for diverse ASR systems. If different ASR systems have similar error distributions, existing auto-regressive generation SNIs would generalize in an ASR zero-shot setting. However, NoisyGen is consistently outperformed by NoisyGen (In Domain) in every task. This result validates the ASR zero-shot setting where existing auto-regressive generation-based SNI models struggle to generalize in the other ASR systems.

Results of ISNI suggest that ISNI can robustify SLU model in the ASR zero-shot setting. ISNI surpassed NoisyGen in every task in every noise level even NoisyGen (In Domain) in high and medium noise levels for QNLI and in low noise levels for SST2. Such results might be attributed to the diversified ASR errors in the ASR GLUE benchmark, which ISNI is specifically designed to target.

ISNI demonstrated robust performance across varying noise levels compared to baselines, maintaining its efficacy as noise levels escalated. This result highlights that ISNI can better robustify against phoneme confusions, which increase under noisy conditions Wu et al. (2022); Petkov et al. (2013).

We demonstrate that ISNI significantly enhances robustness across various SLU tasks in an ASR zero-shot setting, particularly in KS, where lexical perturbation heavily influences retrieval Penha et al. (2022); Chen et al. (2022).

Results from the DSTC10 Track 2 dataset, in Table 3, reveal that while baseline models struggle in the ASR zero-shot setting, showing R@1 score below 60, ISNI consistently outperforms these models across all metrics. This superior performance, especially notable in KS, validates ISNI’s effectiveness against errors from unknown ASR systems, unlike previous models that require identical ASR systems for both training and testing Feng et al. (2022); Cui et al. (2021).

Additionally, the robustification of ISNI extends beyond KS. ISNI excels across all evaluated tasks, markedly improving the BLEU score by 2-3 times for Response Generation (RG). These findings affirm ISNI’s capacity to substantially mitigate the impact of ASR errors across diverse SLU tasks.

To evaluate the contribution of each component in ISNI, we performed ablation studies on both the DSTC10 Track2 dataset in Table 3 and the RTE task of the ASR GLUE benchmark in Table 4.

The results without the phoneme-aware generation are presented in the fourth row of Table 3 and Table 4. Removing the phoneme-aware generation led to a drop in performance across all tasks in DSTC10 Track2, as well as at high and medium noise levels in the RTE task of the ASR GLUE benchmark. This result demonstrates how phoneme-aware generation improves the robustness of SLU models in noisy environments. By accounting for word pronunciation, it ensures pseudo transcripts are ASR-plausible.

We also conducted an ablation study on the intervention by removing the do-calculus-based random corruption. For such goal, we trained a constrained decoding-based SNI model without do-calculus-based random corruption as in Eq. 3. For the constrained decoding method in ASR correction Yang et al. (2022), the corruption module, which determines whether the word $x^{k}$ will be corrupted, is jointly learned with the generation decoder. Such module can be considered as implementing $P(z|x^{k})$ in Eq. 1.

The results, shown in the fifth row of Table 3 and Table 4, indicate a performance decrease, particularly in the KS subtask of DSTC10 Track2 and in the RTE task, which are largely influenced by lexical perturbations due to ASR errors. A similar decline is observed in the RG subtask, further emphasizing the importance of the intervention in generating diverse and generalized ASR errors. However, we noticed a slight performance increase in the KTD task. We hypothesize that this improvement may be attributed to the nature of text classification tasks, where robustness against minor lexical changes may not be as critical. Despite this, previous research suggests that abundant noise in the training set can degrade text classification performance over time Agarwal et al. (2007), which ISNI is designed to mitigate.

Finally, we performed an ablation study on the phonetic similarity loss by training ISNI without phonetic similarity loss, relying solely on phoneme embeddings. The results, presented in the last row of Table 4, show a further reduction in performance across most noise levels. By supervising the phonetic information, phonetic similarity loss ensures that the generated noise words remain phonetically realistic, which is essential for improving model robustness in noisy conditions.

Previously proposed methods can be categorized into three categories. First, TTS (Text-to-Speech)-ASR pipeline Liu et al. (2021); Chen et al. (2017) adopts the TTS engine, the reverse of ASR, to convert written text into audio. The ASR_i transcribes it into pseudo transcription. However, the human recording and TTS-generated audio differ in their error distributions, which makes the resulting pseudo transcriptions different from ASR transcriptions Feng et al. (2022).

Second is textual perturbation, replacing the words $x^{k}\in X$ to the noise word $t_{i}^{k}$ as follows:

$$t^{k}_{i}=argmax_{w\in W}\textit{S}_{i}(w|x^{k}),$$

(17)

where $\textit{S}_{i}(w|x^{k})$ evaluates the score that ASR_i would corrupt the written word $x^{k}$ into each word $w\in W$ in the vocabulary set $W$. A widely adopted type of $\textit{S}_{i}(w|x^{k})$ is a confusion matrix built from the paired written and ASR transcribed corpora Jyothi and Fosler-Lussier (2010); Yu et al. (2016) or phonetic similarity function Li and Specia (2019); Tsvetkov et al. (2014).

A representative of the above two categories is TOD-DA which embodies both categories.

Third is the auto-regressive generation, where auto-regressive PLMs such as GPT-2 or BART are supervised to maximize the likelihood of the ASR transcription $T_{i}$ given its written text $X$. Adopting PLMs, the auto-regressive generation can consider contextual information and generate more ASR_i-plausible noise words Cui et al. (2021).

Auto-regressive generation is biased to ASR_i, limiting the SLU model used for ASR_i.

Our distinction is generalizing SNI so that the SLU tasks can be conducted with ASR_∗.

Using the $do$ operator to the conventional likelihood, $P(Y|do(X))$ is transformed as follows:

	$\displaystyle P(Y|do(X))$	$\displaystyle=\sum_{z}P(Y,z|do(X))$		(18)
		$\displaystyle=\sum_{z}P(Y|do(X),z)\cdot P(z|do(X))$		(19)
		$\displaystyle=\sum_{z}P(Y|X,z)\cdot P(z|do(X))$		(20)

Then, further transition is conducted by applying Rule 3 of $do$-calculus. Rule 3 states that we can remove the $do$-operator if $X$ and $Z$ are independent in $G_{\bar{X}}$, a modified version of the causal graph of SNI where all arrows incoming to $X$ are removed. In $G_{\bar{X}}$, where there are two paths, $X\rightarrow Y$ and $Z\rightarrow Y$, $X$, and $Z$ are independent, $X\perp Z$, as there is no valid path between $X$ and $Z$. Removing the $do$-operator, Eq. 20 is evaluated as follows:

$\displaystyle P(Y|do(X))$

$\displaystyle=\sum_{z}P(Y|X,z)\cdot P(z).$

(21)

For training the SNI model, we used the open-source commercial ASR system, Mozilla DeepSpeech. Mozilla DeepSpeech, which we adopted for SNI model training, is an RNN-based end-to-end ASR system trained on a 1700-hour audio dataset. DeepSpeech shows similar word error rates to famous commercial ASR systems such as Google Translate’s speech-to-text API and IBM Watson Speech-to-text, in various benchmark datasets such as Librispeech clean test and Commonvoice as the table below shows. This similarity in performance makes it a relevant and practical choice for our study, providing a realistic and challenging testbed for our methods.

Specifically, for the DSTC Track2 dataset, an unknown ASR system is used to generate transcriptions Kim et al. (2021). For the ASR GLUE benchmark, the LF-MMI TDNN ASR system is adopted to generate transcriptions Feng et al. (2022). ASR GLUE benchmark adopts an LF-MMI time-delayed neural network-based ASR syetem trained on a 6000-hour dataset.

We used Transformer decoder Vaswani et al. (2017) with 1 layer which has 12 attention heads and 768 hidden layer dimensions. We trained ISNI for 20 epochs with Adam optimizer with a learning rate of 0.00005. Also, we set $\lambda_{w}$ and $\lambda_{phoneme}$ as 0.5 to balance the semantic and phonetic information.

In this section, we quantitatively study the characteristics of the pseudo-transcripts generated by our ISNI. For this study, We generated pseudo transcripts in MSLT, our development set for SNI training. We set $P(z)=0.45$ for pseudo transcripts generation, which is similar to the word error rate of ASR transcription as we will show in Table 7. We analyze the following characteristics:

First, we show that PLMs are insufficient to generate the ASR_∗-plausible pseudo transcriptions. The noise words would be ASR_∗-plausible if it is phonetically similar to its written form as an ASR system would incorrectly transcribe into phonetically similar noise words. Therefore, we compared the phoneme edit distance $\textit{D}(w^{p},w^{q})$ between the written GT and the pseudo transcriptions. For a fair comparison, we set all tokens to have identical $z$ for the noise word generation to ensure that the identical words are corrupted.

Table 6 shows that the pseudo transcriptions without phoneme-aware generation show 17% larger phonetic distance. This result shows that PLMs are not enough to generate the ASR_∗-plausible pseudo transcriptions and ignorance of phonetic information is the obstacle to generating ASR_∗-plausible pseudo transcriptions.

Then, we study the word error rate of the generated pseudo transcripts.

The results in Table 7 show that pseudo transcripts generated by our ISNI contain more errors than P(z). However, compared to the baselines, the word error rate generated by our methods is more controlled, as we control whether to corrupt the word by P(z).

Then, we break down word error rate results by error types of insertion/deletion/substitutions.

Table 8 shows that three error types take up similarly in DeepSpeech transcription and pseudo transcriptions by our ISNI. This result shows that our ISNI can well handle three error types.

We provide the examples of the generated pseudo transcripts in Table 9 and 10.

Among the tokenized input, $z$ of $\#\#ial$ was set to 1, thus the constrained decoder generates its noise word. The constrained decoder made 1 substitution error $(bestial\rightarrow best)$ and 1 insertion error $(atail)$.

We used ChatGPT for grammatical corrections.