Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition

We denote a trained ASR model as $g(y\mid x;\mathbf{\theta})$, parameterized by the pre-trained weight $\mathbf{\theta}$, taking utterance $x$ as input and outputs un-normalized score of vocabulary class $y$. $\mathbf{\theta}$ can be split to two parts, $\mathbf{\theta_{f}}$ and $\mathbf{\theta_{a}}$. $\mathbf{\theta_{f}}$ is the parameters frozen at adaptation, while $\mathbf{\theta_{a}}$ is updated during the inference time. Test corpus $D_{test}$ contains $n$ utterances $\{x_{1},x_{2},...,x_{n}\}$. We focus on adapting $\theta_{a}$ with a single utterance $x_{i}$ itself in an unsupervised manner. This paper uses a transformer encoder with Connectionist Temporal Classification (CTC) loss [12] as an ASR model, but the proposed approach is model agnostic. In the following, we represent the number of character classes plus one CTC blank token as $C$, and the number of time frames from the CTC classifier’s output as $L$. The model output is $\mathbf{O}\in\mathbb{R}^{L\times C}$.

We introduce each component of SUTA in the below sections. Figure 1 illustrates the SUTA framework.

We use open-sourced CTC-based ASR model, Wav2vec 2.0-base CTC model [27], as our source ASR model. This model is fine-tuned on Librispeech 960-hour²²2Checkpoint of the trained ASR model is open-sourced in https://huggingface.co/facebook/wav2vec2-base-960h and obtains 3.4 and 8.6 Word Error Rate (WER) on Librispeech test-clean and test-other set. Since this model is trained on Librispeech, we regard Librispeech as the source domain. The architecture of the Wav2vec 2.0-base CTC ASR model is composed of three parts: a convolutional neural network-based feature extractor, a 12-layer transformer encoder, and a linear CTC classifier.

To evaluate the adaptation capabilities of the proposed SUTA approach, we examine the test-time performance in the following target domains: (1) Librispeech (LS) [28] contains read speech from audiobook in 16kHz. Although Librispeech test data is in-domain for our ASR source model, we want to verify whether SUTA retains performance for in-domain test samples. Furthermore, we inject additive Gaussian noises for different amplitudes ($\delta=\{0.005,0.01\}$) on the test-other set to create covariate shift, testing SUTA’s capability to mitigate Gaussian noises. (2) CHiME-3 (CH) [29] is a noisy version of WSJ corpus with artificial and real-world environmental noises at 16kHz. We utilize the official enhanced evaluation set et05 to examine SUTA’s robustness in noisy acoustic conditions. (3) Common voice (CV) [30] is a crowdsourcing project that is supported by volunteers to read Wikipedia sentences and record samples at 48kHz. We re-sample the sampling rate to 16kHz to match the training condition of the source ASR model. Test set from the En-June-22nd-2020 release version is utilized. (4) TEDLIUM-v3 (TD) [31] consists of oratory speech based on TED conference videos. The audio quality is clean and stored at 16kHz. We use the official test set for experiments. For all these datasets, transcripts are pre-processed by upper-casing letters and removing punctuation except for apostrophes following [26].

Since there is no existing single-utterance TTA study on ASR, here we propose a naive pseudo labeling approach named Single-utterance dynamic pseudo labeling (SDPL) as the baseline method, which uses the source ASR model to predict the pseudo label of the utterance and adapt the model by minimizing CTC loss. The pseudo label is generated by greedy decoding on CTC output and refined at each iteration dynamically. SDPL does not consider the uncertainty of the distribution due to the nature of greedy decoding.

The hyperparameters are chosen from LS test-o with noises ($\epsilon=0.01$), and we use the best-found setup for all other target corpora. We explore different settings of trainable adaptation parameters $\mathbf{\theta_{a}}$ in SUTA. For simplicity, we denote the parameters of layer norm transformation as LN, feature extractor as Feat, and entire ASR model as All. The optimizer is AdamW, and the learning rate is searched from $10^{-4}$ to $10^{-6}$. The best-found learning rate is different for each setup of trainable parameters. Specifically, the best learning rate is $2e^{-4}$, $2e^{-5}$, and $1e^{-6}$ for LN, LN+Feat, and All. It is intuitive as the more parameters are relaxed for adaptation, the lower the learning rate is. We noted that SDPL could only improve WER when adapting with LN; more parameters cause drastic WER deterioration (probably due to errors from greedy pseudo label generation). Thus, we only report results of LN for SDPL in Section 3.5.

For default setup in all SUTA experiments, $\alpha$ is $0.3$, the number of adaptation iterations iter is 10, and $\mathbf{\theta_{a}}$ is LN+Feat. Experiments are run on Nvidia Geforce RTX 3090 GPU. Adaptation speed is about 0.115 second for a 1 second utterance for 10-steps adaptation.

Table LABEL:tab:main summarizes our experiment results. We list the state-of-the-art performances of models trained and evaluated on each dataset as the topline WER in the first row. We then show the RASR’s [26] results as a baseline where source (training) and target (evaluation) domains mismatch. RASR is a strong baseline, which leverages a much bigger network (36 transformer layers) and is trained with data augmentation (SpecAugment [32]). We also present results for our source ASR model, Wav2vec 2.0-based CTC model trained with source domain data (LS). Lastly, we show the performance of two single-utterance adaptation techniques, SDPL and our proposed SUTA, on top of our source model. SOTA results are unsurprisingly better than others since the model is trained and evaluated without mismatch. SUTA outperforms our source model and the SDPL baseline consistently. Moreover, our proposed approach yields comparable results to RASR with smaller networks and a training/adaption process, suggesting SUTA’s efficacy. Noting that we can also adopt SUTA on RASR’s models to improve test-time performance, here the results of RASR are just for showing a strong baseline without adaptation.

We dive deep into the results. For the in-domain evaluation (i.e., LS without additive noise), we observe that the pseudo labeling baseline (SDPL) slightly improves WER on top of the source ASR model (c.f., 8.3 vs. 8.6). SUTA further improves the WER to 7.3, which is surprising that TTA can even enhance the test-time performance for in-domain samples. With the target domain drifting from source by adding Gaussian noises, WER degrades from 8.6 to 13.9 ($\delta=0.005$) and 24.4 ($\delta=0.01$) for our source model. SDPL slightly improves WER, while SUTA drastically reduces WER for 3.9 ($\delta=0.005$) and 7.7 ($\delta=0.01$), suggesting that SUTA can improve the robustness of ASR when Gaussian noises are present.

Besides synthesizing the drifted target domain with LS, SUTA mitigates mismatch from multiple real-world audio corpora. For noisy speech in CH, SUTA successfully boosts WER from 31.2 to 25.0, whereas SDPL can only reduce WER to 30.4. Similar results can be found with the crowdsourcing recording in CV and the clean oratory speech in TD. SUTA improves WER from 36.8 to 31.2 for CV and from 13.2 to 11.9 for TD, outperforming SDPL significantly.

We investigate the importance of different components in SUTA, including the weighted sum coefficient $\alpha$ of the loss function, temperature values, and trainable parameters. Due to space limitations, we only present an analysis on CH.
Ablation study: Ablation studies are summarized in Table 2. Both EM and MCC objectives improve the performance individually, according to row $\alpha=1.0$ and $\alpha=0.0$, and weighted-sum combination further enhances relative WER reduction (WERR) from 18.3 / 18.6 to 19.9 when $\alpha=0.3$. The results indicate that using multiple objectives can improve the TTA, preventing overfitting on a single loss. Besides, the contribution of temperature smoothing is significant, reducing from 29.9 (T = 1.0) to 25.0 (T= 2.5) WER.

Choosing appropriate trainable parameters is also crucial to SUTA. There is 7.7 WERR by solely adapting the layer normalization (LN). If we fine-tune the normalization layer and the feature extractor (LN+Feat), we can further increase performance gain to $19.9$ WERR. Nevertheless, relaxing too many parameters for single-utterance fine-tuning causes slight performance degradation, as indicated by the result for All, which only yields 14.4 WERR.
Iteration step: We analyze the different number of model adaptation steps and their impact on performance. We explore 1, 3, 5, 10, and 20 for the number of iterations. Figure 2 demonstrates that the WER is consistently reduced before 10 steps; however, after 10 steps, the improvement saturates, particularly when not adopting temperature smoothing ($T=1$). On the other hand, utilizing temperature smoothing ($T>1$) stabilizes adaptation, mitigates the performance degradation in the later iterations, and enhances overall effectiveness.
Utterance length: When the length of input audio is short, it only contains one or two words. In this case, the output prediction merely includes a small number of classes and may be prone to class collapse when entropy is minimized. We investigate the relationship between utterance length and WERR in Figure 3. The result indicates that the WERR of short utterances (less than 2 seconds) is about 12 % WERR, which is much lower than the around 17.5 % WERR observed in typical utterances (those longer than 2 seconds). Although the performance gain of short audio is smaller than the normal one, we do not observe any adverse effect, and there is still WER improvement.