Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correction

The automatic generative error correction (GER) task involves using pre-trained models and algorithms (e.g., contextual biasing) to automatically revise and improve the quality of written content, such as documents, articles, or translations. This process typically includes correcting grammar, syntax, style, and coherence errors and adjusting language to meet specific requirements or preferences. Automatic GER tasks aim to streamline the editing process, saving time and effort while enhancing the overall readability and effectiveness of the post-recognition transcript quality.

GER tasks involve revising outputs from automatic speech recognition (ASR) systems. When speech is transcribed into text, errors can arise due to variations in accents, background noise, or misinterpretations of words. Recent advanced GER algorithms, such as text-to-text modeling [1] and speech-language representation alignments [2], address these issues in ASR transcriptions. For instance, GER-based methods correct inaccuracies, enhance grammar and syntax, and ensure that the text accurately represents the spoken content. This refinement process significantly improves the quality and usability of the transcribed text, making it more valuable for applications such as captioning, transcription services, or voice-controlled systems.

In recent years, pretrained big neural network-based language models (LMs) have been used in ASR-related GER tasks. Zhang et al. [3] proposed a spelling corrector based on the transformer [4] to reduce the substitution error in Mandarin speech recognition. Zhang et al. [5] improved the BERT [6] effectiveness in detecting spelling errors with the soft-masking technique as the bridge between the error detector and corrector. Futami et al. [7] generated soft labels for ASR training with the BERT distilling knowledge. There are also some works [8, 9] studying to improve ASR rescoring by BERT. Additionally, BERT has also been successfully applied in multi-modal studies in vision-language pretraining [10, 11, 12, 13] or voice-language pretraining [14, 15, 16].

More recently, [1, 17] proposed combining large language models (LLMs) into a speech recognition system. In this work, we extend the correction methods described in [17] and apply them to Japanese ASR tasks in different settings from $N$-best hypotheses. Additionally, Li et al. [18] investigated knowledge transfer when fine-tuning a multilingual LLM to correct 1-best hypothesis errors generated by different speech foundation models across multiple languages. We introduce a new multi-pass augmented generative error correction (MPA GER) by integrating multiple system hypotheses on the input side with corrections from multiple LLMs on the output side and then merging them. Moreover, We will also address the issue with a fairly low Character Error Rate (CER) for which the conventional Recognizer Output Voting Error Reduction (ROVER) [19] and previous LLM GER no longer provide enhancements. Our results show that MPA GER shows improved performance on ASR quality and generalization.

There are two sets of experiments in total: one with small data (SPREDS-U1-ja) to verify the basic algorithm (LLM GER) at high error rates and the other larger one (CSJ) with low error rates to validate our improved algorithm (MPA GER). The experimental settings are as follows. Table 1 shows the experiment settings for LLM GER and MPA GER.

LLM models

The experiment employs the Japanese LLM (ELYZA-7b) model¹¹1huggingface.co/elyza/ELYZA-japanese-Llama-2-7b and Multilingual LLM (Qwen1.5-7b) model²²2https://huggingface.co/Qwen/Qwen1.5-7B. ³³3In this paper, we refer to the models fine-tuned by these LLMs as Elyza and Qwen1.5, respectively.

Dataset

We use two Japanese datasets for GER. (1) SPREDS-U1-ja ⁴⁴4ast-astrec.nict.go.jp/en/release/SPREDS-U1. Of these, 900 Japanese sentences are used for fine-tuning process, and a separate set of 100 sentences are used for evaluation. (2) the “Corpus of Spontaneous Japanese (CSJ)” [28]. For training ASR models, we use approximately 600 hours of lecture recordings as the training set (Train APS+SPS (academic+simulated)) according to [29, 30, 31, 32]. Three official evaluation sets (Eval₁, Eval₂, and Eval₃), each containing ten lecture recordings [32], are used to evaluate the speech recognition results. For the LLM test set, we extracted each 424 utterances from each Eval₁ (1272), Eval₂ (1292), and Eval₃ (1385) prepared by the ESPnet2 [33] CSJ recipe ⁵⁵5https://github.com/espnet/espnet/tree/master/egs2/csj/asr1. The left Eval₁, Eval₂, and Eval₃ not included in the LLM test set were gathered and used as our LLM train set (2677 utterances in total).

Preparing ASR models

(1) For the SPREDS-U1-ja GER experiments, we prepared the open-source pre-trained ASR models, including Whisper-Large-v3, Meta MMS model, CMU OWSM v3.1 model, and Whisper-Large’s earlier versions (v1 and v2). (2) For the CSJ GER experiments, we used the Whisper models and the Conformer-based model after applying fine-tuning for Whisper (Whisper-Large-v2, Whisper-Medium, and Whisper-Small) and training Conformer from scratch by CSJ data. We applied whisper fine-tuning and conformer training from scratch using CSJ training data, following default settings in ESPnet2 [33] CSJ recipe ⁵⁵footnotemark: 5 unless otherwise mentioned. We did not apply speed perturbation for data augmentation. We fine-tuned all whisper models in 3 epochs with a learning rate of 1e-5. We trained the Conformer-based model in 100 max epochs in 4 early-stop patients. The training is performed on 1 NVIDIA Tesla A6000 GPU.

Training LLM GER models

The implementation of our LLM GER models is based on the repository⁶⁶6https://github.com/Hypotheses-Paradise/Hypo2Trans for GER task [1]. In SPREDS-U1-ja, the training is performed on a NVIDIA Tesla V100 GPU using 8-bit training. The hyperparameters for finetuning are 15 epochs, learning rate 1e-4, batch size 64, and LoRA rank 4. We used an English prompt in the repository⁷⁷7https://github.com/Hypotheses-Paradise/Hypo2Trans/blob/main/H2T-LoRA/templates/H2T-LoRA.json for training and testing without any change. In CSJ data, the training is performed on an NVIDIA Tesla A6000 GPU using 8-bit training. The hyperparameters for finetuning are 20 epochs in Japanese-Llama-2-7b, 10 epochs in Qwen1.5-7b, learning rate 1e-4, batch size 256, and LoRA rank 4. We used an English-Japanese prompt based on automatic translation following the previous default English prompt ⁸⁸8In our pilot study for CSJ data, we got roughly lower loss and lower CER trends in Elyza and Qwen1.5 using the English-Japanese prompt..

For system combination in ROVER

In CSJ data, we applied ROVER [19] when getting a combined result from multiple LLM outputs for the proposed MPA GER in Fig. 2. We also applied ROVER directly for multiple ASR hypotheses as the baselines compared with the LLM GER methods. We tried two types of MPA GER: (1) $1$-best hypotheses combination from $N$-systems, (2) $N$-best hypotheses combination from a single system. In (1), we tried $3$-systems and $4$-systems for $1$-best hypotheses combination. In (2), we tried $3$-best hypotheses combination from a single system.

GER Evaluation

We used NIST-SCTK to evaluate the CER. In CSJ, before the CER calculation, we converted full-width characters to half-width characters and removed punctuation marks from the hypothesis and reference.

In this paper, we applied LLM GER method for Japanese ASR task and introduced a new multi-pass augmented (MPA) GER method that combines hypotheses from different ASR and LLM models. From our experiments, we found that combining different LLMs in MPA GER resulted in better CER trends than the previous standard LLM GER results even in low CER ASR task by reducing hallucinations or repetitions in LLMs. In the future, We will further validate the error correction performance of LLMs in a wider range of ASR scenarios, in a broader range of settings.