Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data

OpenAI Whisper [15] employs a single sequence-to-sequence model to perform multiple speech processing tasks, including LID, multilingual ASR, any-to-English ST, and utterance-level segmentation. Our OWSM mostly follows this design, but extends it to potentially support any-to-any ST. Figure 1 illustrates the multitask data format. Data samples from different tasks are represented in a unified format, which can be predicted by the decoder in an autoregressive manner. Specifically, each sample is converted to a sequence of tokens with two segments separated by special tokens. The first segment (before “SOS”) is an optional text prompt used as a condition, while the second segment is the actual target. The target starts with a special token denoting the language of the input speech. Then, it uses a task token to distinguish between ASR and ST. There is a separate ST token for each target language, which enables translation to any language. Finally, it appends the text transcription either with or without utterance-level timestamps. All timestamps are quantized and represented as special tokens.

Whisper is pre-trained on 680k hours of labeled audio data sourced from the Internet, which is not publicly accessible. To construct a speech dataset for large-scale supervised learning, we combine training sets from various publicly available ASR and ST corpora. These diverse corpora encompass a wide range of speaking styles, recording environments, and languages. Our datasets are prepared using an open-source toolkit, ESPnet [22]. However, OWSM is trained on long-form audio data, which deviates from previous recipes in ESPnet. Consequently, we have developed new data preparation scripts tailored specifically for Whisper-style training. We concatenate consecutive utterances within the same long talk based on their original timestamps. Each long-form utterance is limited to a maximum duration of 30 seconds. During training, all utterances are padded to precisely 30 seconds, optimizing the utilization of computational resources.

To date, we have developed three versions at different scales, denoted as OWSM v1, v2, and v3 in Table 1. Our largest dataset, v3, comprises 180k hours of labeled audio data. This constitutes approximately one quarter of the total data employed by OpenAI Whisper in its training process [15]. The individual datasets utilized by our models are listed below:

• •

  OWSM v1: AISHELL-1 [23], CoVoST2 [24], GigaSpeech [25], LibriSpeech [26], MuST-C [27], SPGISpeech [28], and TEDLIUM3 [29].

• •

  OWSM v2: all data in v1, GigaST [30], Multilingual LibriSpeech [31], and WenetSpeech [32].

• •

  OWSM v3: all data in v2, AIDATATANG [33], AMI [34], Babel [35], Common Voice [36], Fisher (Switchboard) [37], Fisher Callhome Spanish [38], FLEURS [39], Googlei18n³³3Resources 32, 35, 36, 37, 41, 42, 43, 44, 52, 53, 54, 61, 63, 64, 65, 66, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, and 86 from openslr.org., KsponSpeech [40], MagicData [41], ReazonSpeech [42], Russian Open STT [43], VCTK [44], VoxForge [45], VoxPopuli [46], and WSJ [47].

OWSM follows Whisper to utilize a Transformer encoder-decoder architecture [1], where the encoder and decoder have the same number of layers. However, OWSM additionally employs a joint CTC loss for ASR targets [48], which was empirically shown to stabilize our training process. The input waveforms are converted to 80-dimensional log Mel filterbanks with a window length of 25ms and a hop length of 10ms. The extracted features are augmented using SpecAugment [49] and normalized by their global mean and variance. The features are then processed by a two-dimensional convolution module to reduce the sequence length. OpenAI Whisper [15] always downsamples the sequence by 2, resulting in a time resolution of 20ms. Our OWSM v2 and v3 perform 4 times downsampling, which further improves efficiency. The detailed configurations of Transformer encoder and decoder layers are summarized in Table 1. OWSM v1 and v3 use the same configurations as Whisper small and medium, respectively, while OWSM v2 is slightly smaller than v3. ⁴⁴4OWSM has slightly more parameters than Whisper under the same configuration, because the ESPnet model has a larger convolution downsampling module and does not share the input embedding and output projection in its decoder.

For inference, OpenAI Whisper implements both greedy decoding and beam search with temperature fallback. The latter is a complicated procedure relying on many heuristics and hyperparameters such as beam sizes, temperatures, log probability threshold, and gzip compression rate threshold. Our OWSM utilizes the ESPnet framework [22], thereby ensuring compatibility with various decoding algorithms originally supported by ESPnet, including greedy search, beam search, and joint CTC/attention decoding (for ASR only) [50].

OWSM is implemented in ESPnet [22] based on PyTorch [51]. Table 1 compares the training hyperparameters of different models. OWSM uses the same batch size as Whisper, but the number of total updates is smaller. OWSM is trained on NVIDIA A100 (40GB) GPUs. Each GPU takes two samples, and gradient accumulation is applied whenever necessary to ensure the total batch size is 256. Specifically, OWSM v1 is trained for around 7 days on 32 A100 GPUs and OWSM v2 and v3 are trained for around 10 days on 64 A100 GPUs. After training, five checkpoints with the highest validation accuracies are averaged to generate the final checkpoint.

Large-scale distributed training presents significant challenges, particularly when the computation budget is limited. As we scale up from a few thousand hours of data to nearly 200 thousand hours, we have encountered a range of issues. Here, we discuss some of the challenges and provide valuable training tips to help overcome these obstacles effectively. We will release our scripts that support these techniques.

Time resolution: Whisper employs a time resolution of 20ms within its encoder module, resulting in a sequence length of 1500 for 30-second inputs. This significantly increases GPU memory consumption and makes training slower and more difficult. In contrast, contemporary state-of-the-art ASR and ST models [52, 53, 54, 55] adopt larger downsampling rates. Starting from OWSM v2, we have adopted a time resolution of 40ms, effectively reducing the sequence length and mitigating the associated computational demands. We have also found that a shorter sequence length facilitates easier convergence of the model.

Joint ASR CTC loss: In our preliminary experiments, we observed suboptimal convergence of the attention-based encoder-decoder model trained on multiple tasks and diverse data. Incorporating a joint ASR CTC loss [48] to the encoder output can stabilize training and expedite convergence.

Warm initialization: When training our largest model, OWSM v3, we employ a warm initialization technique by leveraging the pre-trained OWSM v2. Specifically, the first 18 layers of OWSM v3 are initialized with v2 (which has precisely 18 layers), whereas the remaining 6 layers are initialized randomly. This v3 model converges much faster than training from scratch. However, it remains to be investigated whether a warm initialization adversely affects the final performance of the model.

Memory and efficiency issues: We have developed several strategies to address memory and efficiency issues caused by large data. To train the BPE tokenization models using SentencePiece [56], we randomly select 10 million text transcriptions instead of using the whole set to reduce memory usage. For training, the entire text file is too large to be distributed across different workers. We partition the training set into 5 to 12 non-overlapping subsets and use multiple data iterators to construct mini-batches. We further filter out samples with extremely long transcriptions (e.g., greater than 600 tokens including both prompt and target) which are caused by incorrect alignments in the original corpus (e.g., Common Voice). Without such filtering, the training will occasionally encounter out-of-memory errors. Additionally, we validate intermediate checkpoints using only 10% of the full validation set. This might generate slightly inaccurate estimates of the actual performance, but it significantly reduces the validation time and thus allows for more frequent validation and checkpoint saving, which is crucial for large-scale distributed training. In fact, we encountered various failures mainly due to file system or communication errors, and we had to manually resume from previous checkpoints.

Inconsistent case and punctuation. Our training data is gathered from many public corpora. Some of them provide raw transcripts in true case with punctuation, but the others only provide normalized transcripts in lower or upper case without any punctuation. During inference, we find that OWSM models are so powerful that they are able to recognize the corpus and generate outputs that are consistent with the training data format. For example, the training data of WSJ is in upper case. When tested on WSJ test sets, OWSM also mostly generates text in upper case. Since only a very small portion of training data is in upper case, OWSM v3 performs poorly on WSJ (see Table 2). In the future, we will normalize the text to address this issue. Note that this analysis demonstrates the benefit of using public data and open-source code, without which we cannot discover such issues.

Table 2 presents word error rates (WER) on standard English ASR benchmarks. Greedy search is employed without any external language models. To ensure fair comparison, we prepare all test data in ESPnet and evaluate Whisper in the same setup instead of reporting results from their paper [15]. The text is normalized using the English or basic normalizer provided by Whisper. Whisper large is not included since it is significantly larger than the other models. Although many public ASR corpora are combined, our English training data is still significantly smaller than that of Whisper (73k vs 438k hours). However, our OWSM models achieve competitive results in most benchmarks. OWSM models even outperform Whisper on LibriSpeech and Switchboard.

By comparing different versions of OWSM, we observe that its English ASR capability is largely improved from v1 to v2, demonstrating the effectiveness of scaling up in terms of the number of model parameters and the amount of training data. However, OWSM v3 does not show a consistent improvement over v2 in all benchmarks. OWSM v3 achieves lower WERs on Switchboard and VoxPopuli test sets, likely because their training sets are newly added (see Section 2.2). OWSM v3 has slight degradations on LibriSpeech and a large degradation on WSJ. This is probably due to the shift of data distributions from v2 to v3. As shown in Table 1, our v3 dataset contains significantly more languages compared to v2 (151 vs 23), but the model size is only slightly increased (889M vs 712M). Hence, the model has to adjust its capacity from English to other languages or from one type of speech to another type. This issue might be mitigated with larger models and more diverse data. We will explore it in the future. Please refer to the last paragraph in Section 2.5 for more discussions.

We have also investigated the inference speed. Specifically, we select 50 utterances of 30 seconds from our prepared TEDLIUM dev set, and decode OWSM v3 with greedy search using a single NVIDIA A40 GPU. The average decoding time for each 30-second utterance is $2.3$ seconds.

Table 3 shows the ASR results on multilingual benchmarks. In general, OpenAI Whisper achieves better performance than our OWSM, because Whisper employs significantly more training data in all languages except Japanese. For Japanese, OWSM v3 outperforms Whisper by a large margin (CER: 11.3 vs 25.3) thanks to the larger amount of training data (19k vs 7k hours) from ReazonSpeech [42]. Notably, OWSM v2 achieves the best results on the English and Chinese test sets from Multilingual LibriSpeech and AISHELL, respectively, despite being trained on less data.

The trend across different versions of OWSM is consistent with that in Section 3.1. OWSM v2 is drastically improved compared to v1 in all languages, which verifies the benefits of scaling up. OWSM v3 outperforms v2 in a few languages but achieves comparable or slightly worse results in the others. Again, this is likely because the model needs to adjust its capacity to support much more languages in v3.

Similar to Whisper, OWSM performs long-form ASR by consecutively transcribing 30-second audio segments and shifting the window based on predicted timestamps. Table 4 presents the long-form ASR results on the TEDLIUM test set, where each input audio is an unsegmented long talk. OWSM v2 achieves 7.2% WER with greedy decoding and 6.6% WER with beam search. Whisper models achieve lower WERs in both cases, likely because: (1) their training set is larger; (2) their data, collected from the Internet, is originally in a long form, which can be more realistic than ours; (3) they apply various heuristics to improve the timestamp prediction and also the quality of text (see Section 4.5 in their official report [15]). In our future work, we will explore more strategies to enhance the long-form performance.

Table 5 shows two examples from TED talks, where timestamps are generated along with text tokens. Although utterances can be segmented in different ways, the boundaries predicted by OWSM are usually very close to the reference.

Table 6 compares different models on two ST benchmarks: MuST-C (English-to-X) and CoVoST (X-to-English). Whisper only supports the latter, while OWSM supports both directions.

OWSM models achieve notable results on MuST-C thanks to the sufficient amount of training data (more than 500 hours for each language). The BLEU scores of Chinese and Japanese are lower than those of European languages even with enough training data. This indicates that the model has difficulty in translating between very different languages.

On CoVoST, the performance of OWSM ranges across language pairs as the amount of training data varies from 1 to 300 hours. On Chinese and Japanese, OWSM outputs have low intelligibility while on the European languages OWSM outputs are moderately intelligible. On the other hand, Whisper is trained on 4k to 12k hours and thus achieves greater BLEU scores on X-to-English in general.

Similar to the findings in Section 3.1 and Section 3.2, OWSM v3 shows comparable or slightly worse performance than OWSM v2. This is because OWSM v3 employs almost the same amount of ST data but it has to recognize drastically more languages (see Table 1). Some of its capacity needs to be assigned to these additional languages.

As described in Section 2.1 and Figure 1, OWSM predicts a language token at the beginning of decoding, which effectively performs the LID task. Table 7 compares Whisper and OWSM on the FLERUS test set prepared in ESPnet. OWSM v3 achieves a top-1 accuracy of 81.4%, which outperforms Whisper small and medium by a large margin. This is because OWSM v3 utilizes the training data from Common Voice and FLEURS, containing 151 languages in total, whereas Whisper supports 99 languages that only cover a subset of the languages in FLEURS. Nevertheless, this result demonstrates that OWSM has a strong capability in speech classification although it is designed as a sequence-to-sequence model.

OWSM is compatible with various decoding algorithms in ESPnet. Table 8 compares three commonly used algorithms: CTC only (greedy), attention only (greedy), and joint CTC/attention (with beam size 10 and CTC weight 0.3). Beam search with joint CTC/attention achieves the best results in all test sets. Attention-only decoding outperforms CTC-only, indicating that the decoder has strong capacity.