Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART

The data augmentation is performed on the Common voice 16.0 dataset[4]. In the most recent release, Common Voice 16.1, English language data spans approximately 3,438 hours of audio, with 2,586 hours validated by a community of 90,474 contributors and Hindi language section comprises about 21 hours of recorded audio, of which 14 hours have undergone meticulous validation A 14-minute segment of custom, personalized audio was meticulously extracted from a YouTube video and subsequently converted into the WAV format. Employing the Ozen toolkit[5], a comprehensive processing pipeline ensued, involving speech extraction and transcription using the Whisper module. The transcribed results were then saved in the LJ format. It is noteworthy that the audio files are systematically stored in the designated ’wavs’ folder. Simultaneously, the transcribed Hindi texts are organized into separate ’train’ and ’valid’ text files, facilitating effective dataset management. It is crucial to emphasize that Whisper primarily defaults to English transcription. To adapt this framework for Hindi transcription, necessary modifications were made within the utils.py file, specifically altering the ”task” to ”transcribe” and setting the ”language” to ”hi”. These adjustments ensure accurate and contextually relevant transcription for the Hindi language. Additionally, the Ozen toolkit incorporates the pyannote[6] framework for speaker diarization, further enhancing its utility in speech and audio processing applications. This integration enables precise identification and separation of speakers within the audio, contributing to the overall robustness of the system.

The Retrieval-based-Voice-Conversion-WebUI by RVC-Project is an open-source tool enabling users to convert voice data using a retrieval-based voice conversion approach. It is freely accessible on GitHub [14] under the MIT software license. Recommended for audio recordings of a minimum 10-minute length for optimal model training, the project offers noise removal capabilities through Vocals/Accompaniment Separation and Reverbation, utilizing the ’HP2-all-vocals’ model. The target sample rate is set at 32,000, and the training utilizes the base v2 pre-trained model (f0G32K and f032K) with a batch size of 40 for 200 epochs on the NVIDIA A5000 GPU. Post 200 epochs, the KL Divergence Loss reaches 0.9292. This can be viewed in Fig.1 KL Divergence Loss (loss_kl) below.

After completing the training phase, the model underwent inference to reproduce the acquired voice characteristics. Utilizing Common Voice audio inputs alongside the trained model’s inferencing, coupled with specified index paths, a bespoke common voice dataset was generated, showcasing the distinctive custom voice. Critical parameters such as volume envelope scaling (0.25), filter radius (3), and search feature ratios (0.75) and 0.33 for safeguarding voiceless consonants and breath sounds were implemented. It is imperative to note that these parameter values may necessitate adjustment based on the audio utilized during training, and users are encouraged to experiment with different settings to identify the optimal configuration. The resulting customized Common Voice 16.0 dataset, featuring the unique custom voice, is accessible under ” Aniket-Tathe-08/Custom_Common_Voice_16.0_dataset_using_RVC_14min_data” on Hugging Face. The pipeline can be viewed in Fig.2 Data Augmentation below.

At the end of this process, we have generated a custom common voice 16.0 dataset with input audio which only comprises the personalized voice with the help of RVC which will be further used to fine tune the XLSR model.

XLSR-Wav2Vec2 is a framework for self-supervised learning of speech representations proposed by Facebook AI Research. It builds on the original Wav2Vec [7] but is trained on much larger speech recognition datasets to learn more robust representations. Specifically, XLSR-Wav2Vec2 leverages the self-supervised objective of masked reconstruction to pre-train a deep convolutional neural network model that encodes speech audio inputs into latent speech embeddings. More than 56,000 hours of unlabeled speech data across 50 languages are utilized to pre-train the XLSR model. This allows it to learn universal speech representations capturing various linguistic properties. Fine-tuning the pre-trained model on downstream speech tasks requires only a few minutes of labeled data to achieve state-of-the-art performance on benchmarks ranging from speech recognition to speech translation. The encoder model architecture and self-supervised pre-training technique allow XLSR-Wav2Vec2 representations to generalize very effectively across languages and domains.

”facebook/wav2vec2-large-xlsr-53” was the model used for fine-tuning and the training process occurred on an NVIDIA A5000 GPU for 40 epochs with a learning rate of $1\times 10^{-4}$ and a weight decay of $2.5\times 10^{-6}$ Although there wasn’t a significant difference, the model exhibited slightly better performance with weight decay. The model gave a training accuracy of 0.80 and WER of 0.53 The WER graph and training loss can be seen below in Fig.3. It was seen that the model was overfitting on the training data. This model trained on a personalized custom Common Voice 16.0 dataset can be found at ” Aniket-Tathe-08/XLSR-Wav2Vec2-Finetuned-14min-dataset”, which accepts Hindi audio input and generates corresponding Hindi text.

mBART (multilingual Bidirectional and Auto-Regressive Transformer) (Liu et al., 2020) is a multilingual sequence-to-sequence model pre-trained using a denoising autoencoding approach on large-scale monolingual corpora spanning 25 languages. It employs a standard Transformer-based encoder-decoder architecture (Vaswani et al. 2017) and is pre-trained to reconstruct input text fragments that have been corrupted through an arbitrary noising function. This self-supervised pre-training enables mBART to learn universal linguistic representations that transfer across multiple languages and downstream tasks. After pre-training on around 200GB of text data, mBART can be fine-tuned on target sequence generation tasks by simply adding task-specific decoder heads. Requiring only a few thousand labelled examples in a low-resource setting, fined-tuned mBART has achieved state-of-the-art performance on various multilingual benchmarks including translation, summarization, and question answering. The pre-trained representations and adapted parameters effectively encapsulate multilingual sequences, allowing mBART to generalize well even for low-resource language generation tasks.

Pyannote is an open-source speech processing toolkit designed for constructing speaker diarization systems. Diarization addresses the question of ’who spoke when?’ in an audio recording. In the end-to-end pipeline, Pyannote determines the start and end times of each speaker, leading to the segmentation of the input audio into multiple segments with the format (start, end, speaker). This information is then used to sequentially process the cropped audio segments through XLSR Wav2Vec2 and mBART for subtitle generation. The output is stored in an ’output.vtt’ file, containing both the start and end times based on the diarization and the output generated by XLSR Wav2Vec2 and mBART. An example of the ’output.vtt’ file is illustrated in the Listing 1 Diarization output below.

After uploading the input video with Hindi audio to the GUI, an ’mp3’ file is generated, containing the extracted Hindi audio from the video. This ’mp3’ file is then input into Pyannote for speaker diarization. The output of Pyannote, in the format (start, end, speaker), is used to crop the audio accordingly using this ‘start’ and ‘end’. The cropped audio segments are then processed through the XLSR Wav2Vec2 model for Hindi audio to Hindi text conversion and mBART for translating Hindi text to English. Subsequently, subtitles are generated. Using the ’start’, ’end’, and subtitle results, an ’output.vtt’ file is created. This ’output.vtt’ file is then overlaid onto the input video, producing the final output video with subtitles, as illustrated in the pipeline below in Fig. 4.