VoxHakka: A Dialectally Diverse Multi-speaker Text-to-Speech System for Taiwanese Hakka

For low-resource languages, Text-to-Speech (TTS) systems play a crucial role in language preservation and revitalization. Speaker-aware and zero-shot TTS systems [1, 2, 3, 4, 5] contribute to preservation efforts by not only capturing the unique phonetic nuances of the language but also by enabling the reproduction of voices belonging to culturally significant individuals, even when data is scarce. Furthermore, these systems facilitate language promotion by enabling the creation of diverse audio content, including audiobooks, short films, educational materials for non-native speakers, and newscasts featuring virtual anchors. This accessibility to engaging audio content can be instrumental in raising awareness and promoting the use of low-resource languages.

Motivated by this need, we introduce VoxHakka, a freely available, Creative Commons (CC)-licensed, high-quality multi-speaker TTS system designed specifically for Taiwanese Hakka (臺灣客家語).

Taiwanese Hakka, a major language in Taiwan, is spoken by the Hakka community, comprising approximately 15-20% of the total population (2-3 million people)¹¹1https://en.wikipedia.org/wiki/Taiwanese_Hakka/.. However, despite its significant speaker base, the number of fluent speakers is dwindling, particularly among younger generations who often favor Mandarin or Taiwanese Hokkien (臺灣台語). This shift in language use has resulted in a dearth of readily accessible, open, and accurately labeled Hakka language resources.

Furthermore, Taiwanese Hakka exhibits significant dialectal variation, primarily categorized into six major dialects: Sixian (四縣), Hailu (海陸), Dapu (大埔), Raoping (饒平), Zhaoan (詔安), and Nansixian (南四縣). Sixian and Hailu are the most prevalent, accounting for 57.5% and 35.8% of speakers, respectively. This linguistic diversity is reflected in Hakka’s rich phonology and intricate grammatical structure. The language features seven tones, with variations in number and values across dialects. Its consonant system demonstrates complexity through contrasts in voicing, aspiration, and a diverse range of vowels, including monophthongs, diphthongs, and nasalized vowels. The presence of syllable-final consonants (codas), particularly in checked-tone syllables ending in /p/, /t/, or /k/, distinguishes Hakka from other Sinitic languages. This, coupled with complex tone sandhi rules influenced by syllable combinations, contributes to the intricate prosodic patterns of the language. Importantly, each Hakka dialect exhibits unique characteristics in its tones, vowels, consonants, and tone sandhi rules, further underscoring the importance of dedicated language resources for each.

These factors—data scarcity and pronunciation complexity—pose significant challenges for developing high-quality Hakka TTS systems. Consequently, there is a dearth of robust Hakka TTS systems suitable for widespread use in academia or industry²²2As far as we know, only Cyberon (賽微科技) and Bronci (長問科技) published Taiwanese Hakka TTS systems, but both of the companies provide only the Sixian dialect.. Existing systems often focus solely on the Sixian dialect and offer limited speaker diversity (typically fewer than three speakers). Furthermore, these systems lack zero-shot capabilities, which would allow users to personalize their learning experience by synthesizing speech with their own voices. Such capabilities would also be invaluable in scenarios requiring voice anonymization or de-identification.

To address these challenges, we present a comprehensive approach to developing VoxHakka, encompassing web-based corpus collection, careful selection of a suitable TTS model architecture, and robust training procedures. VoxHakka distinguishes itself with the following key features:

1. 1.

   Comprehensive Dialectal Coverage: VoxHakka is capable of synthesizing speech in all six major Taiwanese Hakka dialects.

2. 2.

   Ethically Sourced and Reliable Data: The training data is ethically sourced from publicly available resources on the internet, primarily government educational institutions and publicly funded foundations. While not available for redistribution, the data’s accuracy and annotations are inherently reliable due to their origins.

3. 3.

   Zero-Shot Synthesis and Computational Efficiency: Leveraging YourTTS technology [2], VoxHakka operates as a zero-shot model, enabling the synthesis of speech for unseen speakers and even in different languages (e.g., Mandarin). Furthermore, it offers control over speaking rate and facilitates efficient inference using only CPU resources.

4. 4.

   Open Accessibility: VoxHakka is released under a permissive CC-BY 4.0 license³³3https://creativecommons.org/licenses/by/4.0/., granting users the freedom to utilize, modify, and share the model without restrictions.

We make our demo page and model publicly available at https://voxhakka.github.io/.

High-quality TTS models rely on clean, well-articulated speech data with a consistent speaking rate. We describe our methodology for acquiring and processing data from web sources to create a suitable dataset for TTS model training. Figure 1 illustrates this data preparation pipeline.

1. 1.

   Web Scraping: We utilize web scraping techniques to gather audio files and their corresponding transcriptions from government websites and affiliated resources, such as the Dictionary of Taiwanese Hakka⁴⁴4Constructed by Ministry of Education (教育部): https://hakkadict.moe.edu.tw/., the Bank of Hakka Examination⁵⁵5Constructed by Hakka Affairs Council (客委會): https://elearning.hakka.gov.tw/., and Hakka Radio⁶⁶6Funded by Hakka Public Communication Foundation (客傳會): https://www.hakkaradio.org.tw/.. This data can be categorized into two types:

   • •

     Well-transcribed Data: Primarily sourced from example sentences and language learning materials, this data exhibits high transcription accuracy with strong alignment between audio and text.

   • •

     Ill-transcribed Data: Obtained from sources like radio press releases, this data often contains discrepancies between the audio content and its accompanying text. However, the audio quality generally remains suitable for TTS training.

2. 2.

   ASR Training: We leverage well-transcribed data, alongside a six-dialect lexicon from the Dictionary of Taiwanese Hakka, to train an Automatic Speech Recognition (ASR) system. This system’s acoustic model encompasses the full range of Hakka phonemes and tones.

3. 3.

   Speech Cleanup: Employing the trained ASR system’s acoustic model (implemented using the Kaldi toolkit [6]), we refine the transcriptions of the ill-transcribed data. For each phrase, a biased language model is generated using its initial transcription. By adjusting discounting constants, we control the influence of this biased language model during transcription refinement. Larger constants prioritize acoustic information, yielding transcriptions closely matching the actual pronunciation.

4. 4.

   Forced Alignment: To minimize extraneous silences and provide phoneme-level timing information, essential for certain TTS architectures, we perform forced alignment on the cleaned transcriptions using the trained ASR system. This yields precise timing information for each phoneme, including silences.

5. 5.

   Silence Trimming: We segment phrases based on long silences (greater than 0.05 seconds), preserving up to 0.025 seconds of silence before and after each segment to ensure natural-sounding transitions.

6. 6.

   Final Data: Finally, the well-transcribed data and the processed, cleaned data are combined to form the final dataset for training the VoxHakka TTS model.

While a well-constructed dataset is essential, further processing of text and speech data (cf. Figure 1) is necessary before training the TTS model. Figure 2 illustrates our TTS model training pipeline.

1. 1.

   Speech Concatenation: The data preprocessing steps described previously yield short speech segments with minimal silence. To enhance the synthesis of longer, more natural-sounding sentences, and to enable the model to learn appropriate pauses between phrases (indicated by commas in text), we implement a sentence concatenation strategy. As shown in Figure 3, a long sentence is segmented into shorter waveforms (e.g., waveform 1, waveform 2, and waveform 3) with corresponding text segments (text 1, text 2, and text 3). These segments are concatenated pairwise, introducing a 0.05-second pause (0.025 seconds from each segment) at each junction. The concatenated waveforms are paired with their corresponding concatenated text, with a comma representing the pause (e.g., “text 1, text 2”).

2. 2.

   Grapheme-to-Phoneme (G2P) Conversion: Unlike pinyin-based writing systems, Taiwanese Hakka utilizes Chinese characters, including numerous specialized characters. Therefore, a robust G2P conversion mechanism, mapping Hakka characters to their corresponding phonetic representations, is crucial. We have developed a comprehensive G2P conversion table, leveraging the government’s online Hakka dictionary, encompassing all Hakka glyphs and their phonetic transcriptions across the six major dialects. For characters with multiple pronunciations, we prioritize the most frequent variant at this stage.

3. 3.

   TTS Model Training: We employ YourTTS [2], a lightweight VITS-based model [7] incorporating language (or dialect) embedding, as our TTS system. YourTTS is well-suited for CPU deployment due to its smaller model size. Importantly, its speaker embedding mechanism eliminates the need for explicit speaker tokens during training. YourTTS accepts three kinds of inputs: phoneme embeddings derived from G2P conversion, dialect embeddings capturing pronunciation variations across dialects, and speaker embeddings generated by a separate speaker model [8] based on the G2P output. These speaker embeddings encode speaker-specific speech characteristics and speaking rate information.

We have presented VoxHakka, a freely available, CC-BY 4.0 licensed multi-speaker TTS system for all six major dialects of Taiwanese Hakka. Addressing the challenges of data scarcity and pronunciation complexity, we developed a comprehensive system encompassing web-based data collection and processing, a robust G2P conversion mechanism, and a carefully tuned TTS model based on YourTTS. Evaluations using CMOS testing demonstrate VoxHakka’s superior performance compared to commercially available Hakka TTS systems, particularly in terms of naturalness. Importantly, VoxHakka’s zero-shot capabilities, enabled by language-specific speaker embeddings, open new possibilities for personalized and adaptable Hakka speech synthesis.

Future research will focus on refining VoxHakka’s pronunciation accuracy by exploring improvements in acoustic modeling. Additionally, we aim to expand the system’s capabilities by incorporating emotional expressiveness and prosodic control. By making VoxHakka openly accessible, we hope to facilitate research, educational initiatives, and creative applications that contribute to the preservation and revitalization of Taiwanese Hakka.

This study was supported by Hakka language teachers Li-Fen Huang (黃麗芬) and Fon-Siin Qi (徐煥昇), whose guidance was crucial. Thanks also to Hakka Radio (講客廣播電臺) and Hakka Public Communication Foundation (客傳會) for providing valuable resources that helped complete this work.