Cross-Lingual Machine Speech Chain for
Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis

The Indonesian language, Bahasa Indonesia, is derived from the Malay dialect, which was the lingua franca of Southeast Asia [Quinn, 2001]. Bahasa Indonesia is closely related to the Malay spoken in Malaysia, Singapore, Brunei, and some other areas. It is the largest member of the Austronesian language family. The only difference is that Indonesia (a former Dutch colony) adopted the Van Ophuysen orthography in 1901; Malaysia (a former British colony) adopted the Wilkinson orthography in 1904. In 1972, the governments of Indonesia and Malaysia agreed to standardize “improved” spelling, which is now in effect on both sides. Even so, modern Indonesian and modern Malaysian are as different from one another as are Flemish and Dutch [Tan, 2004].

Many words in the Indonesian vocabulary reflect the historical influence of the various colonial cultures that occupied and influenced the archipelago. Indonesian words have borrowed heavily from Indian Sanskrit, Chinese, Arabic, Portuguese, Dutch, and English [Jones, 2007]. Unlike Chinese, it is not a tonal language; it has no declensions or conjugations. It has no changes in nouns or adjectives for different gender, number, or case. Verbs do not take different forms to show number, person, or tense. A time adverb or question word can be placed at either the front or the end of sentences. Since plural is often expressed by reduplication, Indonesian sentences have reduplication words. It is also a member of the agglutinative language family, denoting a complex range of prefixes and suffixes that are attached to base words that can result in very long words [Sakti et al., 2004].

The standard Indonesian language is mostly used in such formal written settings as books, newspapers, and television/radio news broadcasts. Although the earliest records in Malay inscriptions are syllable-based and written in Arabic script, modern Indonesian is phonetic-based written in Roman script [Alwi et al., 2003]. It only uses 26 letters, as in the English/Dutch alphabet.

In this study, we chose to work with four ethnic languages: Javanese, Sundanese, Balinese, and Bataks. Since a large number of the population speak them, the data collection of their native speakers remain possible to reach. However, despite their significant speech communities, the primary usage of these languages is gradually being subsumed by Bahasa Indonesia. The Javanese, Sundanese, Balinese, and Bataks languages also suffer from the inadequate intergenerational transmission, since they are often not being passed on to the next generation. It is pointed out that even such languages are at risk of language endangerment [Cohn and Ravindranath, 2014].

The Indonesian speech dataset was developed by the R&D Division of an Indonesian telecommunications company (PT. Telekomunikasi Indonesia) in collaboration with ATR Japan under the Asia Pacific Telecommunity (APT) project \citelanguageresourcesakti_2004_1,sakti_2008_1. The corpus consists of 80.5 hours of speech spoken by multiple speakers. The following are its details of the data resources.

The Indonesian ethnic languages covered in this work include Javanese, Sundanese, Balinese, and Bataks, all of which were collected and recorded in a previous study \citelanguageresourcesakti_2013_1.

The human speech chain was previously introduced [Denes et al., 1993] as a phenomenon in human communication (Fig. 1). In a conversation, the speaker’s utterance is heard by the listener and also the speaker herself. A feedback chain is established among the speaker’s hearing system, her brain, and the speech production system. When the speaker listens to her speech, she compares it to her intended quality and uses it to improve its quality in the next timestep. The speaking and listening processes occur simultaneously and are continually repeated until the end of the utterance.

Inspired by the human speech chain, a machine speech chain was proposed to jointly train ASR and TTS models in semi-supervised learning. An overview of the machine speech chain [Tjandra et al., 2019b] is shown in Fig. 2(a). The framework consists of a sequence-to-sequence ASR and a sequence-to-sequence TTS. Sequence-to-sequence networks are deep-learning architecture that include an encoder and a decoder with an attention mechanism. The machine speech chain establishes a loop that connects ASR to TTS and TTS to ASR. The components are trained with a semi-supervised approach that consists of two stages: supervised and unsupervised training.

First, we supervisedly train the standard Indonesian ASR and TTS using Indonesian speech-text paired data (see Section 3.1.). Here the Indonesian language serves as prior knowledge for the system. In this stage, since Indonesian speech-text paired data are available, both the Indonesian ASR and TTS components are trained independently, as seen in Fig. 3. The ASR takes a sequence of Indonesian speech features $\mathbf{x}^{(IND)}$ and learns to transcribe it into text $\mathbf{\hat{y}}^{(IND)}$. The TTS takes Indonesian text sentence $\mathbf{y}^{(IND)}$ and learns to generate speech $\mathbf{\hat{x}}^{(IND)}$. The speech data here consist of multi-speaker speech. Therefore, the TTS input also includes speaker embedding vector $z=SPKREC(\mathbf{x}^{(IND)})$, and so the synthesized speech can be compared to the ground speech with appropriate voice characteristics, as proposed in a previous machine speech chain study [Tjandra et al., 2018].

In this phase, we utilized the previously pre-trained Indonesian ASR and TTS in the speech-chain architecture to unsupervisedly train the ASR and TTS of Javanese, Sundanese, Balinese, and Bataks. Since the available data are minimal, we combined all the unlabeled data (only text or only speech) from the Javanese, Sundanese, Balinese, and Bataks corpus (see Section 3.2.). This phase is shown in Fig. 4, and the following are the details of the unrolled processes.

1. 1.

   Unsupervisedly train the system given only the text transcription of the Indonesian ethnic languages (Closed-loop from TTS to ASR): In this process, the provided ground information only consists of text data from the four ethnic languages. The TTS attempted to synthesize a sequence of speech features in particular ethnic language $\mathbf{\hat{x}}^{(ETH)}$, based on given text $\mathbf{y}^{(ETH)}$ and the speaker embedding vector. Speaker embedding vector $\hat{z}$ is generated based on speech sampled from the available speech data ($\mathbf{\tilde{x}}$). After the speech synthesis, ASR attempted to transcribe back ethnic language text $\mathbf{\hat{y}}^{(ETH)}$ based on the TTS output $\mathbf{\hat{x}}^{(ETH)}$. The loss is then calculated by comparing the original ethnic text transcription $\mathbf{y}^{(ETH)}$ and the ASR output $\mathbf{\hat{y}}^{(ETH)}$; and perform back-propagation to improve the ASR.

2. 2.

   Unsupervisedly train the system given only the speech utterances of the Indonesian ethnic languages (Closed-loop from ASR to TTS): This process uses only speech data from the four ethnic languages. First, ASR took original ethnic speech $\mathbf{x}^{(ETH)}$ and predicted its transcription $\mathbf{\hat{y}}^{(ETH)}$. The TTS then reconstructed ethnic language speech $\mathbf{\hat{x}}^{(ETH)}$ by processing the text generated by ASR and speaker embedding vector $z=SPKREC(\mathbf{x}^{(ETH)})$. The loss is calculated by comparing the original ethnic speech utterances $\mathbf{x}^{(ETH)}$ and the generated TTS speech output $\mathbf{\hat{x}}^{(ETH)}$; and perform back-propagation to improve the TTS.

For supervised training on both the ASR and TTS components of the Indonesian language, we chose 10% of the speech-text paired data with 40 speakers for testing. On the remaining speech-text paired data with 360 speakers, 20% of the data were randomly selected for validation or development sets, and 80% of the rest was used as a training set.

For unsupervised training of both the ASR and TTS components of the Javanese, Sundanese, Balinese, and Bataks languages, we randomly chose 50 unpaired data from four speakers (200 speech utterances or text transcription only) of each language as a test set. On the remaining data of 225 speech or text unpaired data with six speakers (1350 speech utterances or text transcription only), 10% of the data were randomly selected for validation or development sets, and 90% of the rest was used for training.

We extracted two different sets of speech features. First, we applied pre-emphasis (0.97) on the raw waveform and then extracted the log-linear spectrogram with a 50-ms window, 12.5-ms steps, and a 2048-point short-time Fourier transform (STFT) with the Librosa package [McFee et al., 2017]. Second, we extracted the log Mel-spectrogram with an 80 Mel-scale filterbank. For our TTS model, we used both log-linear and log-Mel spectrogram for the first and second outputs. For our ASR and speaker recognition components, we used the log-Mel spectrogram for the encoder input.

The text utterances were tokenized as Indonesian characters and mapped into a 33-character set: 26 alphabetic letters (a-z), three punctuation marks (’.-), and four special tags, $\langle$noise$\rangle$, $\langle$spc$\rangle$, $\langle$s$\rangle$, and $\langle$/s$\rangle$ as noise, space, start-of-sequence, and end-of-sequence tokens, respectively. Both the ASR input and the TTS output shared the same text representation in the training and inference stages.

Table 1 shows the character error rate (CER (%)) performance of the ASR systems from multiple scenarios evaluated on those ethnic language test data. In the first block, we supervisedly trained our baseline system just using the speech-text paired data of the Indonesian language. Unfortunately, the number of errors (refer to $S+D+I$ in Eq. 1) in the recognition output exceeds the number of characters in the reference ($N$), resulting in a CER that above 100%. This indicates that directly using the Indonesian ASR is difficult for recognizing ethnic languages. In the second block, we showed our proposed approach that utilized cross-lingual speech-chain framework (see the training process in Section 5.). We utilized the previously pre-trained Indonesian ASR and TTS. We then develop ASR of those ethnic languages given only text or both text and speech (but unpaired). The results reveal that given only text data, the proposed system improved the performance and reduced the average CER from 101.45% to 67.59%, which is 33.86% absolute reduction. If both text and speech data exist (but unpaired), we might further reduce the average CER to 32.08%, which is 69.37% absolute reduction from the baseline. In the last block, we showed the topline system in which the system was trained using both paired speech and the text of the Indonesian and Indonesian ethnic languages in a supervised manner. The system’s performance achieved an average CER of 20.05%.

Similar to the ASR evaluation, Table 2 shows the performance of the TTS systems from multiple scenarios evaluated on the ethnic language test data in L2 norm-squared error between the ground-truth and the predicted speech as a regression task. The baseline model was supervisedly trained using only the speech-text paired data of the Indonesian language, and the performance reached 1.162 of the L2 norm-squared on average. For the proposed system, we observed similar trends with the ASR results, where semi-supervised training with the speech chain method improved significantly over the baseline and achieved a L2 norm-squared reduction to 0.537 of L2 norm-squared on average. This performance is close to the upper-bound result, which is 0.441 of the L2 norm-squared on average.