Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia

Neural Machine Translation (NMT) benchmarks are pivotal in documenting and preserving low-resource local languages like those in Indonesia. Prior works Costa-jussà et al. (2022); Cahyawijaya et al. (2022); Winata et al. (2023) have contributed to the creation of these NMT Benchmarks in two significant ways: (1) the creation and compilation of datasets accompanied by (2) exploration and evaluation of different methodologies.

Costa-jussà et al. (2022) focused on developing NMT Benchmarks for low-resource languages. Their NLLB-200 model supports 200+ languages with more than 40K translation directions. Among these 200+ languages, some are local languages in Indonesia. They obtain state-of-the-art results for many translation directions through massive data collection efforts and computing resources.

Similarly, Winata et al. (2023) also collects a multilingual dataset for both machine translation and sentiment analysis for ten local languages in Indonesia. They use the collected dataset to create a benchmark for both tasks, obtaining impressive results in the machine translation tasks for the review domain by fine-tuning pre-trained models. Winata et al. (2023) also shows that fine-tuning a non-English-centric pre-trained model on local Indonesian languages outperforms its English-centric counterpart for the machine translation task.

However, it is essential to recognize that the NMT benchmark created by Costa-jussà et al. (2022) is challenging to replicate. Many researchers and institutions, including top Indonesian universities Cahyawijaya et al. (2022), do not have access to massive compute resources or extensive and proprietary training data required to train the NLLB-200 models. Meanwhile, the NMT benchmark created by Winata et al. (2023) is limited only to the review domain. These leave a research gap that needs to be filled by benchmark NMT models that are replicable and cover more general domains.

Unlike low-resource NMT systems (where either the source or target language is a low-resource language), NMT systems for high-resource languages have achieved impressive results Costa-jussà et al. (2022). Even with the progress achieved by the grassroots movement mentioned in the previous section, the performance gaps are wide. This phenomenon is due to research in the field of NMT and NLP being dominated by English and other major languages, which means that more efforts have significantly been put into developing language technologies for these significant languages, and more data have been collected and made available for these languages Akhbardeh et al. (2021); Kocmi et al. (2022). This means that while low-resource NMTs face problems no longer found in high-resource NMTs, insufficient resources and attention are being allocated.

One significant issue NMT systems face is the pivotal role parallel data plays in model performance Koehn and Knowles (2017). By definition, low-resource languages have little to no parallel data. One problem that negatively impacts the model performance is out-of-vocabulary (OOV) occurrences Aji et al. (2022); Wibowo et al. (2021), where the model needs to see a token more to learn what it means. While this issue exists even in high-resource languages, the rate of occurrence for low-resource languages is substantially higher. However, usage of byte pair encoding (BPE) Sennrich et al. (2016b) is capable of alleviating this issue to some degree Lample et al. (2018b); Yang et al. (2020).

While previous research has made noteworthy strides in addressing out-of-vocabulary (OOV) occurrences, the most effective solution continues to be expanding available training data. However, building textual resources for translation tasks necessitates a significant investment of money, time, and expertise. Because of this, current research increasingly centers on finding innovative ways to augment the training of NMT models.

To combat the issue of data starvation, many researchers aim to utilize monolingual data to train NMT systems Lample et al. (2018a); Artetxe et al. (2018); Conneau and Lample (2019) and find ways to generate more training data, either comparable or synthetic data. Comparable data are extracted using various bitext retrieval methods Zhao and Vogel (2002); Fan et al. (2021); Jones and Wijaya (2021); Kocyigit et al. (2022), multimodal signals Hewitt et al. (2018); Rasooli et al. (2021), dictionary- or knowledge-based approaches Wijaya and Mitchell (2016); Wijaya et al. (2017); Tang and Wijaya (2022); while synthetic data are created and utilized either through innovative training data augmentation Kuwanto et al. (2021), utilizing automatic back-translation Sennrich et al. (2016a); Wang et al. (2019), or even outright generating synthetic data using generative models Lu et al. (2023), which has gained increasing attention by the community lately due to the advancement of large language models (LLMs).

Both Artetxe et al. (2018) and Lample et al. (2018a) show that NMT systems can be trained using only monolingual data while achieving impressive results. Conneau and Lample (2019) then create the XLM architecture, which allows NMT systems to be traine using monolingual and parallel data. Afterward, Kuwanto et al. (2021) exploits the cross-lingual nature of the XLM models by corrupting the monolingual data using code-switching, which makes a single training instance contain multiple languages. The result is an improvement in the model’s performance for low-resource translation.

In addition, prior works also focus on obtaining synthetic training data by turning monolingual data into parallel data through automatic back-translation Sennrich et al. (2016a) or by using LLMs such as the gpt-family models Brown et al. (2020) that have been gaining popularity in recent years in many fields Lu et al. (2023). While back-translation has evolved, becoming a prominent method in the field of NMT Artetxe et al. (2018); Conneau and Lample (2019), using LLMs to generate synthetic data has yet to be thoroughly explored. This trend of using generative AI to generate synthetic training data displays initial potential, considering their remarkable performances compared to the state-of-the-art in machine translation Zhu et al. (2023). However, further research with ablation studies and the inclusion of more language coverage is still needed.

We employ three primary training approaches to build our NMT models:

From Scratch (Scratch): In this approach, models are trained from the ground up without any reliance on pre-existing pre-trained language models. This approach acts as a baseline and allows us to gauge the performance of the models when trained from scratch.

Pre-trained Cross-Lingual (PreXL): Here, an NMT model utilizes a pre-trained cross-lingual model (XLM) Conneau and Lample (2019) on two sets of monolingual data. One of the sets is the Indonesian monolingual data, and the other is the low-resource local language monolingual data. This provides a strong starting point for the NMT by initializing the model with knowledge from the target and source languages. Therefore, each language pair in this work is given its own respective model. The number of pre-trained models for PreXL equals the number of language pairs in our work, which is four.

Code-switched Pre-trained Cross-Lingual (CodeXL): This approach involves pre-training the language model using additional augmented data from the two sets of monolingual data and a bilingual dictionary through code-switching, explained later in section 3.6. Code-switching allows for a bilingual context within each training instance. The pre-trained model is then fine-tuned for translation. The tasks used to fine-tune CodeXL and PreXL depend on the training paradigm used (section 3.2. The number of pre-trained models for CodeXL is the same as PreXL, which is four.

We chose the XLM architecture due to its modest compute resource requirements and its capability of cross-lingual language modeling. Moreover, the architecture is widely used for many low-resource language pairs and shows impressive results despite its modest size Wang et al. (2019). We use Masked Language Modeling (MLM) Devlin et al. (2019) to pre-train all the XLM models.

Additionally, we also explore two training paradigms, each influencing how the NMT models learn and what data are used for training:

Unsupervised NMT (Unsup): This paradigm trains the NMT system using only monolingual data of the source and target language. Utilizing both denoising-autoencoding Vincent et al. (2010) and automatic back-translation Sennrich et al. (2016a) to train the NMT system. Note that even though CodeXL utilizes a bilingual dictionary, it does not use any parallel data during pre-training.

Semi-supervised NMT (Semisup): This paradigm trains the NMT system using both monolingual data and parallel data of the source and target language. Monolingual data are utilized for training NMT by automatic back-translation.

We employ these shortened terms throughout our experiments to refer to the respective training approaches and paradigms. Our results indicate that the performance of each combination of approaches and paradigms on the evaluation dataset depends heavily on the amount of available data for the language: Unsup paradigm works better for very low-resource languages. In contrast, Semisup paradigm performs better when at least 10K parallel data is available Artetxe et al. (2018). We do not conduct training using a strictly Supervised NMT paradigm because prior work has shown automatic back-translation’s undeniable impact in improving low-resource NMT systems performance Sennrich et al. (2016a).

Following recent trends of using generative AI to generate synthetic training data Lu et al. (2023); Zhu et al. (2023), we explore the impact of synthetically generated data on low-resource language NMT systems. We define two main approaches to generating synthetic data: (1) generating parallel data using generative AI and (2) translating monolingual data using an existing model.

To gauge the impact of the synthetically generated training data, we train NMT systems with these additional data using the Scratch and CodeXL training approaches. Scratch is also used in our preliminary experiments to identify the synthetic data generation approach that would yield the best empirical results. Once we identify the best approach, we apply the same synthetic data generation approach to all our language pairs and use the generated data to augment the training of our NMT approach with the Semisup paradigm.

Through the preliminary experiments (reported in Appendix C), we find that synthetic data generated using generative AI (gpt-3.5-turbo) has the most positive impact on training NMT systems. We generate 5000 parallel sentences for each language pair via a zero-shot prompt²²2Repository of the data we generate using zero-shot prompting: "Generate a long parallel sentence in SRC and TGT", where SRC and TGT is the pair of language we want to generate the sentences in. Appendix C provides justifications for these choices.

Denoising autoencoding (DAE) Vincent et al. (2010) is a popular training objective for fine-tuning pre-trained LM for unsupervised MT tasks Lample et al. (2018b); Wang et al. (2019) for its ability to increase the robustness of NMT models.

By utilizing the XLM architecture, our NMT system can perform multi-way translation. Thus, we also utilize automatic back translation (BT) Sennrich et al. (2016a) during fine-tuning of our NMT models with Unsup and Semisup paradigms. By performing back translation using the same model that is being trained, synthetic parallel data is obtained and used automatically during training.

We obtain our monolingual data from multiple publicly available sources. For Indonesian (id), we use the 201M monolingual sentences available from the Indo4B curated dataset Wilie et al. (2020). We obtain monolingual data for the local languages through publicly available data such as Wikidumps³³3Wikidumps Page, cc100 Conneau et al. (2020), imdb-jv Wongso et al. (2021), jadi-ide Hidayatullah et al. (2020), and su-emot Putra et al. (2020). The amount of monolingual sentences used to train each language is available in Table 1, with further breakdown available in Appendix A.

All parallel data we use to train the model are also publicly available from the NusaCrowd repository Cahyawijaya et al. (2022). We scan the repository for datasets that contain parallel data of the local language paired with Indonesian. The amount of parallel sentences used to train each language pair is available in Table 1. Our largest language in terms of monolingual and parallel sentences, Javanese, is a tiny fraction (almost a 20th and a 500th, respectively) of NLLB-200 reported sentences for Javanese. From publicly available resources in Table 1, we can see that these four languages represent low-resource languages. A further breakdown is available in Appendix B.

The sentence counts in Table 1 are after we perform filtering on both monolingual and parallel data. For monolingual data, we remove sentences that contain less than three words or more than 250 words. We also perform simple filtering for sentences obtained from Wikipedia, including deduplication, removing HTML tags, removing sentences with only numbers, removing sentences that do not start with an alphabet, and removing metadata, bulletin points, or number ordering from sentences. For parallel data, we remove sentences that contain less than three words or more than 250 words and remove sentence pairs whose source sentences have a word count ratio above 1.5 of their translations following the setup of Ghazvininejad et al. (2023).

In this paper, code-switching is done by utilizing the system made by Kuwanto et al. (2021)⁴⁴4Code Switch-based Curriculum Training. Code-switching is used to create synthetic training data by utilizing a bilingual dictionary. The generated data is used only during pre-training and is treated as a third language (labeled cs), where each training instance contains tokens from the other two languages. Results obtained by Kuwanto et al. (2021) imply that this method helps the model by giving stronger cross-lingual signals, which helps translation tasks during fine-tuning.

Creating synthetic training data through code-switching utilizes training data from monolingual datasets from both languages in the system. By utilizing a bilingual dictionary, obtained and parsed from Winata et al. (2023)⁵⁵5NusaX’s bilingual dictionary, each instance of training data from both monolingual datasets is augmented. Figure 1 illustrates this process, while Table 2 shows how much augmented training data is available for each NMT system.

The id-jv language pair is particularly significant due to its relevance in Indonesia, where approximately 198 million people speak Indonesian (id), and Javanese (jv) is spoken by roughly 68.2 million Eberhard et al. (2023).

Looking at Table 3, we observe a substantial gap in translation performance between id$\rightarrow$jv and jv$\rightarrow$id, emphasizing the performance asymmetry. Notably, when training NMT systems using the Unsup paradigm, CodeXL consistently outperforms other approaches for both translation directions, reinforcing the findings of Kuwanto et al. (2021), which highlight the generalization capability of this approach.

Surprisingly, when we train NMT models using additional parallel data generated by gpt-3.5-turbo (CodeXL_AUG), we notice a slight decline in performance for id$\rightarrow$jv translation compared to our best-performing model (CodeXL). A more detailed comparison is discussed in the next section.

Table 3 shows where id-su differs from the id-jv language pair. For id-su language pair, the Unsup paradigm shows a different trend where CodeXL has a slightly worse performance compared to PreXL in the id$\rightarrow$su translation. However, this trend shifts when using the Semisup paradigm, with CodeXL regaining its superiority.

Similar to id-jv language pair, an intriguing phenomenon arises when we train NMT models using additional parallel data generated by gpt-3.5-turbo (CodeXL_AUG) for the id-su language pair. While this approach does not create a better performing model in id$\rightarrow$su translation, it does result in a slightly better model for su$\rightarrow$id. This trend indicates that the generated synthetic parallel data’s impact heavily depends on the generative AI’s translation performance. For both id-jv and id-su language pairs, gpt-3.5-turbo’s zero-shot translation performance on id$\rightarrow$x is worse than CodeXL for each respective language pair, therefore CodeXL_AUG does not result in improved performance. Meanwhile, the reverse is true, gpt-3.5-turbo’s x$\rightarrow$id translation performance is better than CodeXL in x$\rightarrow$id direction, hence CodeXL_AUG has a better performance in this direction.

The results we obtained for id-min follow a similar pattern as id-jv. The trend where CodeXL models performed better than Scratch and PreXL continues for id-min for both translation directions.

However, unlike id-jv and id-su, using synthetically generated parallel data to train NMT systems for id-min (CodeXL_AUG) performed better than CodeXL on the min$\rightarrow$id translation. This is surprising because CodeXL performed better than Turbo on min$\rightarrow$id, yet the parallel data generated by Turbo was able to create CodeXL_AUG, which is a better performing NMT system. This breaks the previous trends set by id-jv, id-su, and even id-ban in the later section.

id-ban continues the trend set by the majority of previous language pairs. Following id-jv and id-min language pairs, CodeXL consistently has superior performance compared to Scratch and PreXL.

Additionally, id-ban follows the trend set by id-jv and id-su, where the use of synthetically generated parallel data from Turbo creates a better NMT system compared to others that do not use them. For id-ban language pair specifically, Turbo’s translation performance is much higher than CodeXL, and the data Turbo generated has a significant impact during training, as seen in CodeXL_AUG. The difference in score for CodeXL and CodeXL_AUG differs by 3+ and 6+ spm200BLEU for id$\rightarrow$ban and ban$\rightarrow$id respectively. This performance difference is much more significant compared to id-jv, id-su, and id-min, where the performance difference is less than 1 spm200BLEU. This finding supports the idea that the generated synthetic parallel data’s impact heavily depends on the generative AI’s translation performance. Moreover, if the initial parallel data is limited, like in the case of id-ban (only 0.9K), the addition of synthetic data can greatly improve performance.

However, Table 3 shows that Unsup training paradigm created the best performing NMT system for id-ban language pair. While it would not be surprising for Scratch due to the limited amount of parallel training data, it is surprising that Scratch_AUG does not result in a considerably better NMT system, as the parallel training data size becomes 6x its original size (i.e., from 0.9K to 5.9K). This indicates that denoising-autoencoding plays a more significant role in model performance than parallel data when the training parallel data is limited.